Skip to content

v1.4.1

Choose a tag to compare

@tomohiro-owada tomohiro-owada released this 26 Mar 20:43

Bug Fixes

  • Fix tokenizer panic root cause (#18): The real crash trigger was runs of 4+ consecutive spaces (deep indentation), not Unicode characters. The sentencepiece normalizer in sugarme/tokenizer v0.3.0 has an off-by-one bug in byte offset tracking for long whitespace runs. SanitizeForTokenizer() now collapses runs of 4+ spaces to 3 before tokenization, eliminating the remaining zero-vector fallbacks.

    Thanks to @hrkzogw for the detailed root cause analysis.