v1.4.1
Bug Fixes
-
Fix tokenizer panic root cause (#18): The real crash trigger was runs of 4+ consecutive spaces (deep indentation), not Unicode characters. The sentencepiece normalizer in
sugarme/tokenizerv0.3.0 has an off-by-one bug in byte offset tracking for long whitespace runs.SanitizeForTokenizer()now collapses runs of 4+ spaces to 3 before tokenization, eliminating the remaining zero-vector fallbacks.Thanks to @hrkzogw for the detailed root cause analysis.