Skip to content

v1.4.2

Choose a tag to compare

@tomohiro-owada tomohiro-owada released this 27 Mar 03:13

Fix: SanitizeForTokenizer space counting with problematic Unicode runes (#18)

Bug: SanitizeForTokenizer() replaced Box Drawing characters (U+2500–U+257F etc.) with spaces but reset the consecutive-space counter, allowing long space runs to survive and trigger the sentencepiece normalizer panic in sugarme/tokenizer v0.3.0.

Fix: Unified the space and problematic-rune branches so both increment spaceCount. This ensures that sequences like " ─── " are properly collapsed instead of producing 9 consecutive spaces.

Impact: Eliminates the remaining 2 chunk failures reported in v1.4.1 testing (ASCII flowcharts and wide Markdown tables with Box Drawing borders).

Credit: @hrkzogw for the thorough root cause analysis and suggested fix.