v1.4.2
Fix: SanitizeForTokenizer space counting with problematic Unicode runes (#18)
Bug: SanitizeForTokenizer() replaced Box Drawing characters (U+2500–U+257F etc.) with spaces but reset the consecutive-space counter, allowing long space runs to survive and trigger the sentencepiece normalizer panic in sugarme/tokenizer v0.3.0.
Fix: Unified the space and problematic-rune branches so both increment spaceCount. This ensures that sequences like " ─── " are properly collapsed instead of producing 9 consecutive spaces.
Impact: Eliminates the remaining 2 chunk failures reported in v1.4.1 testing (ASCII flowcharts and wide Markdown tables with Box Drawing borders).
Credit: @hrkzogw for the thorough root cause analysis and suggested fix.