New public APIs added:

• SemanticSlicer.Slicer.SplitDocumentChunksRaw() - Split content directly without preprocessing
  • Treats content exactly as provided (no normalization/HTML stripping/whitespace collapsing/trimming)
  • Offsets relative to exact input string
  • Supports overlap, metadata, and chunk headers
• SemanticSlicer.TextUtilities - Public preprocessing utilities
  • NormalizeLineEndings(string) - Converts all line endings to Unix-style
  • CollapseWhitespace(string) - Limits consecutive spaces/newlines to max 2
• SemanticSlicer.Slicer.CountTokens() - Count tokens using configured encoder
• SemanticSlicer.Slicer.PrepareContentForChunking() - Get preprocessed content separately

• Add support for custom tokenizers

• Adds optional chunk overlap parameter to improve context retention between chunks

• The library can still be directly used as a Nuget package
• Can now also be run as command-line executable/daemon, service, or REST API
• Supports Mac/Linux/Windows
• See the README.md for details

• Performance improvements

• Update Tiktoken to v2