CodeSight

AI-powered document search engine — hybrid BM25 + vector + RRF retrieval with pluggable LLM answer synthesis.

Quick Start

# Install
pip install -e ".[dev]"

# Install with AST chunking support (Python/JS/TS — higher MRR for code)
pip install -e ".[dev,ast]"

# Index a folder of documents
python -m codesight index /path/to/documents

# Search (hybrid BM25 + vector)
python -m codesight search "payment terms" /path/to/documents

# Filter by file type
python -m codesight search "auth" /path/to/code --glob '*.py'

# Ask a question (requires LLM API key — see Configuration)
python -m codesight ask "What are the payment terms?" /path/to/documents

# Machine-readable output
python -m codesight search "query" /path --json

# Check index status
python -m codesight status /path/to/documents

# Launch the web chat UI
pip install -e ".[demo]"
python -m codesight demo

Related MCP server: semantic-search-mcp

Python API

from codesight import CodeSight

engine = CodeSight("/path/to/documents")
engine.index()                                     # Index all files
results = engine.search("payment terms")           # Hybrid search
answer = engine.ask("What are the payment terms?") # Search + LLM answer
status = engine.status()                           # Index freshness check

The package root exports CodeSight, ServerConfig, Answer, IndexStats, RepoStatus, and SearchResult for stable public imports.

Supported Formats

Architecture

• Document Parsing: PDF, DOCX, PPTX text extraction with page/section metadata

• Chunking: AST-based (tree-sitter) for Python/JS/TS — function/class boundaries preserve semantic units. Regex fallback for other languages. Paragraph-aware splitting for documents.

• Embeddings: voyage-code-3 (API, code files) / all-MiniLM-L6-v2 (local, docs). Auto-detected via VOYAGE_API_KEY.

• Vector Store: LanceDB (serverless, file-based)

• Keyword Search: SQLite FTS5 sidecar

• Retrieval: Hybrid BM25 + vector + code-vector with RRF merge → metadata filename boost → optional reranker

• Reranker: voyage rerank-2 (code-aware, auto-enabled with VOYAGE_API_KEY). Local ms-marco cross-encoder opt-in only.

• Answer Synthesis: Pluggable LLM backend (Claude, Azure OpenAI, OpenAI, Ollama)

See ARCHITECTURE.md for the full system tour.

Performance

Measured on the holusight codebase (96 files, 20 representative queries):

AST chunking is the largest single lever (+0.224 MRR). The local ms-marco cross-encoder hurts code retrieval — only enable it explicitly.

Configuration

See .env.example for all options.

Stack

• Python 3.11+

• LanceDB + SQLite FTS5

• sentence-transformers + voyage-code-3 (optional)

• tree-sitter (optional — AST chunking for Python/JS/TS)

• Anthropic Claude API / Azure OpenAI / OpenAI / Ollama

• Streamlit (web chat UI)

• pymupdf, python-docx, python-pptx (document parsing)