ken v1.0.0 — code search for agents, in one binary

ken is a hybrid code-search engine for AI agents: BM25 lexical + Model2Vec semantic embeddings + reciprocal-rank fusion + a code-aware ranking pipeline, served over MCP. It is a faithful Go port of MinishLab/semble — the retrieval algorithm is ported verbatim and validated against the Python reference — packaged as a single static, pure-Go, no-cgo binary: no Python, no GPU, no vector DB, no API keys, air-gapped friendly.

1.0 means two things: the public API surface is frozen (mcp.Run, mcp.NewServer, mcp.NewCache, FormatResults, the chunk.Chunker interface, and the typed JSON-output structs), and the retrieval quality is measured, reproducible, and at its ceiling for this corpus class — every headline number below has an in-tree reproduction harness.

The numbers (all reproducible — see docs/BENCH.md)

• ~97% recall@10 in the default (hybrid) mode — 0.967 NL / 0.995 symbol on semble's 1,251-query, 63-repo benchmark — at ~46× fewer agent tokens than grep + Read for NL queries. grep still wins exhaustive enumeration (~99.9%); ken's job is "find the chunk that answers this," at 1–2 orders of magnitude lower token cost.
• NDCG@10 0.842 vs semble's published 0.854 — the verbatim-port validation, with the semantic-only arms matching within 0.003.
• Opt-in neural reranker: +0.165 NDCG@10 on CoIR — a CodeRankEmbed transformer running entirely in pure Go (hand-written ARM64 NEON assembly, optional int8 quantization, disk-persisted embedding cache).
• At 280K-file corpus scale, grep+Read needs ~16M tokens per NL query (past any context window); ken answers in ~1,300.

Highlights

Zero-setup onboarding — new installs land on the good path

ken-mcp now auto-fetches the embedding model (~60 MB) in the background on first run: it serves BM25 immediately, then transparently upgrades to hybrid (~0.83 → ~0.97 recall@10) when the model lands. No flags, no setup step, stderr-only progress — the JSON-RPC stream stays clean. (KEN_MCP_AUTO_FETCH=0 restores warn-and-downgrade.)

Structural navigation — nine MCP tools, 13 languages

Beyond ranked search (search, find_related), ken builds a structural index from its tree-sitter parse and answers structural questions exactly, by lookup: definition, references, callers, outline, symbols — plus status and recently_changed. Honest framing baked into the tool descriptions: name-resolved, not type-resolved (for type-level call hierarchy, use an LSP). The callers edge data ships with a published precision sample (100% on 400 edges across 8 languages). Languages: Python, Go, TypeScript, JavaScript, Java, Rust, C, C++, C#, PHP, Ruby, Kotlin, Dart.

Retrieval quality, measured the hard way

• Structural enrichment (default-on): every chunk from a supported language is prefixed with an AST-derived label (# func: NAME | calls: A, B | raises: X) before indexing — +0.02 NDCG@10 on leak-free benchmarks, pure Go, no model, KEN_ENRICH=off to disable.
• The recall numbers come from a decomposition harness (internal/search/recall_decomp_test.go) that separates candidate-generation loss from ranking loss — the same harness that caught our own docs misreporting the BM25-only fallback (~0.84) as ken's recall. The corrected, reproducible story is in docs/BENCH.md.

An index that keeps up

• Watch mode by default: ken index stays alive and re-publishes an atomic snapshot ~2s after any file change; ken-mcp always watches, so agents see their own edits.
• Pre-built indices: serialize an index into your binary (mcp.Run + //go:embed) for instant-start embedded corpora; ken-mcp auto-loads <repo>/.ken/index.bin.
• Database schema indexing: point ken at Postgres/MySQL/MariaDB/SQLite and search your schema + migration history alongside code (reindex_db).

Drop-in for semble

Same search / find_related tool schemas, same markdown wire format. If your agent already uses semble, swap the command: path and nothing else changes.

Runs everywhere

macOS, Linux, Windows (amd64 + arm64). brew install --cask townsendmerino/tap/ken, scoop install ken, direct download, or go install. Single static binaries; slim builds ~23 MB (ken) / ~40 MB (ken-mcp).

Try it in 60 seconds

brew install --cask townsendmerino/tap/ken   # or scoop / download / go install
claude mcp add ken -s user -- $(which ken-mcp)
# ask your agent: "where do we handle retry backoff?" — ken auto-fetches the model on first run

Or try the flagship demo — the Go standard library, pre-indexed, model baked in, zero config: ken-demo-go-stdlib. Every Go developer knows this corpus by heart — verify the answers yourself, then reproduce against your own $GOROOT/src in 30 seconds (demos/go-stdlib/README.md). Supporting demos: kubernetes (90K chunks, scale) and postgres (treesitter on C).

Honest limits

• Exhaustive enumeration belongs to grep. Refactors and pre-rename audits need 100% literal recall; ken optimizes relevance per token, not completeness.
• Structural tools are tree-sitter-grade. Name resolution, ranked candidates — not a compiler. Same-spelled symbols collapse; type-level questions need an LSP.
• The neural reranker is opt-in (--mode=hybrid-rerank / KEN_MCP_RERANK=on): it buys NDCG at real latency cost on first contact with a repo (the embedding cache persists to disk and is fast thereafter).

Provenance & thanks

ken stands on semble and the potion-code-16M model by MinishLab, and CodeRankEmbed by Nomic AI / CoRNStack for the reranker. Tree-sitter parsing via gotreesitter; algorithm packages live in aikit (pinned at its own 1.0). Everything runs in pure Go — including the transformer.

Full change history: CHANGELOG.md. Benchmarks and methodology: docs/BENCH.md.