Vault Cortex implements a high-performance hybrid search and discovery engine powered by SQLite FTS5 and sqlite-vec. Unlike simple grep-based search, this system maintains a structured index of Markdown content, vector embeddings for semantic similarity, frontmatter metadata, a Kanban-aware task index, and the vault's link graph.
The search system is decomposed across nine modules, coordinated by search-index.ts. The createSearchIndex factory src/vault-mcp/search/search-index.ts246-320 manages the database lifecycle, migrations, and the SearchQueryContext src/vault-mcp/search/search-queries.ts64-88 that binds queries to the active database instance.
The system follows a "rebuild-then-incremental" lifecycle. On server bootstrap, rebuildFromVault src/vault-mcp/search/search-index.ts476-545 performs a full scan. Once the initial index is ready, startFileWatcher src/vault-mcp/search/file-watcher.ts64-190 manages incremental updates.
Search and Indexing Pipeline
Sources: src/vault-mcp/search/search-index.ts322-420 src/vault-mcp/search/file-watcher.ts76-131 src/vault-mcp/search/search-queries.ts251-348
Hybrid search combines the precision of keyword matching with the conceptual reach of vector embeddings. Sequential execution runs FTS5 keyword search followed by vector KNN, fusing results via RRF.
embedder.ts, chunker.ts)chunkNoteContent src/vault-mcp/search/chunker.ts89-158 splits notes into ~450-token fragments. It is heading-aware src/vault-mcp/search/chunker.ts101 sub-splitting oversized sections at paragraph boundaries and prefixing every chunk with the note title for context src/vault-mcp/search/chunker.ts80-84createEmbedder src/vault-mcp/search/embedder.ts108-142 provides a lazy-loading ONNX pipeline for bge-small-en-v1.5 (INT8 quantized) with a singleton guard against concurrent model downloads. It employs content-hash gating (SHA-256) src/vault-mcp/search/embedder.ts21-36 to skip re-embedding unchanged chunks on both incremental updates and full rebuilds.EMBEDDING_ENABLED=false disables the model download and reverts search to FTS-only mode.rrf.ts, reranker.ts)computeRrfScores src/vault-mcp/search/rrf.ts16-56 merges FTS and Vector results. It uses $k=60$ src/vault-mcp/search/rrf.ts18 and applies top-rank bonuses to reward results highly placed by either system.RERANK_MODE="blended", the top candidates are rescored using ms-marco-MiniLM-L-6-v2 via createReranker src/vault-mcp/search/reranker.ts19-80blendScores src/vault-mcp/search/reranker.ts157-179Sources: src/vault-mcp/search/search-queries.ts251-348 src/vault-mcp/search/rrf.ts16-56 src/vault-mcp/search/reranker.ts157-179 src/vault-mcp/search/embedder.ts21-36
The database uses better-sqlite3 and sqlite-vec for persistence.
| Table | Purpose |
|---|---|
notes | Core metadata: path, title, tags, folder, properties (JSON), leading_callout. |
notes_fts | FTS5 virtual table with porter unicode61 tokenizer for title, content, and metadata. |
vec_notes | sqlite-vec virtual table storing 384-dimension embeddings for note chunks. |
tasks | Dedicated task index storing every checkbox line with Kanban and priority metadata. |
links | Source-to-target mapping for backlink and orphan detection. |
non_md_files | Registry of assets (.canvas, images, PDFs) to prevent "broken link" false positives. |
memory_entries | Entry-granular index for About Me/ files, reconciled by content-hash identity. |
Sources: src/vault-mcp/search/search-index.ts322-420 src/vault-mcp/search/search-index.ts221-230
Vault Cortex implements a Kanban-aware task index that parses both Tasks plugin emoji and Dataview inline-field formats using tasks.ts src/vault-mcp/obsidian-markdown/tasks.ts1-20
TaskEntry src/vault-mcp/search/search-index.ts154-178 carries its full context: path, folder, heading (Kanban lane), and line number.kanban-plugin frontmatter to set is_kanban_task src/vault-mcp/search/search-index.ts147 Each task carries the Kanban lane as its heading src/vault-mcp/search/search-helpers.ts155listTasks: Supports structured filters for 6 date fields (due, scheduled, start, created, done, cancelled), priority, folder, tag, and heading. It implements date cascade sorting (fallback through related date fields with per-field direction) and position sorting for Kanban board order src/vault-mcp/search/search-queries.ts360-505Sources: src/vault-mcp/search/search-index.ts126-178 src/vault-mcp/search/search-queries.ts360-505 src/vault-mcp/obsidian-markdown/tasks.ts1-20
search-queries.ts)The search system provides specialized query methods as named exports taking a SearchQueryContext.
fullTextSearch: BM25 keyword search with dynamic SQL filters for folders, tags, and properties src/vault-mcp/search/search-queries.ts152-249hybridSearch: Sequential FTS and Vector KNN fused via RRF, optionally reranked via the cross-encoder src/vault-mcp/search/search-queries.ts251-348memoryRecall: Entry-granular retrieval for the memory layer (powering vault_memory_recall). It uses a combination of FTS search src/vault-mcp/search/search-queries.ts881 and vector KNN src/vault-mcp/search/search-queries.ts887 against the memory_entries table.getBacklinks / getOutgoingLinks: Navigates the link graph. getOutgoingLinks distinguishes between note targets, file targets (tracked via non_md_files), and broken links src/vault-mcp/search/search-queries.ts570-655vaultStats: Aggregates totals for notes, untagged notes, and notes without properties src/vault-mcp/search/search-queries.ts805-832Sources: src/vault-mcp/search/search-queries.ts1-1050
file-watcher.ts)The watcher maintains index parity with the filesystem using chokidar.
pendingEmbeds Map src/vault-mcp/search/file-watcher.ts71 to serialize embedding operations per-path, preventing race conditions during rapid saves.stabilityThreshold src/vault-mcp/search/file-watcher.ts152-153 ensures that multi-chunk writes (common in Obsidian Sync) are complete before indexing.usePolling is enabled (triggered by WINDOWS_MODE), the watcher uses filesystem polling (300ms interval) to propagate events across the Docker ↔ WSL2 bridge src/vault-mcp/search/file-watcher.ts15-19File Watcher Event Logic
Sources: src/vault-mcp/search/file-watcher.ts76-146
sanitizeFtsQuery: Protects FTS5 from syntax errors by escaping reserved characters and stripping operators from user-provided strings src/vault-mcp/search/fts-query.ts10-43buildFtsMetadataText: Flattens frontmatter into a searchable text block for the FTS metadata column, including keys but excluding the title src/vault-mcp/search/search-helpers.ts82-101noteMatchesSearchFilters: A pure TypeScript implementation of the SQL filter logic, used to filter vector-only hits src/vault-mcp/search/search-helpers.ts223-264rowToMetadata, rowToTaskEntry, and noteRowToSearchResult transform raw SQLite rows into typed wire objects src/vault-mcp/search/search-helpers.ts115-194Sources: src/vault-mcp/search/search-helpers.ts1-264 src/vault-mcp/search/fts-query.ts1-43
Refresh this wiki
This wiki was recently refreshed. Please wait 6 days to refresh again.