Member-only story
Structuring 40 Million Documents into an Agentic Knowledge Graph
929 million edges in 10.8 minutes, and 83.2% on 600 held-out questions
Read this story for free: link
A knowledge graph makes the route to an answer visible, and an agent makes that route accurate, because it walks the path, cites it, and refuses when there is none. Handling over 40 million documents is where that breaks, since the usual way to build the graph is to have a model read every document, which is why GraphRAG indexing costs about 33,000 dollars for one corpus. But some corpora already ship their edges, so the graph becomes a parse rather than an extraction, and we get 929,824,202 edges in 10.8 minutes with zero model calls, running an agent we build from scratch in Python.
Here is everything we build, 46 Python modules with no graph database and no extraction model, top to bottom, one component at a time:
- Parse forty million records: six columnar edge tables out of 1,334 gzipped XML files, and not one row inferred by a model.
- Size the corpus before building it: citation retention measured first, because a recent ten million slice keeps only 28.1% of its edges.
- Build the ontology layer: 31,110 MeSH descriptors and 267,012 entry terms, with ancestry as…
