We gave our AI coding agent a CodeGraph. Then we took it away
A five-task trial on a 900-file Python app: equally correct answers, no measurable speed-up, a higher bill, and a blind spot right where we needed help.
Code-graph tools for AI coding agents make an appealing promise. Parse the whole repository once, store every symbol and call edge in a local database, and let the agent ask “who calls this?” instead of grepping its way through the code. Fewer tool calls, fewer tokens, faster answers.
We tested that promise with codegraph 1.6.2, an MCP server for Claude Code and other agents, on our own production codebase. We expected to keep it. We removed it two days later. This is what we measured, and why we think a code graph is not the fix for the problem it targets — at least not on code shaped like ours.
A note before you read on. This is a one-case evaluation of our own repository, not a review of codegraph. Your codebase may get a very different result. But that is the point: test any code-graph tool on your own repository before you adopt it. On ours it added overhead and cost, and gave nothing in return.
The test
The codebase is a Flask app that turns PDF into manuals, 897 Python files, about 424,000 lines, 516 of them test files. It leans on LangGraph pipelines, registries and string-keyed dispatch, which is ordinary for a Python app that grew fast.
We picked five tasks from our own recent work. Four were structural: an impact question (“what breaks if this function gets a new required argument?”), a refactor across 18 call sites, a “which production code reaches this function?” trace, and a config-to-client trace. The fifth, explaining a status rule, was a control.
Each task ran as a fresh headless Claude Code session (Opus) in its own copy of the repo, twice: once with the codegraph server and a fresh index, once with neither. We graded every answer against a ground truth worked out by hand, and graded the refactor by running the full test suite against an unmodified copy.
One accident turned out useful. Our first round prepared the “on” copies with codegraph index, which silently does nothing until codegraph init has run. So both arms ran without an index, and that round became a same-versus-same baseline: it shows how much two identical runs differ by chance.
What it got right
On ordinary calls, codegraph was exact. Before the speed runs we checked its callers answers against grep plus reading the code:
It also kept up with edits. A new file was indexed in about 2 seconds, and an edit showed up in about 1 second. LangGraph wiring was fine too, because the node function is passed as a reference that a parser can follow.
The last row is where the trouble starts.
Where it went blind
Our app has 45 learning-object types rendered by 31 modules. None of them is called by name. A dispatcher looks the type up in a table of module paths, imports the module with importlib, and calls its render attribute:
module_name = _GENERATORS.get(lo.type)
mod = importlib.import_module(module_name)
renderer_fn = getattr(mod, "render")No static parser can follow that hop, and codegraph did not. Ask it who calls a renderer and it lists the tests. Ask what the dispatcher calls and no renderer appears. So for the 31 modules that produce everything our users see, “what breaks if I change this?” comes back with the production path missing.
That would be a known limitation and nothing worse, except for what the tool tells the agent. codegraph’s MCP instructions say, in effect: trust the results and don’t re-verify them with grep. In our trial the agent with codegraph got the dispatch question right only because it ignored that advice: after one codegraph call it ran 11 greps and read the dispatcher itself. An agent that did as it was told would have answered wrongly, and confidently.
This is the core problem. The dynamic seams in a codebase — registries, plugin loaders, string dispatch, framework routing — are exactly where agents get lost and exactly where a static graph cannot see. The graph is most reliable where help is least needed.
What it cost
The agent with codegraph cost more on four of five tasks, and was not measurably faster.
Most of the extra cost comes from a prompt hook the installer adds. On every prompt in an indexed repo, it injects about 30 KB of graph context before the agent’s first turn. Our “on” runs paid 47,000–64,000 tokens of cache writes, against 15,000–32,000 without. On two of the tasks, the agent never called the codegraph tool at all; it just paid for the context.
Get Fernando Dougnac’s stories in your inbox
Join Medium for free to get updates from this writer.
Speed showed no clear win. In the same-versus-same round, two identical setups differed by up to 28 seconds on one task. Every time difference in the real round fell inside that range, except one: an agent without codegraph spent 1,167 seconds on the refactor, because it chose to run the entire test suite. That was a test-selection choice, not navigation.
Correctness was equal. Both arms answered all four questions correctly, and both refactors left the test suite exactly where it started.
Setup surprises
None of these is a dealbreaker alone, but together they show how much a code graph touches beyond the code.
- It installs wide. The install script also ran
codegraph installwith targetauto, which wired codegraph into every agent it detected: eight on our machine, including Codex, Gemini, VS Code Copilot and JetBrains. Each got a user-level MCP entry. Claude Code also got an auto-allow rule, the per-prompt hook and a block in the globalCLAUDE.md. Usage telemetry was on. - The index lives in your project. It is always
.codegraph/at the repo root (about 112 MB for us). Our repo sits in Dropbox, so we had to mark the folder ignored before the first build, and add it to.gitignore,.dockerignoreand our deploy script's rsync excludes. - It inherits your
.gitignore. Our.gitignorehas atest_*.pypattern, and the test files are force-added to git. The default index therefore skipped 509 test files, socallersnever showed a test. Acodegraph.jsonwith anincludelist fixed it; the similarly namedincludeIgnoredoption does something else. - Freshness has gaps. Right after a rename,
callersreturned zero for about 4 seconds; after a delete, search still found the deleted function for about 5 seconds. An agent that queries straight after its own edit can get a wrong answer. - Processes linger. Our ten headless sessions left 39
codegraph serveprocesses running after they ended.
To its credit, codegraph uninstall was clean. It removed every agent's entry and deleted a config file only when codegraph's block was all that was in it.
The bug was in our search, not our map
The most useful thing the trial found had nothing to do with codegraph. The same .gitignore pattern that hid our tests from the index also hid them from the agent's ordinary search.
In Claude Code’s shell, grep is a wrapper around ugrep with --ignore-files, and the built-in Grep tool also respects .gitignore. A recursive search from the repo root silently skips files that git tracks but .gitignore matches. In our repo that is all 509 tests. During the trial, an agent without codegraph stated that "no test calls" a function that two tests call. It wasn't careless; its search never looked there.
No code graph fixes that, because the graph made the same mistake. Searching with command grep -r, or fixing the ignore pattern, does.
So is a code graph ever worth it?
Possibly, on different code. A large, statically typed codebase with explicit imports, where the expensive questions really are “who calls this?”, gives a parser far more to work with than a Python app built on registries. A graph also helps a tool that cannot grep at all.
Be honest about what our numbers can carry. That’s one repository, five tasks and one run per arm. The differences in time are within the noise we measured, and a bigger trial could move the cost figures either way. What a bigger trial would not change is the blind spot: string-table dispatch is invisible to a static index, whatever the sample size.
So, again: this is our case, not a verdict on the tool. Run the same test on your repository before you adopt it.
Our takeaways, if you are weighing one of these tools:
- Test it on your dynamic seams first. Pick the function your agents most often get wrong, and check
callersagainst a hand trace before trusting anything else. - Budget for the context it injects. A per-prompt hook is a cost on every message, used or not.
- Check what your agent’s search can actually see. A tracked file your search skips will cost you more wrong answers than any missing graph.
- Read the installer before you run it. Know which agents and files it will change.
Modern coding agents already search and read code well. The places they still fail are the places no static map covers, so a better index of the easy parts isn’t the fix.
