We Tested How Much Context Our Analytics Agent Actually Needs
Same model, same questions, five ways of filling the context window, and what the published research says you’ll find.
I’ve spent three essays arguing that what goes into an AI agent’s context window matters more than which model reads it. I believe that. I also run a company whose product depends on it being true, which is a good reason not to take my word for it.
You don’t have to. Whether your own agent needs more context, less, or different context is a question you can answer on your own data, without a research team. The test is simple: same model, same questions, and only the context changes. Here’s how to set it up, what the research predicts you’ll see, and what to do with each result you might get.
The five setups
Hold everything constant except the context: the same model version and settings, the same system prompt, the same tools and output format. Then run every question through five setups.
- Schema only. Table and column names with types, and nothing else. This is roughly what you get from a tool that points a model at your database and hopes.
- Everything. The full schema, every metric definition, all the documentation you have, and every example query you’ve collected. As much as fits, on the theory that the model will find what it needs.
- Curated. Only the tables relevant to the question, the governed definitions it touches, and a handful of vetted examples, assembled per question. If your product already builds context this way, this is your production setup, which is exactly why it needs testing.
- Curated plus distractors. The curated context with near-duplicate definitions mixed in: an outdated version of each metric used, plus a few synthetic look-alikes. Write these yourself. Never use another customer’s real definitions or data, even in a test.
- Curated, smaller model. The one setup where the model changes. Give the curated context to a smaller, cheaper model and see whether it keeps up.
The questions, and how to score them
The questions matter more than the setups. Use at least 60, ideally 100 or more, taken from what people actually asked your agent rather than questions you wrote yourself. Every one needs an answer your data team has verified. Mix four kinds: simple lookups, metrics with a company-specific definition, questions that join several tables, and deliberately ambiguous questions where the right behavior is to ask for clarification.
Don’t skip the verification, and don’t borrow answers from a public benchmark to save time. When researchers audited two of the most widely used text-to-SQL benchmarks, they found annotation errors in 52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow: wrong reference answers, ambiguous questions and similar problems. Rescoring 16 agents on a corrected subset changed their performance by as much as 31% in relative terms and moved some up or down the leaderboard by nine places. If the answer key is wrong, your test measures the answer key.
Run each question three times per setup, because the same model can answer the same question differently. Compare setups on the same questions, and at this sample size treat a difference of a few points as noise rather than a winner.
Setting temperature to zero doesn’t fix this. Thinking Machines Lab sent the same prompt to a large open model 1,000 times at temperature zero, the setting meant to make output repeatable, and got 80 different completions. They matched word for word for the first 102 tokens, then split: 992 gave Queens as Feynman’s birthplace and 8 said New York City. The cause was how inference servers batch requests together as load changes, which you don’t control when you call a hosted model. In an analytics agent, a split like that can be the difference between two WHERE clauses. One run per question tells you what happened once.
Score results, not SQL. An answer is correct if its result matches the verified one, allowing for row order and rounding. For the ambiguous questions, it’s correct if the agent asks which definition you mean, or states the one it used and that one is right.
Then split every miss into two kinds. Either the agent said it couldn’t answer, or it returned a confident wrong number. That second kind is the one that costs you, because nobody notices it until a decision has been made on it. Track it separately, alongside input tokens, cost and latency for every question. And log the full assembled context for every run, so you can see why each answer came out the way it did.
Don’t expect a better model to bring that number down on its own. Researchers at Meta tested 20 frontier models on questions that were underspecified, built on false premises or simply unanswerable, and found that knowing when not to answer is still an unsolved problem, one that bigger models barely help with. Reasoning models did worse: their reasoning training cut abstention by 24% on average. A carefully written system prompt helped, but didn’t fix it. That makes the ambiguous questions the most important part of your set, and the confident-wrong count the number to watch most closely.
What the research predicts
Published research makes fairly specific predictions about how this should turn out, which gives you something to check your results against.
Schema only should come last. In a 2023 benchmark on enterprise SQL questions, GPT-4 scored 16% working from the raw database and 54% once it was given a knowledge graph of the business’s definitions. A bare schema leaves the model guessing what every metric means.
Get Ka Ling Wu’s stories in your inbox
Join Medium for free to get updates from this writer.
Everything should lose to curated, despite containing strictly more information. Chroma’s context rot study gave models the same questions with either a focused prompt of about 300 tokens or a full history of about 113,000 tokens containing the same answer. Every model family did significantly better on the focused version.
Distractors should cost accuracy. The same study found that even one passage that looks relevant but doesn’t answer the question lowered performance. It also found model families fail differently: some tend to abstain when uncertain, others answer confidently and wrongly. That second pattern is exactly what the confident-wrong count is there to catch.
Position should matter too. The Lost in the Middle study found that models use information best when it sits at the start or end of a long context, and noticeably worse when it’s buried in the middle, even models built for long inputs. That suggests a cheap extra run for the everything setup: once with the relevant definitions near the top, once with them in the middle. If accuracy moves, your production context has an ordering problem as well as a size problem.
The Chroma study turned up something stranger. Across all 18 models, the answer was found more reliably when the surrounding text was shuffled into nonsense than when it read as coherent documents. The researchers don’t claim to know why. That’s a useful reminder that intuition about what a model will find easy is unreliable, which is the whole case for testing.
The last prediction is the least settled, and the one I’d most like to see tested widely: a smaller model with curated context should beat a larger model working from the schema alone. If context really matters more than the model, that’s the result it implies, and it would change how most teams think about cost.
How to read your results
Each pattern you might see points to a different next step.
- Curated beats everything by a wide margin. Your agent is paying for context that makes it worse. Start trimming: cap what retrieval returns, drop stale examples, and audit the system prompt for rules nobody can explain.
- Everything roughly matches curated. Either your model handles long context well at your sizes, or your “everything” isn’t big yet. If accuracy is close, compare cost, because the cheaper setup wins a tie. Then rerun the test as your data grows.
- Distractors hurt, mostly as confident wrong answers. That’s the dangerous pattern. Find where look-alike content gets into real contexts, such as old metric versions, duplicated docs or examples copied between customers, and remove it or version it.
- Schema only isn’t far behind. Your questions may be simpler than you think, or your schema unusually well named. Look at the company-specific metric questions on their own before concluding that context doesn’t matter for you.
- The smaller model keeps up. You may be paying for a bigger model than your context requires. Before switching, compare confident-wrong rates, not just accuracy.
Mistakes that make the numbers lie
A test like this is easy to run badly, and a bad run is worse than none because it looks like evidence.
- Writing the questions yourself. Your questions match how you think about the data, not how users ask about it. Pull them from real logs.
- Grading with the setup visible. If people check any answers by hand, hide which setup produced them. Everyone has a favorite.
- Distractors that are too easy. A look-alike definition that’s obviously wrong tests nothing. The useful ones are close enough that a tired analyst might mix them up.
- Reporting only accuracy. Two setups with the same accuracy can differ a lot in confident wrong answers and cost, and those are often the numbers that decide.
- Treating one result as general. One deployment and one model family tell you about your agent, not every agent. If you change models or your data grows a lot, run it again.
Start with the log
You’ll probably learn something before you score a single answer. The test requires logging the full context for every question, which is worth doing anyway. Most teams that do it for the first time are surprised by what’s in there.
If you want the techniques behind a curated context, I’ve written a longer guide to context engineering.
Everyone has an opinion about how much context an agent needs. A week of testing gets you an answer.
Follow me on Medium and LinkedIn for more on analytics agents, context engineering, and the practical infrastructure behind production-grade AI.
