You Changed the Model. What Else Changed?
How to compare old and new models when someone has also rebuilt the search index.
A replacement model gets a lower score on the same test set. Before anyone starts rewriting the prompt, I would ask what else changed between the two evaluations.
Consider a fictional technical-support assistant that answers questions from equipment manuals. The team switches models while another team rebuilds the search index, adding manuals for newer equipment and changing the passage boundaries. The evaluation questions stay untouched. Answers about maintenance procedures get worse.
That result gives the team a reason to investigate. It gives them very little basis for blaming the replacement model. The answer may have been generated from a different manual, or from a passage that now stops before an important qualification.
What stayed the same?
An evaluation file can be unchanged while the workload it exercises has moved on. The questions are only the starting point. An assistant may retrieve passages, look up reference tables and call tools before producing its answer.
I would want a record of the source documents, index configuration, filters and reference tables used in the test. I would also preserve the prompt, tool definitions and relevant preprocessing code. For calls into a changing external service, captured responses can support an offline replay, provided the test does not need to reproduce the service’s live behaviour.
Microsoft’s migration guidance recommends holding inputs, expected answers and success criteria fixed, then replaying the existing workload on the replacement before tuning it. For a retrieval-based assistant, applying that advice requires attention to the material fetched during execution as well as the questions in the test file.
There is a deadline to this work if the old model is being withdrawn. Run and save the baseline while access remains. An old score without its inputs and outputs is a poor substitute. Once the model is unavailable, you cannot generate a fresh result from it to fill a gap in the comparison.
The baseline also needs to meet an agreed standard. Reproducing an old mistake is not a successful migration. For a maintenance question, the reference should identify the relevant equipment revision and the required steps. The old model’s answer is useful evidence about past behaviour; it is not automatically the correct answer.
Three configurations, with a narrower question for each
For the support assistant, I would start with three configurations. This is a comparison plan, not a claim that three individual calls are enough.
A uses the current model with the preserved documents and index. This is the baseline.
Get Minchan Chung’s stories in your inbox
Join Medium for free to get updates from this writer.
B uses the replacement model with that same data setup. Keep the application prompt, tools and retrieval configuration unchanged where compatible. Record any required adapter or parameter changes; if those also change behaviour, the comparison is between configurations rather than an isolated model swap.
C keeps the replacement setup from B and introduces the rebuilt index. B to C measures what that index change does with the replacement in place.
Repeat each configuration enough to understand ordinary variation, and compare the same cases using the same criteria and evaluator. If model-based grading is involved, preserve that grader’s configuration too. A small movement in the average may disappear across repetitions; a recurring failure on one equipment family may remain worth fixing.
If B behaves similarly to A but C repeatedly misses the same instruction, inspect the passages returned by the rebuilt index first. If B already fails with the old index, the replacement configuration needs attention. Save those cases before tuning anything; they give the next experiment a stable starting point.
The limits matter. A to B estimates the model change under the old data setup. B to C estimates the data change under the new model. They do not establish that the data change would affect every model equally. Testing the old model with the new index, when possible, adds the missing comparison and helps reveal an interaction.
The broader migration practice has a useful precedent. In a 2021 engineering article, DoorDash described moving risky components sequentially and checking metrics after each change. That was an ML service migration, not an LLM experiment, but the sequencing makes a regression easier to locate.
The same index does not guarantee the same evidence
An agent may write its own search queries. Change the model and it may ask the same index a different question, retrieving different passages. That can be part of the model change’s effect on the complete workflow. It does not automatically mean someone changed the underlying data.
Saving the retrieved passages helps separate these questions. Feed both models identical saved passages and you can examine answer generation with the evidence held fixed. Let each model retrieve through the same tools and you are testing the wider workflow. Both are useful; the result should say which experiment you ran.
Some migrations do not permit a neat intermediate configuration. Replacing an embedding model may require rebuilding the index. Keep the source documents and evaluation cases fixed where possible, but assess that migration as a combined retrieval-system change. A better final answer alone cannot tell you how much came from the embedding model and how much from the rebuilt index.
If the previous index was overwritten and cannot be recovered, record that limitation. Historical traces may still help explain individual failures. They do not recreate an end-to-end baseline that no longer exists.
Before changing the prompt
I would take one failed maintenance question and put the two execution records beside each other: model version, application configuration, index version, retrieved passages and final answer. Then I would run the missing comparison while the old setup is still available. That gives the engineer a specific next experiment, before an afternoon of prompt edits makes the original difference harder to recover.
I’m Minchan Chung, co-founder of CUBIG. I write about AI-ready data and what I’m learning while building the company. New essays every Tuesday and Thursday. Follow me on Medium to read the next one.
