A closed issue against my slice plots

For readers who tune LightGBM or XGBoost with Optuna and read plot_slice to decide what goes into the search space.

I came across CatBoost issue #1006. In September 2019 a user asked the CatBoost team to add DART, the boosting mode that randomly drops already built trees during training. XGBoost and LightGBM had it, so the request looked natural, though the author doubted that DART leads to better models in general. The maintainers closed it as “won’t fix”. Their reply: “We don’t see the profit from it.” They asked for an example where DART improves quality and the same gain cannot be reached by tuning other parameters.

This surprised me. In my LightGBM work I often check optuna.visualization.plot_slice after a search. When boosting_type is in the search space, DART trials often had a longer lower tail than the gbdt and goss ones, with more trials at low log loss.

Who is right? The CatBoost team gave a clear test, and I took it literally. DART counts as a win only if it beats a second, independent round of tuning of the plain booster. That answers whether DART helps. A second question stayed open: why did my plots say otherwise, and what else can such a plot hide?

On 131 small datasets the short answer is no. Against pooled re-runs of the plain booster, LightGBM’s DART scores 0.0033 lower in mean PR AUC (area under the precision-recall curve) and XGBoost’s 0.0020 lower, so a mean gain of 0.001 is excluded for both. The median shortfall is 0.0002 to 0.0003, so on most datasets DART and plain boosting are level and a few datasets carry the loss. My slice plots were not wrong either. In a learning-rate range that starts at 0.1, DART’s built-in step decay gives it room plain LightGBM lacks, and in almost every study DART won exactly when the sampler had given it the most trials.

The second answer matters beyond DART. A slice plot shows the trials the sampler chose, so it cannot show what each setting can do. Here it hid three things, and I now read the plot as a hint to test, not as evidence.

What DART does to a boosted ensemble

Gradient boosting builds trees one after another, each fixing the errors the earlier trees left. The DART paper notes a side effect: trees added late tend to change the prediction of only a few rows, so the first trees dominate. DART, introduced in 2015 by Rashmi Korlakai Vinayak and Ran Gilad-Bachrach, borrows a trick from neural networks. Before each new tree, it switches off a random part of the existing trees, as dropout does with neurons, and fits the new tree to what the remaining trees still get wrong. The new tree must cover for the missing ones. Then the dropped trees return, and the weights are rescaled so the sum of all trees does not overshoot.

In LightGBM the main controls are:

• drop_rate (default 0.1): the fraction of existing trees dropped in a round.
• max_drop (default 50): the cap on dropped trees per round.
• skip_drop (default 0.5): the probability that a round skips dropout altogether.
• drop_seed (default 4): the seed that picks which trees to drop.

XGBoost has rate_drop, skip_drop, one_drop (always drop at least one tree), sample_type (uniform or weighted choice of trees) and normalize_type (tree or forest rescaling). Two details matter later: the weight of the new tree shrinks as more trees are dropped, and the choice of dropped trees depends on a seed.

Why a benchmark of small datasets

I ran the check on SmallDataBenchmarks, my fork of the Data Cowboys benchmark of classifiers on small tabular datasets. Three reasons.

First, I know it well, and its controls, bugs and noise levels are documented. Second, it is cheap to rerun: a whole booster runs in hours on one laptop, so a methodological mistake can be rerun instead of explained away.

Third, DART is a regularization method, and small data needs regularization most. If dropping trees helps anywhere, it should help on datasets with a few hundred or a few thousand rows.

XGBoost also implements DART, so I added it as a second booster. That broke my second reason, because XGBoost’s DART search took 101.8 hours against 2.4 hours for plain XGBoost.

DART against a second round of tuning, decided before the run

I added DART to both boosters and changed nothing except its own parameters:

• lgbm_dart is LightGBM with boosting_type="dart" and max_drop=-1. The default caps a round at 50 dropped trees, which would flatten drop_rate on larger ensembles.
• xgboost_dart is XGBoost with booster="dart", uniform sampling and tree normalization.
• Both search their library’s usual space plus a drop rate, log-uniform on [0.01, 0.5], and skip_drop, uniform on [0, 0.5]. With this range, dropout runs in at least half of the rounds.

Everything else follows the benchmark’s protocol: nested cross-validation with 4 outer and 4 inner folds, 50 trials per outer fold from Optuna’s TPE sampler (Tree-structured Parzen Estimator), at most 500 trees, a learning rate between 0.001 and 0.3, and selection on PR AUC.

The control decides what counts as a win. A plain booster tuned once is a single draw from a noisy search. So DART faced two controls, the first run of each plain booster and a second run with another sampler seed (the seed control). For LightGBM I added a third run and one with twice the trial budget.

Before the run I fixed the analysis as one family of four tests, DART against each control in each library, with Holm correction (a standard fix for testing several hypotheses at once). “DART beats tuning” needed a significant gain against both controls, plus a larger effect than the seed alone produces. After seeing the results I added bootstrap intervals against the pooled controls, with a margin of 0.001 chosen after the fact. Those were not planned, and I mark which is which.

Neither booster gains, and a gain of 0.001 is excluded

Each DART lost to each of its controls. Mean PR AUC over the datasets each pair shares:

LightGBM has 131 datasets, XGBoost 129: its DART search on the two 100-class plant-species datasets was projected past my 12-hour limit and skipped.

Only the comparisons with the first runs are significant, but that does not mean DART ties the seed control. The gap to it is as large, -0.0035 for LightGBM. Most datasets sit near zero and a few carry the loss, so a rank test has little to work with.

Against the mean of the controls (the pooled analysis, added after these results), LightGBM’s DART is 0.0033 behind, with a 90% bootstrap interval of [-0.0068, -0.0008]. XGBoost’s DART is 0.0020 behind, [-0.0032, -0.0008]. The upper bounds sit below zero, so a mean gain of 0.001 is excluded at one-sided 95% for both libraries. The medians are -0.0003 and -0.0002, so on a typical dataset you will not notice the difference.

Three checks could have rescued DART. None did:

• DART’s own seed. A second LightGBM DART search moves the mean by +0.0017, all of it from one dataset. The two seeds together are still 0.0024 behind, [-0.0043, -0.0008].
• A higher learning-rate cap. DART picked a rate above 0.2 in 21% of folds, plain LightGBM in 7.6%. With the cap raised from 0.3 to 1.0 on 31 datasets, DART scores 0.0076 lower than with the old cap.
• Other outer splits. On 30 datasets under five outer splits, the mean gap moves between -0.0012 and +0.0018 and averages +0.0001. The sign changes with the split, so on a small set of datasets DART and plain LightGBM are within split noise.

Small data was my third reason, so I checked it separately. On the smallest third of the datasets (50 to about 540 rows) LightGBM’s DART is 0.0018 behind the pooled controls, XGBoost’s 0.0024. The seed noise is also largest there: a re-tune alone moves a small dataset’s score with a standard deviation of 0.017 for LightGBM and 0.014 for XGBoost, against 0.003 on the largest third. Where regularization should matter most, the search noise is larger than any gain DART could bring.

Back to the slice plots: a hidden step decay and a sampler that follows the winner

If DART does not win on the benchmark, why did it keep winning my slice plots? My work searches differed in three ways: hundreds of thousands of rows, 50 to 300 trees with no early stopping, and a learning rate searched on [0.1, 1]. The benchmark searched it on [0.001, 0.3]. I checked the difference in a separate notebook, on Bank Marketing (45,211 rows), six more OpenML datasets and synthetic data with label noise.

DART has a built-in learning-rate decay. LightGBM adds each new DART tree with weight learning_rate / (1 + k), where k is the number of trees dropped in that round, and shrinks a dropped tree by k / (k + 1) each time. The trained model shows it: at learning rate 0.5 and 300 trees, the tree weights of a DART model sum to 41.3, against 149.5 for plain LightGBM. DART takes 28% of the step it was given.

So DART needs a higher learning rate to reach its best. In sweeps with 300 trees and everything else fixed, DART’s best learning rate was higher than plain LightGBM’s on all seven real datasets, by a median factor of 3.1 (the weights predict 3.6). Plain LightGBM’s best rate was below 0.1 on all seven. DART’s best log loss was lower on only one of them. The decay moves where the best learning rate sits, not how good the best model is.

A range that starts at 0.1 cuts plain LightGBM off from its best step, and DART is not cut off. Over 36 Optuna studies on Bank Marketing and the synthetic data, DART’s best trial beat plain LightGBM’s in 14 of 18 studies on [0.1, 1] and in 2 of 18 on [0.001, 1]. On synthetic data, DART’s lead on the narrow range grows with label noise in the sweeps, because noise pushes plain LightGBM’s best step further down.

The sampler does the rest. With boosting_type as one categorical parameter, TPE sends most trials to the mode that looks best. In one Bank Marketing study on [0.1, 1], DART got 43 of 60 trials, plain LightGBM 8. Over all 36 studies:

More trials in one mode give that mode a longer lower tail on the slice plot. Equal counts do not remove it: when I draw only as many DART trials as plain LightGBM has, DART keeps its lead in all 14 of the 18 studies on [0.1, 1] where it had it. The count is not the whole story, and the leader’s trials also sit closer to the region the sampler has already found. But allocation follows the outcome, so this table cannot say which causes which. The control is to give each mode its own study, so the sampler cannot move trials between them. On Bank Marketing, over three seeds, DART’s lead on [0.1, 1] then shrinks to 0.0010 in test log loss, averaged over three seeds with a standard deviation of 0.0019. DART wins in two of the three seeds. Three seeds are too few to say more than that the lead is small.

One of my three predictions failed, and I keep it in. I expected DART’s lead on [0.1, 1] to grow with noise in the studies as it does in the sweeps. In 1 of 3 seeds it did not, because plain LightGBM got too few trials near its own optimum. The search can also shorten plain LightGBM’s total step by choosing fewer trees, which the sweeps do not allow.

Three things a slice plot hid here, none visible in the plot itself:

• It shows what the sampler chose, not what each setting can do. Trial counts follow the results, and the leader’s trials are better placed, so the categories are compared on unequal and unequally tuned samples.
• One parameter name can mean two scales. In my runs a DART model took about 28% of the step that plain LightGBM takes at the same learning rate, so a single learning-rate axis carries two different units.
• The search range decides who is in the picture. On [0.1, 1] plain LightGBM never reaches the rates where it does best, and the plot shows no sign that anything is missing.

Both sides were right. My slice plots showed a real effect of the search space and the sampler. The CatBoost team asked for a gain that tuning other parameters cannot reach, and here the gain disappears as soon as plain LightGBM can tune its learning rate down to 0.001.

The benchmark also turned up three smaller costs of DART, and they matter before you ship it.

One fold where the drop pattern decided everything

Half of LightGBM DART’s loss comes from a single fold. On the fourth outer fold of pendigits, a 10-class handwriting dataset, the final DART model predicts class 8 for 91% of the rows. It scores 0.101 PR AUC, chance level for ten classes. Its other folds, and plain LightGBM on every fold, score 0.999. Without pendigits the gap to the first run halves, from -0.0033 to -0.0016.

My first guess was overfitting, and it was wrong: the refit model collapses on its own training rows too, while the same parameters score above 0.998 on all four inner folds.

The cause is the drop pattern. With these parameters (learning rate 0.3, drop_rate 0.033, skip_drop 0.44) the model survives with drop_seed 1 and collapses again with drop_seed 2 and 3. Of seven patterns I tried on the full training rows, three collapse, which overstates the rate because I found the first one by its collapse. LightGBM draws the pattern from drop_seed and the iteration number only, not from the data. So the inner folds replayed the same pattern on three quarters of the rows and survived. Cross-validation could not catch it.

This is rare. Out of 524 outer folds it is the only chance-level one, it disappears with the second DART seed, and XGBoost’s DART has none. But plain boosting does not fail this way. A configuration that passes cross-validation can still break at refit. If you ship LightGBM DART, score the final model on its training rows before you trust it.

DART costs calibration when you select on PR AUC

PR AUC only scores the ranking of predictions, so I also measured calibration on the 108 datasets where it can be estimated. ECE (expected calibration error) is the average gap between predicted confidence and observed accuracy.

Under PR AUC selection LightGBM’s DART raises ECE from 0.067 to 0.106. The rise is heavy-tailed: the median increase is only 0.013. XGBoost’s DART pays less, ECE from 0.063 to 0.074.

My work searches selected on log loss, not PR AUC, so I ran the LightGBM DART search once more with log loss as the objective, same space and budget. Then DART calibrates close to plain LightGBM selected the same way: ECE 0.042 against 0.040, and PR AUC does not change (0.8277 against 0.8278). The Brier score (the mean squared error of predicted probabilities) is still slightly worse, 0.227 against 0.223. The log-loss search also chose less dropout and a higher learning rate, so part of the fix may be that it selects less DART. Why DART calibrates worse under PR AUC selection, I did not find out.

XGBoost’s DART costs at least 43 times the search time

LightGBM’s DART search took 9.27 hours against 3.95 hours for its seed control, 2.35 times as long. XGBoost’s DART took 101.8 hours on 129 datasets against 2.36 hours, 43 times as long. That is a lower bound, since the two skipped datasets were projected to be the slowest. These are recorded times of whole searches on a shared machine, with parallel outer folds and pauses removed.

The ratio grows with the number of classes:

The source code explains the shape. Plain boosting adds each new tree to a running prediction. XGBoost’s DART (Dart::PredictBatchImpl in src/gbm/gbtree.cc, version 3.2.0) rebuilds the prediction every round from every tree that was not dropped, so the cost grows with the trees and with the classes. LightGBM's DART (src/boosting/dart.hpp, version 4.6.0) only subtracts and re-adds the trees it drops. I did not profile the individual terms, so this is consistent with the numbers, not a measured breakdown.

What I do with DART now

The list below covers LightGBM only, since I don’t use XGBoost in practice.

• I no longer put DART in a search space by default. It did not beat a second round of tuning on 131 small datasets, and it costs calibration and time.
• When DART wins a slice plot, I check the learning-rate range first. If the range starts at 0.1 or higher, I widen it to 0.001 on a log scale or run one study per boosting mode with the same range and budget, and I compare refit test scores, not the best trial. I cannot equalize the modes with early stopping, because LightGBM has no early stopping for DART. If DART’s lead survives that, it is worth a closer look.
• If I ship LightGBM DART, I score the final model on its training rows. A configuration that passed cross-validation can still collapse at refit.

This test has limits. It used 50 trials per fold, at most 500 trees, datasets of at most 10,000 rows, selection on PR AUC (log loss only for LightGBM’s DART), one sampler seed for XGBoost’s DART, and five outer splits on only 30 of the 131 datasets.

The slice-plot notebook shows a mechanism on seven real datasets and one synthetic generator, but not how often it decides a real search. The searches in my own work and research projects mostly use far more rows than the benchmark does. DART may still help on data I did not test, and the CatBoost team’s question remains open in the only useful form: “Show me a case where tuning cannot match it”.

Resources

• SmallDataBenchmarks: code, results and the full DART write-up in Findings_notes.md (commit a781b23)
• lightgbm_dart_slice_plot.ipynb: the slice-plot experiments, with cached results (commit f2041a8)
• CatBoost issue #1006: the 2019 request to add DART and the maintainers’ answer
• LightGBM src/boosting/dart.hpp, v4.6.0: the learning_rate / (1 + k) step and the weight normalization
• XGBoost src/gbm/gbtree.cc, v3.2.0: Dart::PredictBatchImpl, the per-round rebuild of the prediction
• LightGBM parameters: the DART parameters and their defaults
• XGBoost parameters: the Dart booster parameters
• Optuna plot_slice
• LightGBM early_stopping callback: no early stopping with boosting_type="dart"
• DART: Dropouts meet Multiple Additive Regression Trees, Rashmi Korlakai Vinayak and Ran Gilad-Bachrach, AISTATS 2015, PMLR 38:489–497: the original DART paper

Why DART wins your Optuna slice plot and loses the benchmark was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

I build tools for writers. Every day, my agentic pipeline scans a few thousand Substack stories and decides which ones deserve attention. Claude, used as the final LLM-as-judge, is costly: up to $3 per batch of 70 stories.

This is why I looked at Jev: a decision model that cannot produce text, but can make decisions, cheaply: $0.042 per million tokens. Could Jev replace Claude? I tried… and failed. But then I saw another possibility: Jev could make Claude’s job easier. Rather than replacing the expensive model, I used Jev to reduce its workload. After a week, I saw a 54% reduction in costs, and significantly better ranking of content.

All that is already in production. Read below.

What my pipeline does

I’ve been working on analytical tools for authors who publish on Substack and Medium. While Medium promotes posts automatically, relying mostly on its discovery algorithm, Substack is more relationship-driven: writers are expected, and encouraged, to engage with each other. So my tools help authors find other, kindred authors to engage with. You will find some details on the mechanics of my scripts in my older stories: “Who wrote similar?” and “Evaluating Retrieval Without Ground Truth”. Since then, the toolset has grown into a number of agentic pipelines working daily over a large database of blogs.

The idea is simple. Take the author’s recent story, search all of Substack for stories with an intriguing similarity to it, and propose them to the author, who may then decide to engage with the other authors. One form of engagement is mutual recommendation, beneficial to both if the audiences overlap. Alternatively, commenting under the post also increases the author’s visibility, especially if his or her writings are related.
So the script crawls the blogosphere and identifies candidate stories, ranked by cosine similarity and a few other factors.

But cosine similarity is a “shallow” metric. It captures shared words and themes, but not well the deep, underlying ideas. Here is an example. One of the blogs the pipeline serves is Nomadic Mind, a travel blog. Its series “Where Europe Still Dances” explores the experience of the traditional French dance balls. On the other end of the blogosphere, another author wrote “Why Making Music Together Feels So Good” — featuring a Chilean teacher performing Depeche Mode for his class. Both stories are kindred, similar in spirit. Their underlying theme is the joy of participating in music together. But the cosine ranker missed that deeper connection and ranked the similarity low. Well, after all, France is so far from Chile…

My architecture so far

This is why the cosine ranker, a bit dumb, is only a preliminary ranking. My pipeline, until now, was:

1. scoring: cosine similarity of embeddings
2. vetting: a cheap model, Gemini 2.5 Flash or Claude Haiku, filters out unwanted topics
3. triage: Claude, expensive, acting as an LLM-as-judge, making the final decision, often overturning the initial judgement.

On the test run, 492 articles were scored. The vetting filtered out most of them, and the remaining 61 went to Claude. Claude triaged them, rejected most and eventually selected 18 fair and 9 highly promising candidates for an engagement. The pipeline works well.

Except it costs $3.02, most of which are Claude tokens.

My pipeline sits on Databricks, ready to be parallelized as multitenant Spark jobs, run for many clients at once, simultaneously, and regularly. There is nothing blocking it technically, except the prohibitive cost of tokens. Too high — by two orders of magnitude.

This is when decision models caught my attention.

Jev, a model that cannot write

While an LLM produces text, a decision model produces, you guessed it, a decision. Conceptually, it’s like a good old classifier: “Look at this picture. Is it a human? Is it a bicycle?” But it’s better: modern decision models are built on transformer architectures, so they accept text as input. Still, the output is not text.

Jev comes from TypeSafe AI, a San Francisco lab founded by Diogo Almeida, a former OpenAI researcher. On 15 September 2026 it was released and marketed as “System One Model”, after System 1, the fast thinking in Daniel Kahneman’s famous book Thinking, Fast and Slow. All of a sudden everyone was talking about it. Vercel added Jev to its AI Gateway on the second day, and reported that nearly 13% of its paid teams tried it within 24 hours. Welcome to the world where things happen faster than we’ve been used to…

Also OpenRouter added Jev to its collection. OpenRouter is a public gateway that provides a uniform interface to many models. So I used Jev through OpenRouter.

How does Jev work in practice? It is simple. In a JSON document, you ask Jev a question, with some context (text). The questions can be of three types:

• Choice: “Is this email intended for sales, invoicing or complaints?”
• Score: “Rate this student between 0 and 100”
• Noul: “Provide probability that a statement is true.”

Here below is an example request I was sending, shortened. In the first question (choice), I am asking to classify the story. In the second (noul), I am asking if the story is politically biased.

{
  "model": "typesafe/jev-1.14",
  "state": "Title: ...\nSubtitle: ...\nBlog: ...\n\n(the first 3,000 characters of the article)",
  "questions": {
    "topic": {
      "type": "choice",
      "instructions": "What is the main topic of the post?",
      "criteria": {
        "travel_places_cultures_abroad": "Travel, specific places, other cultures, or living abroad",
        "partly": "Touches on travel or places but mainly about something else",
        "other": "Not about travel, places, cultures or living abroad"
      }
    },
    "political": {
      "type": "noul",
      "instructions": "The post takes one political side as unquestioned fact, uses derogatory labels for political opponents, or is written for one political camp. A post that discusses politics, policy or political events analytically does not count."
    }
  }
}

And here is the answer:

{
  "topic": { "choice": "partly",
             "probabilities": { "travel_places_cultures_abroad": 0.02, "partly": 0.90, "other": 0.08 } },
  "political": { "noul": 0.92 },
  "usage": { "input_tokens": 1249, "cost": 0.000052 }
}

This approach has tremendous advantages. First, it is fast and cheap: the call above cost five thousandths of a cent. Secondly, it radically simplifies the architecture. Many guardrails become unnecessary: since text is not part of the output, it cannot be toxic, harmful or adversarial. Also, prompt hacking has a smaller effect. An attacker can, at most, influence or bias the output categories.

TypeSafe even claims that Jev “can’t hallucinate”. I don’t know about that. Jev’s architecture has not been published. As Armin Ronacher put it, the design “delegates the hallucination problem a little bit to the user”.

I realized quickly Jev could solve two problems at once: reduce cost and simplify architecture. Could it also improve the results?

Jev judged bias better than the LLM

Now, I need to explain why I need the vetting phase.

In the process of identifying kindred stories, I learned that embedding similarity paints only part of the picture. Especially in the American blogosphere, there are two particularly nasty types of texts that pass unnoticed through the embedding sieve: politically biased and religiously biased ones. They use similar wording and similar themes, yet they are far from “kindred”.

Take France. Nomadic Mind has a story called “Paris: Propriety and Urban Rebellion”, in which its author boards a train to a clandestine anarchist meeting on a Parisian suburb. Its tone is playful, tongue-in-cheek. The embeddings ranker did not get the humor and rated the story closest to “What’s Really Going on in France?”, written by Nick Griffin, the former leader of the British National Party, blaming the recent riots on the far left. One story is sarcastic, the other is political, but they were rated alike for similar vocabulary. The similarity ranking was initially driven by 384-dimensional embeddings of MiniLM over Chroma. Increasing the dimensionality to 768 (replacing MiniLM with all-mpnet-base-v2) did not help either. Possibly, scaling further to deeper models with 3K dimensions would change something, but that is outside my options.

Religion works the same way. Nomadic Mind has a story from West Africa, where its author listens to people who believe in the devil of the forest. The story landed close to various obscure conspiracy theories explaining how the magical operations of freemasonry hidden lodges lead to control of the Global Elite. Both stories speak of the devils, but the authors are very unlikely to want to engage mutually: the first author looks at the phenomenon as folklore, the other one honestly believes in the paranormal and takes religious stance.

This is where the vetting comes in. In the second phase of the pipeline, I used a cheap model, Gemini or Haiku to filter out various unwanted features of the candidate stories. Among them, importantly, I wanted the politically and religiously biased texts out.

Could the vetting model be replaced by Jev?

I sent the same 492 articles to Jev asking him to do the same job. To my disappointment, it flagged only 16 articles as political, and 5 as religious (as opposed to Gemini, which filtered out over 100 for bias). I thought Jev was wrong, so I reviewed what was happening, and it turned out that the initial sieve had been wrong! Gemini (or Claude Haiku) classified stories as “political” when they just spoke about politics. Jev was sharper for the price: it distinguished political topic from political bias. It wasn’t perfect, but it was better: when I manually ranked stories for political bias, it agreed with me 21 out of 24 times.

All in all, the effect was bizarre: even though Jev was generally more effective, we ended up filtering fewer stories, not more. Not good! Our goal was to filter early as much as possible, so that the LLM phase would be cheaper. So I looked at the more interesting part: could Jev replace the LLM in the third phase (triage)?

Jev for triage

Triage (phase 3) was the most expensive phase. Cutting the cost of this phase would have the highest effect.

I first tried to bluntly replace the LLM-as-Judge by Jev. This failed. Claude understood the context and nuance better. On the sample of 52 stories a human ranked top (most aligned with target), Claude agreed with 40, while Jev agreed with 17. But this gave me an idea. Seventeen, which comes almost free, is already something. Could I use Jev to better rank stories, so Claude could just read the top of the list and burn fewer tokens?

Here, I used Jev’s Noul-type query. The goal was to understand whether a candidate article had an interesting angle relative to the author’s story. So I asked Jev a simple question: “Would the author of story A be interested in commenting on article B?” A Noul yes/no answer carried probability, which I later interpreted as score. It could be more natural to use Jev’s native Score-type query, but somehow I was unsuccessful in it. The requests were blocked, probably at OpenRouter’s API level. Whatever. Probability of True was good enough.

Jev’s scores were promising, but not great. It ranked somewhat better than embeddings, but not well enough to replace Claude. Initially, the numbers looked poor, confirmed by this visualization. Both rankers correlated poorly with each other, and also with human judgement. If they did, the blue dots on the diagram below should sit in the top right corner, where they are not.

Two rankers, two kinds of kinship

But a seasoned data scientist learns to distrust numbers. I dived deeper and saw something intriguing: the two rankers ranked different kinds of articles high.

The embeddings found kinship in vocabulary, but this included location and subject. “The women banished as witches in West Africa” scored 0.80 against “How the Witch Saved Aisha’s Baby”, Nomadic Mind’s story also set in West Africa. Jev instead found kinship in ideas. He correctly decided that an Irish music session in a Dublin pub was related to a sagra (village festival) in Puglia, southern Italy.

Once I understood this, the solution was straightforward.

I tweaked some things, and hacked an algorithm that blended both rankers (score fusion) — copying the idea of machine learning models ensemble: “can a set of weak classifiers create, as an ensemble, a powerful classifier?” Yes it can:

The new architecture

So we have built the final architecture:

1. Discovery. The script finds candidate stories, as before
2. Vetting. Jev filters out biased posts and other unwanted categories
3. Ranker. The embeddings and Jev, blended, tell Claude what to read first
4. LLM-as-judge, only on the top of the list. Claude is still there, but he reads only very few articles, so he is much cheaper.

Performance? Twice as good as before. At the top of the list, Claude used to find 11 of the 52 best stories. Now he finds 24.

Cost? $3.02 dropped to $1.39. Claude no longer loads a list of 60 to 90 articles into its context, but reads only the very best candidates and stops. This is still too expensive, but it defines an avenue for further tuning and cost saving. I have a number of ideas how. Hint: Jev’s part of today’s $1.39 cost is $0.02 (yes, two cents)

Architecture? It is not only simplified (less worry on LLM guardrails), but also it became ready to scale over Databricks, enabling multitenancy over massively parallel runs that scale not only technically, but also financially.

Are decision models a game changer?

Yes. For bounded questions, I believe they are. And, as this project shows, many problems can be redefined as such.

Jev could not replace Claude as the judge. But on bias, it was better than the LLM, and as a ranker it let Claude read much less. Per article, it was 130 to 180 times cheaper. The cheap layer already costs almost nothing: what remains is Claude’s reading. Jev writes no text, so fewer things can go wrong and output processing is simpler, although its answers can still be wrong. This fits a pattern that is becoming common in AI engineering: a cascade, or model routing, where cheap “System One” models make the fast decisions, and an expensive LLM is called only sparingly.

Jev is not alone. Since it launched, several alternatives appeared. For instance, Remigiusz Kinas published Basal, a small model (1.5 billion parameters), fine-tuned from the Polish model Bielik, and free under the Apache 2.0 license. In its author’s benchmark, it beats Jev. Then came Cloudflare’s Clef, OpenAI’s Decisions API, and Fastino’s GLiNER2.5-Decide. I tested only Jev, not because I found it superior, but because I wanted to test the concept: decision model instead of LLM-as-judge.

Decision models such as these, together with sentence-transformers and a vector database such as my Chroma, are the base for the cheap layer. Then, the LLM-as-judge reads only what is worth reading.

Thank you for reading. Do share your take.

I Tried to Replace Claude With Jev, and Kept Both was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

How a deterministic shell, a small LangGraph agent core, and two separate “judges” exposed what a perfect answer-key score could not.

In this article, I’ll share the approach I took during the project organized by Great Learning in collaboration with the University of Texas at Austin and the McCombs School of Business to solve a very common supply chain problem: the last-mile logistics delivery.

A practical case study in multi-agentic-AI orchestration, progressive and parallel task execution in ambiguous contexts, LLM as a judge, and rigorous evaluation methodology.

1. The Problem

In last-mile logistics, roughly 10% of all shipments encounter delivery exceptions — failed attempts, address mismatches, damaged packages, refused deliveries, and weather delays. Each failed delivery costs in direct reattempt expenses, while the downstream impact is far steeper: consumers stop shopping with a retailer after just 2–3 failed deliveries, and supply chain disruptions cost companies millions per year.

Despite these costs, most organizations still handle exception triage and resolution manually. Operations staff read through status logs and messy driver notes, cross-reference customer profiles and internal playbooks, decide on a resolution action, and draft customer notifications — all under time pressure. Teams rely on support tickets, tribal knowledge, and rigid rule engines that cannot handle the nuanced, multi-factor judgment calls that real exceptions demand. A VIP customer with a perishable package and a broken gate code on a second attempt requires a fundamentally different resolution than a standard customer’s first failed attempt on a non-perishable item — but manual processes treat them with the same slow, inconsistent workflow.

Key KPIs affected include:

- Exception resolution time (constrained by manual triage throughput)
- Escalation accuracy (missed or unnecessary supervisor escalations)
- Customer communication quality and personalization
- Cost per exception (reattempt costs, spoilage write-offs, unnecessary truck rolls)
- Customer retention and lifetime value

If unaddressed, logistics providers face rising exception-handling costs, inconsistent service quality, preventable customer churn, and loss of competitive ground to players who are already investing in AI-powered exception detection and response automation.

By applying an AI-powered multi-agent system that ingests delivery logs, cross-references customer profiles and locker availability, retrieves resolution policy from an operational playbook, and generates validated decisions with personalized customer communications, logistics providers can automate the full exception-handling pipeline from detection through resolution, with built-in quality validation and supervisor escalation where policy requires it.

2. Technology Stack

The project was developed in VS Code as a modular Python package (src/), using OpenAI APIs, GitHub for version control, LangGraph for agent orchestration, LangChain for LLM integration, Pydantic for structured output contracts, PyPDFLoader for PDF ingestion, SQLite for data storage, and LangSmith for tracing agent behavior and monitoring drift. This architecture was a deliberate choice: it provided a cohesive foundation for development and helped keep agent interactions grounded, observable, and controlled.

Splitting generation and review across two models is a design choice, not a measured optimum. There was no cost or model-ablation experiment.

The two models are created once and reused:

from langchain_openai import ChatOpenAI
 
# Cheaper, faster: Resolution and Communication agents
gen_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
 
# Stronger reasoning: both Critic nodes and the judges
eval_llm = ChatOpenAI(model="gpt-4o", temperature=0)

3. Deterministic Logic vs. Probabilistic Reasoning: Knowing When to Use Each

3.1 A Mixed Challenge: delivery exceptions need judgement and deterministic rules

A delivery exception is any event that needs a decision: a failed attempt, an address that cannot be found, a damaged parcel, a refused delivery, a weather delay.

For each one, an operator has to decide:

- Is it really an exception? Routine scans and successful deliveries are noise.
-What is the resolution? One of RESCHEDULE, REROUTE_TO_LOCKER, REPLACE, or RETURN_TO_SENDER.
-Does a supervisor need to be involved?
-What should the customer be told, and in what tone?

The inputs were three types of data:

A separate ground-truth file holds the reference answer for each shipment: whether it is an exception, the expected resolution, the expected tone, and whether it should escalate. It is an evaluation reference only and is never shown to any agent.

This prototype evaluated only 10 shipments that are small but deliberately varied:

3.2 The Approach: use an LLM only where there is no single right answer

The design principle is stated in the notebook’s plan: be deterministic where there is one right answer, and use retrieval and LLMs where judgement over policy text is needed.

That produces four layers:

1. A rule-based preprocessor. Deduplication, consolidation, prompt-injection scanning, and a noise filter. No LLM.

2. Deterministic tools. Locker eligibility and an escalation rule engine. No LLM.

3. A small multi-agent core. A Resolution agent and a Communication agent, each reviewed by a stronger Critic.

4. An evaluation layer kept separate from the agents. Answer-key checks, a coherence judge, and a playbook-adherence judge.

3.3 The Deterministic Layer: where correctness is cheap

If a decision has a single, explicitly defined outcome, it belongs in code — not in an LLM prompt. This principle was fundamental to the robustness of the pipeline. The key lies in establishing a clear distinction between qualitative inference and deterministic operations.

Interpreting ambiguous situations, extracting meaning from unstructured information, and reasoning through conflicting evidence are sophisticated tasks that may justify the flexibility — and computational expense — of an LLM.

Conversely, arithmetic calculations, threshold evaluations, conditional logic, score-based outputs, and rule-based validations are better handled by conventional code: faster, cheaper, more predictable, and easier to audit.

The architectural mistake is treating every decision as a reasoning problem when many are simply execution problems. Too many organizations blur this distinction, introducing unnecessary complexity, latency, and cost into their agentic workflows — and then struggle to understand why those systems fail to deliver consistent, scalable performance.

And yes, now you know why your evaluation score keeps changing every time you run the same pipeline.

Back to our case, the locker eligibility is arithmetic, not opinion. For illustration purposes, this simple logical function can solve the headache.

@tool
def check_locker_availability(zip_code: str, package_size: str) -> list[dict]:
    """Find compatible lockers in the same zip code, with eligibility reasoning.
    (abridged)"""
    size_hierarchy = {"SMALL": 1, "MEDIUM": 2, "LARGE": 3}
    pkg_level = size_hierarchy.get(package_size, 0)
    # ... query lockers in this zip code, then for each one:
    for row in rows:
        locker = dict(row)
        locker_max = size_hierarchy.get(locker["max_package_size"], 0)
        if locker_max < pkg_level:
            locker["eligible"], locker["reason"] = False, "Locker max size too small"
        elif locker["capacity_status"] == "FULL":
            locker["eligible"], locker["reason"] = False, "Locker is FULL"
        elif locker["capacity_status"] == "LIMITED" and package_size != "SMALL":
            locker["eligible"], locker["reason"] = False, "Locker is LIMITED - SMALL only"
        else:
            locker["eligible"], locker["reason"] = True, "Compatible"

Following the same line of thought, escalation is a rule engine, split into AUTOMATIC and DISCRETIONARY

@tool
def check_escalation_rules(customer_tier, exceptions_last_90d, attempt_number,
                           package_type, status_code, status_description) -> dict:
    """Deterministic escalation rule engine (abridged)."""
    triggers = []
    if attempt_number >= 3:
        triggers.append("AUTOMATIC: 3rd failed delivery attempt")
    if customer_tier == "VIP" and exceptions_last_90d >= 3:
        triggers.append("AUTOMATIC: VIP customer with 3+ exceptions in 90d")
    if status_code == "DAMAGED" and package_type == "PERISHABLE":
        triggers.append("AUTOMATIC: Damaged perishable package")
    # Weather delay over 4 hours for a perishable: parsed from free text like "5hr"
    # Discretionary: a Standard customer with more than 5 exceptions in 90 days
    if customer_tier == "STANDARD" and exceptions_last_90d > 5:
        triggers.append("DISCRETIONARY: Standard customer with >5 exceptions in 90d")

The engine also derives a triage priority (AUTOMATIC → P2-HIGH, discretionary only → P3-ELEVATED, none → P4-ROUTINE, with a perishable moved up one level to P1-CRITICAL). Priority only orders the queue; it never changes the resolution or the escalation decision.

Run over the ten shipments with no LLM calls, it produced this queue:

Guardrails run BEFORE any LLM call. The preprocessor scans driver notes (and later the retrieved playbook chunks) for prompt-injection keywords, and filters routine status codes with no anomaly words as noise:

def check_noise_override(consolidated: dict) -> bool:
    """Flag routine status codes with no anomaly indicators."""
    routine_codes = {"DELIVERED", "IN_TRANSIT", "OUT_FOR_DELIVERY", "SCANNED"}
    if consolidated["status_code"] not in routine_codes:
        return False
    anomaly_indicators = ["damage", "wrong", "suspicious", "overdue", "missing",
                          "lost", "stolen", "leak", "delay", "late", "fraud"]  # abridged
    desc = consolidated["status_description"].lower()
    return not any(word in desc for word in anomaly_indicators)

Both noise shipments (SHP-001, SHP-010) closed in about zero seconds with no agent calls. The injection guardrail never blocked a shipment in this dataset (all ten showed Guardrail: CLEAR), so its blocking path was not exercised by these cases.

3.4 The Multi-Agent Core: typed outputs and a rule-based orchestrator

3.4.1 Schema-enforced decisions

The Resolution agent must return a typed object. A model validator rejects internally inconsistent answers:

class ResolutionOutput(BaseModel):
    is_exception: Literal["YES", "NO"]
    resolution: Literal["RESCHEDULE", "REROUTE_TO_LOCKER", "REPLACE",
                        "RETURN_TO_SENDER", "N/A"]
    rationale: str
 
    @model_validator(mode="after")
    def validate_consistency(self):
        if self.is_exception == "YES" and self.resolution == "N/A":
            raise ValueError("resolution cannot be N/A when is_exception is YES")
        if self.is_exception == "NO" and self.resolution != "N/A":
            raise ValueError("resolution must be N/A when is_exception is NO")
        return self

3.4.2 The orchestrator is not an LLM

A LangGraph conditional edge reads next_agent from state, and every worker returns to the orchestrator:

workflow = StateGraph(UnifiedAgentState)
workflow.add_node("preprocessor", preprocessor_node)
workflow.add_node("orchestrator", orchestrator_node)
workflow.add_node("resolution_agent", resolution_agent_node)
workflow.add_node("critic_resolution", critic_resolution_node)
workflow.add_node("communication_agent", communication_agent_node)
workflow.add_node("critic_communication", critic_communication_node)
workflow.add_node("finalize", finalize_node)
 
workflow.add_edge("preprocessor", "orchestrator")
ROUTES = ["resolution_agent", "critic_resolution", "communication_agent",
          "critic_communication", "finalize"]
workflow.add_conditional_edges("orchestrator", lambda s: s["next_agent"],
                               {r: r for r in ROUTES})
for worker in ["resolution_agent", "critic_resolution",
               "communication_agent", "critic_communication"]:
    workflow.add_edge(worker, "orchestrator")
workflow.add_edge("finalize", END)

Its routing order is fixed and documented: guardrail, noise, resolution, critic, capped revision loop, enforce AUTOMATIC escalation triggers, communication, communication critic, finalize. Because the rule engine is authoritative, an LLM cannot talk its way out of a mandatory escalation.

3.4.3 The Critic reports playbook alignment on every review

The Resolution Critic returns a typed verdict that includes how the decision compares with the retrieved playbook text:

class CriticResolutionOutput(BaseModel):
    decision: Literal["ACCEPT", "ESCALATE", "REVISE"]
    playbook_alignment: Literal["ALIGNED", "ALIGNED_WITH_NOTE",
                                "CONFLICT", "NO_RULE_FOUND"]
    playbook_rule_cited: str   # "[Page N] ..." or empty
    alignment_note: str
    rationale: str

The Critic is told it may only ask for one of the four valid resolution values, and that a perishable must never be handled as routine.

4. Evaluation methodology

Every shipment is scored on five metrics, plus a sixth added in version 3. We will explain the 3 main iterations performed to fine-tune the model later.

The new check (not in the original plan) is the interesting one. Its verdicts are ALIGNED, ALIGNED_WITH_NOTE, CONFLICT, and NO_RULE_FOUND.

def compute_playbook_adherence(pred: dict) -> dict:
    """Check the final decision against the retrieved playbook text.
    The ground truth is never passed in."""
    audit = {
        "consolidated_event": pred.get("consolidated_event", {}),
        "customer_tier": pred.get("customer_profile", {}).get("tier"),
        "locker_availability": pred.get("locker_availability", []),
        "escalation_signals": pred.get("escalation_signals", {}),
        "retrieved_playbook_sections": [
            f"[Page {c['page']}] {c['content']}" for c in pred.get("playbook_context", [])
        ],
        "final_resolution": pred["resolution_output"].get("resolution"),
        "final_rationale": pred["resolution_output"].get("rationale"),
        "escalated": pred.get("escalated"),
    }
    # eval_llm returns {"verdict", "rule", "justification"}

Aggregates then report two new rows: the adherence rate (aligned, with or without a note) and the count of shipments that pass the answer key but conflict with the playbook.

5. Three iterations, and what each one taught

No weights changed. Three versions of the notebook changed schemas, prompts, and how results were scored. The headline metrics from each executed run:

Iteration 1: The agent invented a fifth action

In v1 the output schema accepted a fifth resolution, HOLD_AT_DEPOT, because the playbook’s third-attempt and address-not-found paths describe holding a parcel at the depot. The agent used it for SHP-003 and SHP-005. Against the original answer key, both failed: task completion was 8/10.

The v1 notebook also scored against a second, playbook-reconciled answer key, in which those two shipments were relabeled HOLD_AT_DEPOT. Against that key v1 scored 10/10. That is a flattering number, and it is the pattern this article warns about: when you reconcile the answer key to the system’s behaviour, the system always passes.

Iteration 2: Tighten the contract, keep the original labels

v2 made two changes that are visible in the code diff:

· HOLD_AT_DEPOT was removed from the resolution schema, leaving the four valid actions plus N/A.

· The reconciled-label column was removed, so scoring returned to the original answer key. The system prompts for the Resolution agent, Communication agent, and both Critics were rewritten as new cells.

Task completion rose to 10/10 against the original labels. But escalation accuracy fell to 88% (7/8), because of one shipment.

What happened on SHP-003 (address not found): the Resolution agent proposed RESCHEDULE, the right answer. The Critic, reading a playbook that says to hold at the depot and verify the address, sent it back to revise, twice. On the retry the model returned N/A for an exception, which the Pydantic validator rejected as invalid. After three failed attempts the pipeline hit its safe fallback: default to RESCHEDULE with forced escalation. The final resolution was correct, but the escalation was wrong, the run took 18.91 s, and it used two revisions.

That is a good failure. The guardrail worked as designed, the audit trail explained everything, and a human would have been asked to look at a case the answer key says needs no escalation. It was a prompt-contract problem, not a model problem.

Iteration 3: Reconcile in the prompts, and make conflicts visible

v3 changed prompts and the evaluation layer only. Architecture, tools, data, and labels were unchanged. There were two kinds of change.

A. Reconcile the playbook and the schema in the prompts. The Resolution agent and the Critic are told that “hold at the depot” and “contact the customer” are the first part of RESCHEDULE, not a separate action, and that the Critic may only request one of the four valid values:

- ADDRESS_ISSUE (building or street not found, missing unit): RESCHEDULE.
  The playbook's hold-at-depot and contact-the-customer steps are the first part
  of this resolution, not a different one: say in the rationale that the package
  waits at the depot until the customer verifies or corrects the address, then
  is redelivered.
 
The resolution field accepts only RESCHEDULE, REROUTE_TO_LOCKER, REPLACE,
RETURN_TO_SENDER or N/A. Playbook wording such as "hold at the depot" describes
an interim step: write it in the rationale.

The retry logic was also changed so a failed attempt is told what was invalid, instead of repeating an identical prompt:

# abridged: last_error starts as "" before the loop
for attempt in range(max_retries):
    # After a validation failure, tell the model what was wrong
    retry_note = (
        f"\n\nYOUR PREVIOUS OUTPUT WAS INVALID ({last_error}). Return exactly one of "
        f"RESCHEDULE, REROUTE_TO_LOCKER, REPLACE or RETURN_TO_SENDER for an exception, "
        f"and put any interim step in the rationale."
        if last_error else ""
    )
    try:
        result = structured_llm.invoke([
            SystemMessage(content=system_prompt + retry_note),
            HumanMessage(content=user_content),
        ])
        break
    except Exception as e:
        last_error = str(e)[:150]

B. Make conflicts visible. The Critic’s playbook_alignment and the new playbook-adherence judge were added, and the aggregate table gained the two new rows.

Results. SHP-003 was accepted on the first review, not escalated, and took 7.89 s instead of 18.91 s. Escalation accuracy recovered to 8/8, mean latency fell from 7.76 s to 7.19 s, and zero revisions were needed across all ten shipments. The labels, the data, and the answer key were unchanged, so this is a prompt effect.

6. The AHA moment — What the evaluation can see now: the answer key versus the playbook.

Here is the finding that shaped the rest of this article.

Every answer-key metric is green for SHP-002 and SHP-005. But the playbook judge, which never sees the answer key, flagged both as CONFLICT.

The rollup: playbook adherence 62% (5 of 8 exception shipments aligned, with or without a note), and 2 shipments pass the answer key but conflict with the playbook (SHP-002 and SHP-005).

Two checks that agree on SHP-005 and SHP-003 but disagree on SHP-002 and SHP-008 are not two verdicts. They are a signal for a human reviewer. Neither the Critic nor the Judge is ground truth, and the judge sees only retrieved chunks, so NO_RULE_FOUND can mean retrieval missed a clause rather than that no rule exists.

7. Lessons Learned

Anthropic Research recently highlighted why agent evaluations are essential — not merely as instruments for measuring quality and performance, but also as powerful mechanisms for discovery, continuous improvement, and challenging our own operational assumptions.

In their article, Demystifying evals for AI agents, the researchers illustrate this point with a particularly interesting example:

“Agent evaluations are even more complex. Agents use tools across many turns, modifying state in the environment and adapting as they go — which means mistakes can propagate and compound. Frontier models can also find creative solutions that surpass the limits of static evals. For instance, Opus 4.5 solved a τ2-bench problem about booking a flight by discovering a loophole in the policy. It ‘failed’ the evaluation as written, but actually came up with a better solution for the user.”

Sound familiar? That’s precisely the kind of problem our case study uncovered.

In our shipment exception workflow, the agents produced resolutions that initially failed against the predefined ground-truth labels. Yet, when we examined their decisions against the actual exception resolution playbook, we discovered something more interesting: the agents had correctly prioritized the policy’s immediate operational requirements over the expected outcomes encoded in the evaluation dataset.

The issue wasn’t necessarily that the agents were wrong — or even that the policy itself was flawed. Rather, the evaluation exposed a disconnect between what the ground truth expected, what the policy required, and when a particular action was permissible.

And that’s where evaluations become far more valuable than a simple PASS or FAIL.

These discrepancies provide an opportunity to refine ground-truth definitions, clarify policy prerequisites, strengthen enforcement mechanisms, and improve operational coverage for those pesky, persistent edge cases that rarely reveal themselves under ideal conditions.

There’s an interesting parallel with the legal world. Legislators cannot anticipate every factual scenario, interpretation, or unintended consequence that may emerge when a law is applied in practice. Legal frameworks therefore evolve through judicial interpretation, amendments, and the resolution of previously unforeseen cases.

Corporate policies are no different. They are written under assumptions about how the real world operates, and those assumptions are not always complete.

The difference today is that well-designed agentic systems, combined with rigorous evaluations, can help organizations uncover those hidden inconsistencies systematically — before they propagate into operational failures.

The real value of an agentic evaluation framework, therefore, isn’t just determining whether an AI agent made the right decision. It’s also helping us discover whether we defined the right decision correctly in the first place.

8. What worked, and what to treat with caution.

What worked

· The deterministic layer carried most of the safety. Deduplication, the noise guardrail, locker eligibility, and the escalation rule engine closed both noise cases and forced five of the six correct escalations without relying on an LLM. The sixth, SHP-008, rests on a discretionary trigger, so the Critic had to choose to escalate. It did.

· Perishables were never held or sent to a locker. The triage queue put SHP-007 and SHP-009 first.

· No cell raised an error, and the injection guardrail never blocked a shipment.

· Structured outputs plus bounded retries turned a model error into a controlled fallback (SHP-003 in v2), with a full audit trail rather than a silent wrong answer.

What to treat with caution

· Ten shipments, one run per version. Version comparisons are between single runs.

· LLM judges grade LLM output. The Critics, the coherence judge, and the playbook judge are all LLMs. The coherence judge gave 5/5 to every shipment in v2 and v3, so it is not discriminating much and cannot guide improvement on its own.

· Tool-call accuracy is a log check. It confirms the required tools and agents ran, not that their output was good.

· The judge sees only retrieved chunks, so NO_RULE_FOUND is ambiguous.

· The Critic’s full rationale is not printed. Only its alignment fields are shown.

· The answer key may itself be debatable. SHP-005 shows that the labelled “correct” action and a literal reading of the playbook can disagree. That is a question for the policy owner, not for a prompt.

9. LangSmith: traceability today, drift detection as the next step

Seven node functions (preprocessor_node, orchestrator_node, resolution_agent_node, critic_resolution_node, communication_agent_node, critic_communication_node, and finalize_node) are decorated with @traceable, and tracing is configured from the project’s settings file. Each run therefore yields a trace of which node ran, in what order, with what inputs and outputs. That is what made the SHP-003 failure easy to diagnose: the retries, the invalid N/A, and the forced escalation are all visible as steps rather than inferred.

from langsmith import traceable
 
@traceable(name="critic_resolution_node")
def critic_resolution_node(state: UnifiedAgentState) -> UnifiedAgentState:
    """Critic Agent: validates the resolution decision against the playbook."""
    result = eval_llm.with_structured_output(CriticResolutionOutput).invoke([...])
    state["critic_resolution_output"] = result.model_dump()
    state["trajectory_log"].append(
        f"critic_resolution: decision={result.decision}, "
        f"playbook_alignment={result.playbook_alignment}"
    )
    return state

Tracing is the foundation. It is not drift detection by itself. The notebook does not implement versioned cohorts, alerts, or a regression gate.

To turn traces into drift traceability, I would add:

1. Version metadata on every run: code revision, prompt version, playbook-PDF hash, model and embedding versions, dataset version. LangSmith supports run metadata and tags for this.

2. Signals tied to this system’s failure modes:

· Playbook drift: a changed PDF changes the retrieved text, so re-baseline the labels and the judge.

· Retrieval drift: a rising share of NO_RULE_FOUND verdicts.

· Critic-judge disagreement rate, the measurable version of the SHP-002/SHP-008 gap.

· Retry and forced-escalation rate, which would have flagged the SHP-003 problem in v2.

· Escalation-rate shift and latency percentiles.

3. A frozen regression set, with repeated runs per version and human review of every CONFLICT.

4. Governance: redact customer names and unnecessary personal data from traces, control access, and set a retention period. Logging every prompt is not automatically an acceptable audit policy.

An illustrative sketch of the metadata (not implemented in the notebook):

import hashlib, os
from pathlib import Path
 
playbook_hash = hashlib.sha256(
    Path("exception_resolution_playbook.pdf").read_bytes()
).hexdigest()
 
run_config = {
    "tags": ["delivery-exceptions", "regression"],
    "metadata": {
        "revision_id": os.environ["GIT_SHA"],     # supplied by your build
        "prompt_version": "v3",
        "playbook_sha256": playbook_hash,
        "dataset_version": "capstone-10",
    },
}

10. Recommendations

These are recommendations. None has been run.

1. Ask the policy owner to resolve SHP-005. Should a third attempt with a full locker reroute, hold, or escalate? Change the answer key or the playbook, not the prompt.

2. Grow the evaluation set well beyond ten shipments, including more third-attempt, locker-constrained, and ambiguous-playbook cases.

3. Repeat each case several times per version, and report spread, not a single run.

4. Make the judges more discriminating. A coherence score that is always 5 cannot guide improvement. Add human-reviewed samples.

5. Give the playbook judge full-page context for a fair NO_RULE_FOUND, or separately score retrieval recall.

6. Add a deterministic playbook-rule check for the cases where the playbook states a literal rule, so conflicts are caught without an LLM.

7. Implement the observability loop above before any production use.

11. Conclusion: measure what the answer key cannot

Every number below comes from the saved outputs of the executed notebooks (v1, v2, v3), not from a new experiment. Temperature is 0, but LLM behaviour can still vary between runs, and the critics and judges are themselves LLMs. No model weights were fine-tuned. All “tuning” described is prompt, schema, and evaluation-layer engineering. Comparisons between versions are between single runs: evidence, not a guarantee.

The system’s final scorecard is excellent on the metrics it was built to optimise:

100% task completion,

100% escalation accuracy,

100% tool-call accuracy,

5.00 / 5 coherence,

7.19 s mean latency.

The most useful line in the report is 62% playbook adherence with 2 answer-key passes that conflict with the playbook, because it is the only line that could tell us something we did not already believe.

Three lessons carry over to any agentic system:

1. Put rules in code and judgement in models. The deterministic layer, not an LLM, carried most of the safety.

2. A reconciled answer key will always flatter you. Keep the original labels, and add an independent check that does not see them.

3. Make disagreement visible. When two checks differ, that is a review queue, not a verdict.

References

· LangSmith observability

· LangSmith evaluation

· Trace metadata and tags

Let’s Connect

LinkedIn

GitHub

My Multi-Agentic framework scored 100% on Accuracy Task Completion — It Was Not the Most Important… was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

In Fitting a Stronger Model Into Less Memory, two quantized weights flipped a toy business from retail to repair, and nothing about the arithmetic was wrong. Rounding did exactly what rounding does. That example was narrow on purpose: small enough to check by hand, so the flip couldn’t be waved away as a measurement artifact.

Here, the same question gets a sharper stand-in. Instead of a business classifier, a two-choice scorer meant to resemble a decision an analytics agent actually makes, such as picking which denominator a retention query should use. The mechanics are the same scale, rounding, and clipping as last time. What changes is what’s riding on the answer.

My first instinct when learning quantization was to keep reducing the scale. If the values are closer together, the rounding error should get smaller. I was thinking of a scale of 0.01, but with only four codes that leaves us with 0, 0.01, 0.02, and 0.03. We have better spacing and almost no range. A value like 0.5 no longer fits.

That was the part I needed to work through. The number of bits limits the choices we can store. The scale decides how far apart those choices sit, and values beyond the range have to be clipped. Making the scale smaller does not give us more choices.

For a model, the next question is what those rounding errors do to the calculation. With an analytics agent, I’m interested in whether they change a choice that reaches the query or the final answer. The model can still produce readable text after choosing a different metric definition.

A small calculation makes this easier to inspect. The example below is a constructed two-output scorer, evaluated in NumPy, rather than a measurement from an LLM. It has two choices, A and B, with these weights and one shared input. Multiplying each row by the input gives A a score of 0.662 and B a score of 0.639: A wins by 0.023.

Now quantize the weights using four-bit integer codes from −7 to 7, leaving −8 unused. Use zero as the zero-point, round-to-nearest-even, and one scale for the whole matrix. The largest absolute weight is 2.8, so the scale is 2.8 / 7 = 0.4. Divide each weight by 0.4, round it to a code, then multiply by 0.4 to reconstruct it. A now scores 0.720 and B scores 0.760. The choice has switched to B. Nothing was clipped.

The 2.8 is worth looking at here. Its input is zero, so it contributes nothing to this particular answer, but it still sets the scale for the other weights. Those smaller weights lose the differences that previously gave A the higher score.

We can give the first three weights in each row their own scale and put the final weight in a separate group. Doing that restores A on this input and reduces the average squared weight error by about 30 times. These tiny groups illustrate the calculation; production groups also have to account for kernel and storage overhead.

That fixes the input we started with. Changing the third input from 0.1 to a variable t shows what happens elsewhere. Keep the other inputs fixed at [1, 0.2, t, 0]. The original score difference, A minus B, is 0.028 − 0.05t. It changes sign at t = 0.56.

Global INT4 chooses B for every positive t. Split INT4 keeps choosing A across the plotted range. INT8 moves the boundary to about 0.3333. So the grouping that fixed our first input still fails elsewhere, despite its smaller weight error. These eight constructed weights show how a decision can change; they cannot estimate an actual agent’s failure rate.

For a fixed input x, a weight change ΔW changes the output by Δy = ΔW x. The input determines how much the rounding matters. In a full model, earlier layers can also change the inputs to later ones.

For paid-account retention, accounts that became paid after the opening snapshot should not enter the opening cohort. Choosing the wrong denominator can still produce executable SQL and a plausible percentage. Quantization could change that choice, but this numerical example does not establish that it happened in a real agent.

I’d test the same checkpoint at BF16, W8A16, and W4A16, keeping the instructions, tools, data, activation precision, and attention-cache precision fixed. W4A16 means four-bit weights and sixteen-bit activations. Replay identical prefixes and tool results to locate differences, then let each version complete the task with the same retry budget.

Grade against reviewed answers and executed queries; BF16 can be wrong too. Include user corrections and missing evidence. Keep those tests separate from calibration data, which should reflect the schemas and tool responses the agent handles.

Then measure memory and end-to-end latency. Scales and the attention cache add memory beyond the weights. Kernel performance, tool waits, and extra retries determine whether the smaller model actually saves the user time.

For this agent, I’d keep the retention question in the release test. I want to see the right denominator survive the change in precision, alongside a measured improvement in memory or latency.

Notes

A follow-up to: Fitting a Stronger Model Into Less Memory: A Quantization Walkthrough.

The scorer, weights, and input sweep here are constructed for illustration, independently re-derived and verified: the score computations, the per-group rescaling, the ~30× error reduction, and the INT8 crossing point at t ≈ 0.333 all check out exactly against hand and NumPy calculation.

W4A16 / W8A16 notation and the BF16/INT8/INT4 release-testing approach follow common LLM-serving practice (four- or eight-bit weights, sixteen-bit activations).

Quantizing an Analytics Agent Without Losing the Decision was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.