You Don’t Need Jev
Cheap LLM labels, 220 hand-labeled examples, and a 150M-parameter model on a CPU can match or beat a hosted decision model at a fraction of the cost and latency
Many products need fast, typed judgments about text. Which team should handle this ticket? Is this message spam? Is this prompt trying to jailbreak my agent? How similar are these two sentences?
Decision models sell this as an API. TypeSafe’s Jev is a good example. You send text and a question, and it answers in one of three forms:
- Noul: is this true? It returns a probability.
- Choice: which of these options? It returns a distribution over the options.
- Score: where does this fall on a scale? It returns a distribution over the levels and their expected value.
Jev’s pitch rests on four claims: it is accurate without any training, its probabilities are calibrated, it is fast, and it is cheap.
I wanted to know whether you could get all four from a model you own: trained in an afternoon for a few dollars, running on your own CPU. So I tested it on Jev’s own benchmark, on exactly the test examples Jev was scored on.
Short answer: yes.
- Accuracy: 8 wins, 4 ties, and 0 losses across 12 tasks.
- Calibration: better calibrated than Jev on 12 of 14 questions.
- Latency: 12–30 ms median on 4 CPU threads, against Jev’s 70–500 ms.
- Cost: 12–84× cheaper per decision. Each model cost $0.15–$2.57 to build, every failed experiment included.
The code, the paper, and every experiment, failures included, are open source: https://github.com/erfaneshrati/ownyourjev.
The recipe
1. Write the question exactly as you would for Jev. The tool uses the same spec format.
2. Hand-label about 220 examples: 20 “seeds” to show the LLM what you mean and 200 “dev” examples to make every decision on.
3. Have a cheap LLM label your unlabeled text, guided by short written guidelines (more on these below). A flash-class model labels 10k texts for roughly $0.10–$0.30.
4. Fine-tune a 150M-parameter encoder (gte-modernbert or UltraBERT) on those labels.
5. Calibrate on your 200 dev examples, export to int8 ONNX, and serve it behind a Jev-compatible endpoint. Switching from Jev is a URL change.
Accuracy: 8 wins, 4 ties, no losses
The 12 tasks cover spam, paraphrase, prompt injection, toxicity, intent routing (Banking77, CLINC150), news topics, financial sentiment, emotion, sentence similarity, movie-review sentiment and implicit hate speech. “Win” means our whole 95% confidence interval lies above Jev.
The biggest margins came where Jev’s zero-shot reading of the question misses the dataset’s labeling convention:
- Emotion: 0.811 vs Jev’s 0.585
- Prompt Injections: 0.957 vs 0.741
- Banking77: 0.861 vs 0.797
- Financial PhraseBank: 0.794 vs 0.731
- CLINC150: 0.934 vs 0.895
- PAWS: 0.884 vs 0.850
This is the core advantage of owning the model. Jev reads your question the way it reads everyone’s, while your model learns your convention.
Get Erfan Eshratifar’s stories in your inbox
Join Medium for free to get updates from this writer.
The four ties are STS-B, SST-5, ToxicChat (toxic), and ToxiGen. In none of the 12 tasks is Jev significantly better.
Calibration: better on 12 of 14 questions
A probability is only useful if 0.8 means “right about 80% of the time”. Jev sells calibrated answers, so this had to be tested too.
I fitted a calibrator on the same 200 dev examples: one temperature parameter for Choice and Score questions, and a monotone map for yes/no questions. The calibrator can never change an answer, only its confidence. Results on the test sets:
- Expected calibration error was lower than Jev’s on 12 of 14 tasks, with a median of 0.027 vs 0.086.
- Log-loss was lower on 12 of 14, and Brier score on 10 of 14.
- Jev is badly overconfident on some tasks. Its error is 0.279 on Emotion and 0.169 on Prompt Injections, against 0.026 and 0.022 for ours.
Jev is still better calibrated on a couple of questions: ToxiGen’s toxic head, and CLINC150, where our out-of-scope rule reports low confidence on answers that are mostly correct.
Speed and cost
- Latency: 12–30 ms median and at most 68 ms at p95, per request, on 4 CPU threads. Jev’s published end-to-end latency is 70–500 ms.
- Running cost: $0.52–$1.34 per million decisions, assuming $0.04 per vCPU-hour, against $13–$61 for Jev.
- Build cost: $0.15–$2.57 of LLM calls per task, $12.53 for all 12, counting every failed experiment. Owning the model pays off after 4k–181k decisions.
There’s also the part that doesn’t fit in a table: no API key, no rate limits, no outages, no silent model updates, and your data never leaves your machine.
What actually mattered
I logged 318 experiments. These are the lessons that held up.
1. The labeling prompt mattered more than the model.
Label your dev set with the LLM and read its mistakes. Write down your conventions as short guidelines, then check them on the half of dev you didn’t read. This raised the LLM’s agreement with human labels by 6–30 points. On Emotion it went from 0.49 to 0.785, because the LLM had to learn the dataset’s rule: the label is the emotion word attached to “feel”, even when negated. Stop when agreement stops rising, because that number predicts your ceiling.
2. Cheap labelers with good guidelines rarely lost to frontier labelers. Spend on guidelines, not on a bigger labeler.
3. Use your whole unlabeled pool. Labeling all 15k CLINC150 utterances instead of an early 4.5k sample was the main change behind accuracy rising from 0.846 to 0.934.
4. Synthetic data only fills gaps. LLM-written out-of-scope utterances lifted CLINC150’s out-of-scope recall from 0.81 to 0.935. Synthetic data that didn’t look like the real inputs hurt.
5. Bigger models didn’t help under a 70 ms CPU budget. ModernBERT-large didn’t beat the 150M model. Averaging three fine-tunes that start from the same initialization (a “weight soup”) removed most seed noise at no inference cost.
6. Keep the right layers in fp32. Full int8 quantization destroyed one backbone. Keeping only the MLP output projections in fp32 recovered fp32 accuracy at near-int8 speed.
7. Evaluate the way you serve. Dynamic int8 quantization makes a text’s output depend on which other texts share its batch. Re-scoring every model one request at a time, the way it’s served, changed no verdict, but it flipped up to 9.5% of decisions on one task.
How I tried not to fool myself
Beating a benchmark is easy if you peek at the test set, so the protocol was strict:
- A fixed human-label budget: 220 examples per task, all drawn from the training split.
- Every decision made on dev: prompts, models, hyperparameters, thresholds and calibrators.
- Audited test access: only audited scripts read test labels, and every read is logged. The whole study made 14 test scorings, and all of them are reported.
- No calls to Jev: it is scored from its published per-example responses (Deußer et al., 2026), on exactly the same examples.
- Everything in the repo: every experiment, including failures, is in a per-task ledger.
Limits. Four tasks are ties, not wins. Jev is still better calibrated on a few questions. Reading comprehension and judging generated text (BoolQ, SummEval, HelpSteer2) are future work. And you do need those 220 human labels: a practitioner can make them in an afternoon, but this is not zero-shot.
Try it
For the decisions most teams actually need (routing, topic, sentiment, spam, prompt injection, paraphrase, similarity), you don’t need a hosted decision model. A cheap LLM, an afternoon of labeling, and a small model on a CPU will match or beat it, and the model is yours.
