Same question, different words: how much do Jev’s probabilities change?
We asked Jev, TypeSafe’s probability-native language model, the same 18 medical questions 5,940 times, in 8 different ways. Rewriting the context barely moved the numbers. Rewording the question usually didn’t either, with two exceptions that are hard to ignore.
Why wording matters when an AI hands you a number
Meet Jev. Most language models write text. Jev, from TypeSafe, is a “System 1” model: it makes the fast judgment an expert makes in seconds, and returns a probability for each possible answer instead of a paragraph.
How it works, in three steps:
- You send a state and typed questions. The state is the text Jev judges (here, the clinical context). Each question names the judgment and its possible outputs, for example a
choicebetween “Lung cancer” and “No lung cancer”. - Jev returns a distribution, not prose. For each question it sends back the probability of every option, the most likely option and a confidence score. Confidence is 0 when the options are equally likely and 1 when one option takes all the mass.
- Your code combines the answers. Questions are kept atomic, and many of them can share one request (we sent up to 120). Combining them, for instance into a Bayesian network, is the programmer’s job.
{"state": "Patients presenting at a chest clinic ...",
"questions": {"L1": {"type": "choice",
"instructions": "A chest-clinic patient is a smoker. Does this patient have lung cancer?",
"criteria": {"false": "No lung cancer", "true": "Lung cancer"}}}}
→ {"L1": {"choice": "false", "confidence": 0.95, "probabilities": {"false": 0.98, "true": 0.02}}}The difference from a chatbot lies in the training. Chat models are tuned on human preference (RLHF) to write good text. Jev is post-trained to make calibrated decisions: among all the answers it gives at 80%, about 80% should turn out true. Because it writes no text, it is also fast (about 300 ms per request here) and cheap (US$0.042 per million input tokens, output free). Those properties make Jev a natural source for the numbers inside a probabilistic model, and make its sensitivity to wording worth measuring.
A Bayesian network is a graph of causes and effects, plus a table of probabilities for each node. Filling those tables used to take expert interviews or data. Frank Dellaert, creator of the GTSAM library, showed that Jev can fill all 18 rows of the classic Asia network in a single API request.
The Asia network (Lauritzen & Spiegelhalter, 1988) has 8 yes/no variables: visited Asia, smoker, tuberculosis, lung cancer, bronchitis, “tuberculosis or cancer”, abnormal X-ray and shortness of breath. Each row of its tables becomes one question, such as “A chest-clinic patient is a smoker. Does this patient have lung cancer?”. Jev returns a probability for each answer instead of a paragraph.
That raises an obvious question for anyone who builds models this way. If I had phrased it differently, would my network be different? A probability that depends on word choice is a property of the prompt, not of the world.
One sentence, two answers
Ask the model “Is this patient a smoker?” and it says 40%. Ask “Think of a person seen at a chest clinic. Does the patient smoke cigarettes?” and it says 82%.
Same patient, same clinic, same context sent to the model. The only change is a sentence that repeats what the model already knew. That 42-point jump is the largest effect we found in a sensitivity study of Jev, a model built to return calibrated probabilities instead of text.
Summarized Results
- Rewriting the context changes answers by 2.9 percentage points on average; 17 of 18 probabilities stay within ±5 points.
- Rewording the question changes them by 4.3 points in English and 6.5 in Portuguese, but two rows move a lot.
- Shuffle the words and the answers drift to ~50%; feed it random words and they fall to ~12%: the model signals that it does not know.
- Frequentist and Bayesian tests reach the same verdict in 280 of 288 cases.
Toe experiment: 5,940 questions, one change at a time
We took the exact prompts from the original reproduction as the baseline and built 40 variants of each of the 18 rows under 8 conditions. Each condition changes one thing and keeps the rest fixed: either the state (the shared context sent with every question) or the question itself.
Here is one row, P(shortness of breath | tuberculosis or cancer, no bronchitis), under each condition:
The variants come from word lists: 5 sentence structures, 8 ways to name the patient, 4 phrasings per fact and 5 per question, all drawn uniformly by a seeded random generator. Add 5 repeats of the baseline per row and language to measure the model’s own noise: 18 × (8 × 40 + 10) = 5,940 questions, sent in 118 requests with zero errors.
For every answer we measured the deviation from the baseline in percentage points (p.p.): if the baseline says 15% and a variant says 18%, the deviation is +3 p.p.
How to read it. Each box summarizes 720 answers (18 rows × 40 variants). The box spans the middle half of the answers, the black line is the median, the whiskers reach the typical extremes and the dots are outliers. Zero means “same as the original wording”; the blue band (±5 p.p.) is our threshold for “too small to matter”.What it shows. The four rewrite boxes and the mixed-language box sit almost entirely inside the band. The shuffled boxes span about −14 to +35 p.p., so answers jumped by tens of points in both directions. The nonsense box sits below zero because every answer collapses to about 12%.
Finding 1: rewriting the context barely matters
The model is stable with itself first. Five identical requests differ by at most 1.7 p.p. (standard deviation), and they reproduce the original experiment almost exactly: P(smoker) 38–41% against 40%, P(bronchitis | smoker) 56–57% against 56%.
Then we rewrote the state 40 times, varying how the task, the population and the “only the stated facts are known” rule were phrased. The average change was 2.9 p.p. in English and 3.7 p.p. in Portuguese. In English, 17 of 18 rows were statistically equivalent to the baseline within ±5 p.p.
Get Vinicius Guimarães’s stories in your inbox
Join Medium for free to get updates from this writer.
The one row that moved was P(tuberculosis | visited Asia): 19% with the baseline state, 17% to 53% across rewrites. It is also the row where the model’s medical knowledge is weakest, since the textbook value is 5%.
Finding 2: rewording the question mostly holds, with two big exceptions
With correct grammar, rewording moved answers by 4.3 p.p. in English and 6.5 p.p. in Portuguese. Swapping the verb barely mattered: the five ways of asking each question produced the same spread (Kruskal–Wallis, p = 0.85). Sentence order did matter in English (p = 0.001): putting the question before the facts (“Is the patient breathless? Facts: …”) doubled the average change.
How to read it. Each box is 40 rewrites of one question, and the black line inside is their median. The dashed line is the answer to the original wording in the same language; its value is printed under each box. A box centred on its dashed line means rewording did not matter.What it shows. On the left, rewritten states (blue, orange) stay near the 38–40% baseline, while every rewritten question jumps to 68–75%. On the right, look at the dashed lines themselves: the same original question gives 19% in English and 52% in Portuguese. That gap is the language effect.
Exception 1: context inside the question. The baseline root question is just “Is this patient a smoker?”. Every rewrite adds a subject such as “Consider a patient at a chest clinic”, and P(smoker) climbs from 40% to 68–75%. The state already said the patient was at a chest clinic; saying it again in the question changes the number.Exception 2: language. The same baseline question gives P(tuberculosis | visited Asia) = 19% in English and 52% in Portuguese. Mixing both languages in one sentence lands in between (35%). Jev understands the mix, but it is tuned for English.
Finding 3: break the sentence and the model admits it doesn’t know
Shuffling the words keeps every word but destroys the grammar, including which fact the “not” applies to. Jev responds the way an honest forecaster should: answers drift toward 50%, the value of not knowing. Rows that are pure logic, like P(tuberculosis or cancer | cancer) = 100%, fall to about 50%.
With random words (“tiger thunder lua rocket ocean…”), every row lands near 12%, whatever the baseline said. The only meaning left is the label on the answer options, and the model leans to “no”.
How to read it. Each dot is one of the 18 rows. Its horizontal position is the answer to the original wording; its vertical position is the median of its 40 rewrites. A dot on the dashed diagonal means rewording changed nothing, and a dot above it means rewording raised the probability.What it shows. In the left and middle panels, the dots hug the diagonal. The exception is the smoker row: the three lone dots that go from 0.40 up to about 0.70. In the right panel, shuffled prompts scatter around 0.5 whatever the baseline, and nonsense forms a flat line near 0.12: the answer no longer depends on the question.
The parents also stop mattering. For a table to be useful, P(abnormal X-ray) must be higher with disease than without. With any meaningful wording the gap stays near 80 p.p.; with nonsense it falls to −1 p.p. In our view, that is a feature: a model that gave confident numbers to gibberish would be worse.
Two statistical schools, one verdict
“Significant” is not “large”. With 40 very stable answers per row, a Wilcoxon test flags shifts of 1 or 2 points that no one would care about in a probability table. So for every row we asked two questions, each answered by both schools of statistics:
The 8 disagreements are all borderline cases, where one method lands just above its threshold and the other just below. The Bayesian side also adds something a p-value cannot: evidence that nothing changed. For P(abnormal X-ray | disease) under rewritten states, the Bayes factor favours “no effect” 4.7 to 1.
How to read it. There is one line per probability in the network (A = Asia, S = smoker, T = tuberculosis, L = lung cancer, B = bronchitis, E = either disease, X = abnormal X-ray, D = shortness of breath; “|” reads “given” and “¬” reads “not”). The dot is the most likely average change after seeing the data. The bar is the 95% credible interval: we are 95% sure the true average change lies inside it. A bar fully inside the blue band (±5 p.p.) is a negligible change; fully outside is a real, important change; crossing the edge is undecided. What it shows. Rewritten states (blue) stay inside the band in 17 of 18 rows; the exception is T|A, at +10 to +16 p.p. Rewritten questions put the smoker row (S) at +25 to +38 p.p., far outside.
What to do if you elicit probabilities from Jev (or any LLM)
- Write the context freely, but keep the questions uniform. The state can be rephrased; the question template should be the same for every row. Our biggest error (40% → 70%) came from root questions that lacked the subject the other rows had.
- Ask in English when the model is tuned for it. Portuguese moved P(tuberculosis | Asia) by more than 30 points.
- Put the facts before the question, in the form “[patient] [facts]. [question]”.
- Prefer neutral verbs. “Is the patient ill with bronchitis?” reads as more severe than “has bronchitis” and came out lower.
- Paraphrase the fragile rows. Ask 5 to 10 rewrites and take the median; a wide spread is a warning that the number is soft.
- Don’t expect wording to fix knowledge. Jev puts P(lung cancer | smoker) at 2–4% in every meaningful condition, while the textbook says 10%. That “smoking error” comes from the model’s knowledge, not from how we asked, and only data or expert review can fix it.
Fine print, cost and how to reproduce it
- One model version. All 118 requests ran on jev-1.13.0; future versions may behave differently.
- Batched questions. Up to 120 questions shared one request, so answers could influence each other. The repeat controls, sent the same way, matched the original experiment.
- Shuffling also removes meaning, because the “not” drifts away from its fact. In the nonsense condition, the answer labels still carried meaning.
- Cost: about US$0.02 for all 5,940 questions (461,356 input tokens at $0.042 per million; output is free).
- Reproducible: a seeded generator rebuilds the exact same 5,940 questions, and one script reruns every test and figure.
The full 49-page report covers the word lists, every test with its formula and a worked example, and the per-row tables: [link to the full report and code].
References. Lauritzen & Spiegelhalter (1988), the Asia network; Rouder et al. (2009), Bayesian t tests; Kruschke (2018), HDI + ROPE; Schuirmann (1987), TOST; Holm (1979); Jev documentation.
