How AI decides what to say next
How Does ChatGPT Decide What to Say Next?
Ai can seem magical at times, always knowing exactly what to say, but it isnt a tiny brain powering the LLMs we use daily, they roll weighted dice — one at a time. Here’s how that works.
TL;DR
- Large language models (LLMs) like ChatGPT come up with text by predicting the next small chunk of text (a token), repeatedly to produce an outcome.
- For each one of these tokens, the model gives each possible outcome a probability; then it picks one at random. The outcome is affected by these probabilities but does not decide it. This is why there for the same prompt you can get different probabilities.
- Tempertaure and other settings can control how different the probabilities it produces are.
Example
Finish this sentence:
The cat sat on the ___
You probably said “mat”. Maybe “floor” or “sofa”. You probably wouldn’t think of “volcano”.
That’s basically what ChatGPT does all day.
When I first used ChatGPT, I thought it was searching the internet or looking at answers in some giant database. It isn’t. Every answer it gives is built one small piece (or token) at a time, and for each piece it’s doing the same thing you just did — what’s most likely to come next?
Here’s the best part. You didn’t just pick one word. In your head there was a ranking: “mat” felt really likely, “floor” a bit less, and “volcano” basically impossible. That ranking is a probability distribution, and it’s the core principle behind every LLM you’ve used.
In this article I’ll go through how that works, step by step:
- How text gets chopped into tokens
- How the model turns what it’s read into probabilities
- How it picks the next token, and why it doesn’t always pick the most likely one
- Where the probabilities come from
Then I’ll talk about why this matters, including why AI can sometimes make things up
Step 1: Chop the text into tokens
A language model doesn’t read letters or words the way we do. It reads tokens, which are chunks of text that the ai splits inputs into. GPT-2, the model I used to test everything in this article, has exactly 50,257 of them [6].
Most common words are usually split into one token. Rarer words can be split into a few pieces. I ran a few through GPT-2’s tokenizer to see:
- “The cat sat on the mat.” → 7 tokens: The cat sat on the mat .
- “unbelievably” → 4 tokens: un bel iev ably
- “Supercalifragilisticexpialidocious” → 11 tokens
The space is part of the token: “ cat” with a space in front is a different token from “cat” at the start of a sentence. On one paragraph tested, it used about 4.4 characters per token, so a token is a bit less than a word.
When I say the model predicts the “next word”, it’s really trying to guess the next token out of that list of 50,257.
Step 2: Turn the text into probabilities
Next, the model reads all the tokens so far and gives a score to every single one of its 50,257 tokens. This is the part done by the huge neural network (the transformer), and it’s where all the “intelligence” is. A high score means something fits really well there, and a low score means that it makes no sense.
The scores returned aren’t probabilities yet, so the model uses a formula called softmax that turns them into percentages that add up to 100%.
I ran GPT-2 on my computer and asked the same question I did at the start, what comes after “The cat sat on the”. This is what it said:
Figure 1 · GPT-2 (124M), my own run with the Hugging Face transformers library
This really surprised me. First, “mat” isn’t even close to the top. It’s 31st, at 0.4%. GPT-2 learned from millions of web pages, not from children’s books, and on the internet it seems that cats sit on floors and beds a lot more than mats. Second, look how spread out it is. The top answer only gets 7.6%, and even the top 10 together only make up 43%. “Volcano” is still technically possible: 0.0001%, ranked 11,229th.
This is exactly what i was talking about, it shows how the model’s “answer” isn’t really an answer. It’s a huge list of options, each one with a different probability, all of them possible.
Step 3: Pick one
Now the model has a list of 50,257 possible answers, and it has to pick one for the token. The obvious way is to always take the one with the highest chance. This is called greedy decoding. It sounds like a good plan, so I tried it and here’s what GPT-2 gave me with “My favourite thing about robots is”:
My favourite thing about robots is that they’re not just robots. They’re also very intelligent. They’re very intelligent. They’re very intelligent. They’re very intelligent…
It just keeps saying the same thing, and researchers have discovered this aswell, always picking the most likely words makes text that is “bland and strangely repetitive” [7].
So instead, most chatbots sample. They pick at random, but weighted by the probabilities, like pulling a ball out of a bag where “floor” has 76 balls out of 1,000 and “mat” has 4. Here are three runs of the same prompt with sampling:
My favourite thing about robots is their ability to sit in rooms far away from humans…
My favourite thing about robots is that they shape the robots in super-addictive ways…
My favourite thing about robots is that you can build that signature on them and instantly else…
No loops, and every run is different. (GPT-2 is a tiny, old model, so it doesn’t make much sense, but you get the idea.) This is exactly why ChatGPT gives you a different answer when you ask the same thing again.
Temperature: the creativity dial
Developers can control how “creative” (and thus how adventurous the model is) the picking is with a setting called temperature. It works by dividing all the scores by a number before softmax turns them into probabilities. A low temperature makes the gaps between scores bigger, so the top choices take over. A high temperature squashes the scores together, so everything becomes more equally likely.
Figure 2 · GPT-2 (124M), my own run
At temperature 0.5, “floor” jumps from 7.6% to 24.9%, and just 9 tokens cover 90% of the chance. At temperature 2, “floor” drops to 0.6%, and you’d need over 15,000 different tokens to cover 90%, which means loads of random stuff is picked.
Top-k and top-p: cutting off the nonsense
Get Ybro’s stories in your inbox
Join Medium for free to get updates from this writer.
Even at normal temperature, the bottom of the list is full of tokens that make no sense, and with 50,257 options, all those tiny chances add up. To avoid returning nonsense occasionally, models often cut the list down before picking:
- Top-k only keeps the k most likely tokens, for example the top 40, and throws away the rest.
- Top-p (also called nucleus sampling) keeps the smallest group of tokens that adds up to a chance of p, for example 90% [7]. For “The cat sat on the”, that’s 728 tokens. For a sentence where the next word is obvious, it could be just one or two.
Top-p allows the list to automatically gets shorter when the model is confident on some choices and longer when it isn’t, making it really clever.
Step 4: Where do the probabilities come from?
The probabilities dont come pre-made, the model learns them by guessing what comes next billions of times, generating its own set or probabilities (parameters).
- Take some text from training data and hide the next token.
- Let the model guess the probabilities.
- Check how “surprised”it was by the real answer. If it gave the right token a high chance, it’s barely surprised. If it gave it a tiny chance, it’s very surprised. This “surprise score” is called the loss (cross-entropy).
- Nudge the model’s internal numbers (its parameters) slightly, so it would be a bit less surprised next time.
Then repeat. A lot. (GPT-3 did this over 300 billion tokens of text )[5].
What did I do?
To really get this, I followed Andrej Karpathy’s makemore tutorial [1], where he builds the simplest language model there is. It predicts the next letter of a name using only the letter before it, and “training” is literally just counting, you go through 32,033 real names, count how often each letter follows each other letter, and turn the counts into percentages.
I rebuilt it in Python, after an “m”, my model thinks the next letter is “a” 39% of the time, “i” 19%, “e” 12% and “end of name” 8%. Nobody told it that names like Maria and Mason are common. It worked that out just by counting.
Counting works when you only look back one letter, but it explodes as you look further back. With 27 symbols (26 letters plus “end”), remembering two letters needs a table with 729 rows, and three letters needs 19,683. For GPT-2’s 50,257 tokens, a counting table that only remembers one token back would need over 2.5 billion entries, making it take up a lottttt of storage and being incredibly inefficient. That’s why real models use neural networks instead, they learn to estimate the probabilities without storing every single combination, so it can look back over thousands of tokens. In the second half of his video, Karpathy trains a tiny neural network on the names and shows that it learns almost the same probabilities as the counting table, but far more efficiently.
Putting it all together
Figure 3 · the predict, pick, repeat loop
That’s it (albeit a very simplified model), a reply from ChatGPT is just this loop running hundreds of times, one token at a time. The neural network does the hard part, working out good scores, but the actual choice of each token is basically a weighted dice roll.
This idea is over 100 years old
What surprised me most while researching is that “predict the next thing using probability” isn’t new at all:
- 1913: Andrey Markov counted the first 20,000 letters of Pushkin’s poem Eugene Onegin by hand, and showed that whether a letter is a vowel depends on the letter before it [3]. That’s the first time his idea of a “chain” of probabilities was used on language.
- 1948: Claude Shannon generated fake English by picking each letter based on the one before it [2]. His results looked like ON IE ANTSOUTINYS ARE T INCTORE ST BE S DEAMY ACHIN. Not great, but it already looks more like English than random letters do.
- 2017: Researchers at Google invented the transformer [4], the design that lets a model look back over a whole piece of text instead of just the last word or two. It’s the “T” in GPT.
- 2020: GPT-3 had 175 billion parameters and was trained on 300 billion tokens of text [5].
The big jump between Shannon and ChatGPT isn’t the basic idea. It shows just how much the model can remember, and how good it is at working out the probabilities.
Why does this even matter?
Once you know AI is picking from probabilities, a lot of its weird behavior suddenly makes sense.
Why it makes things up.
AI “hallucinations” are when a model confidently says something false. A 2025 paper from OpenAI explains part of the reason [8]. During training, the model just learns to predict the next word, but there isn’t anything in its training data that shows it which sentences are true. Patterns like spelling and grammar are easy to learn that way, but random facts, like a particular person’s birthday, aren’t. On top of that, most AI tests only check if the answer is right, so a model that guesses gets a better score than one that says “I don’t know”. Their example: if a model guesses someone’s birthday, it has a 1 in 365 chance of being right, but saying “I don’t know” scores zero every time. So models learn to guess confidently, leaving us with hallucinations.
Why do we get different answers with the same prompt?
That’s the sampling from Step 3. Every token is basically a dice roll, so a long answer is hundreds of dice rolls, making it almost impossible for two answers to be exactly the same.
Why is it hard to have a creative and reliable model?
A lower temperature (randomness) makes a model safer and more predictable, which is good for maths or coding. A higher temperature makes it more varied, which is better for stories and brainstorming, however it’s also more likely to pick something that sounds right but really isn’t. There’s no single perfect setting, which is why developers choose it for each job.
What I’m doing next
The thing that I found most interesting is that the core of a language model is simple enough to physically do. Shannon did it with a book in 1948. So my next project is to build one you can actually hold: like Karpathy’s name model made out of boxes of letter beads, where you draw beads randomly to make names. I want to find out whether ideas like temperatures still work when they’re physical, and whether this makes it easier to understand.
Key takeaways
- AI doesn’t understand what it is saying, merely spewing out what it thinks is most likely.
- For every token, the model gives all its options (50,257 for GPT-2) a probability, then picks one at random, weighted by those probabilities.
- Always picking the top option makes boring, repetitive text, so models sample instead, using settings like temperature and top-p to control how risky the picks are.
- The probabilities are learned by guessing the next token billions of times and producing a set of parameters.
- This shows us how AI can be creative, give different answers, and sometimes hallucinate.
Thanks for reading! If you have questions or want to talk about AI or robotics, connect with me on LinkedIn: https://www.linkedin.com/in/yibo-liu-850734235/
References
- Karpathy, A. (2022). The spelled-out intro to language modeling: building makemore. YouTube. Video · Code
- Shannon, C. E. (1948). “A Mathematical Theory of Communication.” Bell System Technical Journal, 27, 379–423 and 623–656. PDF
- Hayes, B. (2013). “First Links in the Markov Chain.” American Scientist, 101(2), 92.
- Vaswani, A. et al. (2017). “Attention Is All You Need.” NeurIPS.
- Brown, T. B. et al. (2020). “Language Models are Few-Shot Learners.” NeurIPS.
- Hugging Face. GPT-2 model documentation (vocabulary size 50,257).
- Holtzman, A. et al. (2020). “The Curious Case of Neural Text Degeneration.” ICLR.
- Kalai, A. T. et al. (2025). “Why Language Models Hallucinate.” OpenAI.
