Sampling Techniques in Large Language Models (LLMs)
Ever wonder how AI models like ChatGPT choose the next word when generating text? It’s all thanks to sampling techniques like temperature, top-k, and top-p, which control how predictable or creative the model’s responses are. In this article, we’ll break down these methods and show how each one shapes AI’s ability to generate compelling text.
Why Sampling Matters in LLMs?
Imagine if every time a language model generated text, it simply picked the most likely word. This results in repetitive responses. Sampling techniques bring the creativity into AI responses by adding an element of controlled randomness. Here’s a quick look at the main methods:
1. Temperature Sampling: Adjusting the Creativity
Temperature controls the randomness in text generation by adjusting the probabilities of words before the model makes its choice. A high temperature adds more variety, while a low temperature makes the model play it safe.
- High Temperature (≥ 1): Creates more varied and creative outputs. Ideal for brainstorming or fiction writing.
- Low Temperature (~0): Produces more focused, predictable responses, ideal for tasks like text summarization or factual Q&A.
To see temperature sampling in practice, let’s try generating a joke about chickens with different temperature settings
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
"microsoft/Phi-3-mini-4k-instruct",
device_map="cuda",
torch_dtype="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
# Create a pipeline
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
return_full_text=False,
max_new_tokens=500,
do_sample=False,
)
# Prompt
messages = [
{"role": "user", "content": "Create a funny joke about chickens."}
]Example with Temperature = 0 (Conservative Output)
Using a low temperature keeps the model focused on the most likely response, giving a more traditional, straightforward joke.
# Using a low temperature (more deterministic)
output = pipe(messages, do_sample=False, temperature=0.0)
print(output[0]["generated_text"])Output: Why did the chicken join the band? Because it had the drumsticks!
Example with Temperature = 1 (Creative Output)
Raising the temperature makes the model explore a wider range of responses, resulting in a more creative joke!
# Using a high temperature (more creative)
output = pipe(messages, do_sample=True, temperature=1.0)
print(output[0]["generated_text"])Output: Why don’t chickens use computers? Because they get too flustered trying to figure out the Wi-Fi!
Tip: If you’re looking for a story full of surprises, crank up the temperature. If you need precise, informative answers, lower it down.
2. Top-K Sampling: Limiting the Word Pool
Top-K sampling limits the model’s options to the top K most likely words, removing any with lower probabilities. A smaller K keeps responses on track, while a larger K offers a broader range of potential responses.
- Low K (e.g., K=10): Keeps responses concise and focused.
- High K (e.g., K=40): Introduces more creative possibilities.
Example with Low Top-K (K=5)
With a small K, the model chooses only from the five most probable next words, making the response more straightforward.
# Using a low top-k value (concise response)
output = pipe(messages, do_sample=True, top_k=5)
print(output[0]["generated_text"])Get Shashank Agarwal’s stories in your inbox
Join Medium for free to get updates from this writer.
Output: Why did the chicken join the circus? Because she wanted to learn how to “cluck-et” in the big top!
Example with Higher Top-K (K=40)
A larger K value allows the model to consider more options, leading to a more varied and creative response.
# Using a high top-k value (more creative response)
output = pipe(messages, do_sample=True, top_k=40)
print(output[0]["generated_text"])Output: Why did the chicken join an acting troupe? Because it wanted to get out of the coop and let its feathers fly in the spotlight!
Tip: For factual responses, use a smaller K. For more creative or narrative text, try increasing K to open up the model’s options.
3. Top-P (Nucleus) Sampling: Dynamic Creativity
Also called nucleus sampling, this method sets a probability threshold ppp. The model chooses from a dynamically changing pool of words until the cumulative probability reaches ppp. It’s adaptive and works well for maintaining coherence in longer responses.
- Low P (e.g., P=0.5): Selects only the most probable words, producing more focused responses.
- High P (e.g., P=2.0): Expands the word choices, leading to richer and more varied language.
Example with Low Top-P (P=0.5)
With a low P, the model only picks from highly probable words, resulting in a focused, straightforward response.
# Using a low top-p value (focused response)
output = pipe(messages, do_sample=True, top_p=0.5)
print(output[0]["generated_text"])Output: Why did the chicken join the band? Because it had the drumsticks!
Example with Higher Top-P (P=2.0)
A higher P allows the model to consider a broader range of options, making the response more expressive and detailed.
# Using a high top-p value (more varied response)
output = pipe(messages, do_sample=True, top_p=2.0)
print(output[0]["generated_text"])Output: Why did the chicken join the book club? Because it finally felt like it had found the best way to give its allviews!
Tip: Use lower P for factual responses and higher P for storytelling to add depth without losing coherence.
Combining Sampling Techniques
Often, techniques are combined to create a balance between creativity and accuracy:
- Temperature + Top-K: Use a medium temperature with a top-k value (e.g., 0.7 temperature with K=10) to keep text grounded but still varied.
- Temperature + Top-P: Try combining temperature with top-p for adaptability. For example, 0.8 temperature with P=0.9 can yield fluid, coherent narratives.
Real-World Applications
Sampling adjustments can completely change the tone and usefulness of AI-generated text. For example:
- Customer Service Chatbots: Lower temperature, lower K and P to ensure consistent, on-point answers.
- Creative Writing AI: High temperature, high K and P for original, creative language.
- Educational Tools: Medium temperature with a reasonable P to generate content that is both informative and engaging.
