What Are Top-k and Top-p in LLMs? How Do They Control AI Output?
Top-k and Top-p are sampling techniques that control which candidate tokens an LLM can choose from during generation. Understanding them is important for controlling creativity, diversity, and consistency in AI applications.
What Are Top-k and Top-p?
When an LLM generates text, it predicts probabilities for possible next tokens.
For example:
"The sky is..."
The model may assign probabilities to:
blue → 70%
cloudy → 15%
clear → 10%
green → 1%
others → 4%
Top-k and Top-p control which of these candidate tokens the model can consider.
What Is Top-k?
Top-k sampling limits the model to the k most probable tokens.
For example, if:
Top-k = 3
the model considers only the three highest-probability candidates.
This reduces the possibility of selecting unlikely tokens.
Lower k → More focused
Higher k → More diverse
What Is Top-p?
Top-p (Nucleus Sampling) selects the smallest group of tokens whose cumulative probability reaches a specified threshold.
For example:
Top-p = 0.90
The model considers enough of the highest-probability tokens to cover approximately 90% of the probability mass.
Unlike Top-k, the number of candidate tokens can change from one generation step to another.
Top-k vs Top-p
Top-k | Top-p |
|---|---|
Uses a fixed number of candidate tokens | Uses a probability threshold |
Example: k = 20 | Example: p = 0.9 |
Candidate count stays fixed | Candidate count can vary |
Controls candidate pool size | Controls probability mass |
Why Are They Important?
They can help control the balance between:
Predictability ↔ Diversity
For example:
Customer support → More controlled output
Creative writing → More diverse output
Brainstorming → More variety
Interview Tip
Don't confuse Top-k/Top-p with Temperature.
Temperature changes the probability distribution used for sampling.
Top-k/Top-p restrict the set of candidate tokens considered for sampling.
They can be used together, depending on the model/provider.
Key Takeaway
Top-k → Select from the k most probable tokens.
Top-p → Select from the smallest set of tokens whose cumulative probability reaches p.
These sampling techniques help AI Engineers control how focused, diverse, and creative LLM-generated responses can be.
Related articles
More in LLM
Comments
Sign in to join the discussion.