When a Large Language Model generates text, it assigns scores to many possible next tokens. Sampling controls decide which candidates remain eligible and how one is selected. Two common controls are top-k sampling and top-p, also called nucleus sampling.

These controls can adjust variety and reduce the chance of selecting extremely unlikely tokens. They do not add knowledge, verify facts, or correct an unclear prompt.

Understanding Next-Token Probabilities

Suppose the model is continuing the sentence The sky is. For illustration, imagine the candidate distribution below. Real models consider token pieces rather than only complete words and have much larger vocabularies.

Without filtering, every candidate may remain eligible. Top-k and top-p remove part of this distribution, then the remaining probabilities are normally renormalized so they add up to one.

What Is Top-k Sampling?

Top-k keeps only the k candidates with the highest probabilities. If k equals 3 in the example, blue, cloudy, and clear remain. Every other candidate is assigned zero probability for that sampling step.

Before top-k:
blue 45%, cloudy 20%, clear 15%, beautiful 10%, ...

Apply top-k = 3:
blue, cloudy, clear

Renormalized probabilities:
blue 56.25%, cloudy 25%, clear 18.75%

Benefits of Top-k

• Simple fixed upper limit on candidate count.
• Excludes the long tail of unlikely tokens.
• Can reduce unusual or incoherent choices.
• Easy to compare in controlled experiments.

Limitation of Top-k

A fixed k does not adapt to uncertainty. When one token is overwhelmingly likely, k may retain unnecessary alternatives. When many tokens are similarly reasonable, a small k may remove useful choices.

What Is Top-p Sampling?

Top-p sorts candidates from highest to lowest probability and keeps the smallest set whose cumulative probability reaches at least p. This selected set is the nucleus.

With top-p equal to 0.90 in the example, blue, cloudy, clear, and beautiful are retained because their cumulative probability reaches 90%. The candidate count can change at every generated token.

Benefits of Top-p

• Adapts candidate count to the distribution's shape.
• Keeps fewer options when the model is confident.
• Keeps more options when probability is spread across alternatives.
• Offers a flexible balance between focus and variation.

Limitations of Top-p

A threshold does not guarantee a specific number or quality of candidates. A high-probability distribution can still be wrong, biased, or based on missing context.

Top-k vs Top-p

Neither method is universally better. A model provider may expose one, both, or neither, and may recommend a particular configuration for its decoding system.

Python Sampling Simulation

This teaching example filters an already normalized probability distribution. It does not call an LLM.

candidates = [
    ("blue", 0.45),
    ("cloudy", 0.20),
    ("clear", 0.15),
    ("beautiful", 0.10),
    ("dark", 0.05),
    ("green", 0.03),
    ("purple", 0.02),
]

def renormalize(items):
    total = sum(probability for _, probability in items)
    return [(token, probability / total) for token, probability in items]

def top_k(items, k):
    if k <= 0:
        raise ValueError("k must be positive")
    selected = sorted(items, key=lambda item: item[1], reverse=True)[:k]
    return renormalize(selected)

def top_p(items, threshold):
    if not 0 < threshold <= 1:
        raise ValueError("threshold must be in (0, 1]")
    selected = []
    cumulative = 0.0
    for item in sorted(items, key=lambda item: item[1], reverse=True):
        selected.append(item)
        cumulative += item[1]
        if cumulative >= threshold:
            break
    return renormalize(selected)

print("top-k:", top_k(candidates, 3))
print("top-p:", top_p(candidates, 0.90))

Production implementations work from logits, use optimized kernels, and handle ties and numerical precision according to their framework.

Using Top-k and Top-p Together

Some systems apply both filters, leaving the intersection of their permitted candidate sets. This can cap candidate count while also requiring a probability-mass threshold. The order and exact behavior can vary by implementation.

Combining restrictive values can collapse diversity more than expected. Begin with the provider's documented defaults and change one control at a time.

Top-k, Top-p, and Temperature

• Temperature reshapes probability differences before sampling.
• Top-k removes every token outside a fixed highest-probability count.
• Top-p removes tokens outside a chosen cumulative probability mass.
• A sampler selects from the remaining renormalized candidates.

Implementation order and parameter semantics vary. Temperature can change which tokens fall inside a top-p nucleus because it changes their probabilities.

{
  "temperature": 0.7,
  "top_p": 0.9,
  "top_k": 40
}

This object is conceptual. Not every API supports top_k, and valid ranges or recommendations differ.

Choosing Settings by Task

These are directions, not universal numeric presets. Structured-output features, deterministic decoding, or model-specific guidance may be more appropriate for narrow tasks.

Evaluation Workflow

• Define what useful diversity and unacceptable variation mean for the task.
• Create representative prompts, edge cases, and safety tests.
• Generate several samples per prompt for each configuration.
• Measure correctness, format adherence, coherence, diversity, and safety.
• Compare latency and cost because longer or failed outputs affect both.
• Change one sampling setting at a time when possible.
• Record the model version, prompt version, seed where supported, and every decoding parameter.

Advantages

• Reduce selection from extremely unlikely token tails.
• Control response variation without retraining the model.
• Allow focused or exploratory behavior for different product features.
• Provide measurable settings for decoding experiments.

Limitations

• They do not improve learned knowledge or verify factual claims.
• Restrictive settings can make text repetitive or remove a needed token.
• Loose settings can increase incoherence and instruction failures.
• The same values behave differently across models and prompts.
• Parameter availability and implementation order vary by platform.
• A single successful output is not enough to evaluate a stochastic configuration.
• Sampling controls do not replace schemas, tests, safety controls, or human review.

Best Practices

• Start with current model-provider defaults and guidance.
• Fix the prompt and context before using sampling to solve task ambiguity.
• Use the fewest decoding controls needed for a measured goal.
• Tune with repeated outputs from representative real-world inputs.
• Validate factual, coded, and structured results independently.
• Pin and log production settings where supported.
• Retest after any prompt, model, provider, or decoding update.

Real-World Applications

• Chat and writing assistants.
• Programming and educational tools.
• Customer-support drafting.
• Marketing variation and brainstorming.
• Story and dialogue generation.
• Multi-candidate generation followed by ranking or human selection.