Skip to main content

Generation, Decoding, and Randomness

A language model produces logits over a vocabulary. A decoding policy turns those logits into token choices and repeats the process. Greedy decoding, beam search, top-k, nucleus sampling, temperature, stopping rules, and random seeds belong to the output contract; they do not change what the model learned.

Prerequisites​

Understand logits, softmax, vocabulary IDs, and causal masks.

Learning goals​

  • distinguish model logits from a decoding policy.
  • compare deterministic greedy selection with seeded top-k sampling.
  • explain why a seed improves repeatability but does not make generation universally deterministic.

Worked example​

These are fixed logits for one step, not claims about a language model. Temperature rescales logits before softmax. Top-k sets every excluded logit to negative infinity before sampling.

import torch

vocabulary = ["<eos>", "red", "blue", "green"]
logits = torch.tensor([0.2, 2.0, 1.0, 0.5])
temperature = 0.7
probabilities = torch.softmax(logits / temperature, dim=0)
greedy_id = logits.argmax().item()

top_values, top_ids = torch.topk(logits, k=2)
top_probabilities = torch.softmax(top_values / temperature, dim=0)
generator = torch.Generator().manual_seed(45)
sampled_slot = torch.multinomial(top_probabilities, 1, generator=generator).item()
sampled_id = top_ids[sampled_slot].item()

print(f"probabilities={[round(value, 3) for value in probabilities.tolist()]}")
print(f"greedy={vocabulary[greedy_id]}")
print(f"top_k_ids={top_ids.tolist()} top_k_probabilities={[round(value, 3) for value in top_probabilities.tolist()]}")
print(f"seeded_sample={vocabulary[sampled_id]}")

Output with the pinned CPU environment:

probabilities=[0.053, 0.698, 0.167, 0.082]
greedy=red
top_k_ids=[1, 2] top_k_probabilities=[0.807, 0.193]
seeded_sample=red

One seeded CPU run produces the documented sample. Different devices, kernels, library versions, batched request order, or floating-point reductions can still alter a long generated path because every sampled token changes the next input.

Common mistakes​

  • Calling temperature a factuality or confidence control.
  • Applying top-k after softmax without renormalizing retained probabilities.
  • Reporting one sampled answer as representative of a stochastic policy.
  • Forgetting end tokens, maximum length, repetition behavior, and stop-string edge cases.
  • Comparing decoding methods without holding prompts, model revision, and evaluation set fixed.

Practice​

Run seeds 0 through 9 and count selected tokens. Then lower temperature and explain why the highest-logit token becomes more likely without becoming guaranteed under sampling.

Next steps​

Evaluating NLP Systems measures classification behavior across classes, confidence, lengths, and concrete errors.

Primary sources​

Checked 2026-10-16: PyTorch softmax, PyTorch multinomial sampling, Hugging Face generation strategies, and The Curious Case of Neural Text Degeneration.