Skip to main content

How Large Language Models Generate Text

A causal language model maps the tokens currently inside its finite context window to logits for the next token. A decoding policy converts those logits into a choice, appends it, and repeats. The model does not retrieve a hidden fact record or verify the completed sentence merely because its continuation is fluent.

Prerequisites​

Review Transformer Architecture and Generation, Decoding, and Randomness.

Learning goals​

  • trace one autoregressive generation step.
  • distinguish context, model scores, probabilities, and decoding.
  • explain why token probability is not factual confidence.

Worked example​

The table is a deterministic stand-in, not a trained model. It makes the interface visible: the current context selects fixed logits, softmax normalizes them, and greedy decoding appends one token.

import math

vocabulary = ["ready", "late", "missing"]
context = ["the", "report", "is"]
logits = [2.0, 1.0, 0.0]
weights = [math.exp(value) for value in logits]
probabilities = [value / sum(weights) for value in weights]
next_id = max(range(len(logits)), key=logits.__getitem__)
next_token = vocabulary[next_id]

print(f"context={context} next={next_token} probability={probabilities[next_id]:.3f}")
context = (context + [next_token])[-3:]
print(f"extended_context={context}")

Output:

context=['the', 'report', 'is'] next=ready probability=0.665
extended_context=['report', 'is', 'ready']

The number 0.665 is conditional on this toy vocabulary and context. It is neither the probability that the report is actually ready nor a calibrated confidence in the completed claim. Real systems also define tokenization, context truncation, stop conditions, decoding parameters, model revision, and serving precision.

Common mistakes​

  • Equating a likely continuation with a true statement.
  • Saying an LLM reads an entire unlimited conversation; every deployment has a context policy and limit.
  • Treating temperature as a knowledge or truth control.
  • Comparing outputs without fixing model revision, prompt, decoding, seed where supported, and evaluation cases.

Practice​

Replace greedy selection with a seeded draw and record how the continuation changes. Then write down which parts belong to the model and which belong to the application.

Next steps​

Prompting as Interface Design turns the free-form prompt into an explicit application contract.

Primary sources​

Checked 2026-10-16: Attention Is All You Need, Language Models are Unsupervised Multitask Learners, and PyTorch softmax.