Skip to main content

Text Data and Tokenization

Models do not receive prose. They receive integer IDs under a particular tokenizer and vocabulary version. Normalization, segmentation, special tokens, truncation, and padding are therefore part of the model contract—not harmless cleanup.

Prerequisites​

Know Python strings and tensor shapes. The neural-network foundations path is useful, but this lesson begins with standard-library code.

Learning goals​

  • distinguish tokens, vocabulary entries, IDs, and an attention mask.
  • fit vocabulary state on the training split only and map unseen pieces to an out-of-vocabulary token.
  • make truncation and padding visible instead of silently changing examples.

Worked example​

This intentionally simple word-and-punctuation tokenizer is not a production tokenizer. It makes every state transition inspectable. <pad> is ID 0 and <unk> is ID 1; the vocabulary sees training text only.

import re

def tokenize(text):
    return re.findall(r"[a-z]+|[!?]", text.lower())

train_texts = ["Reset password now", "Password reset worked"]
pieces = sorted({piece for text in train_texts for piece in tokenize(text)})
vocabulary = {"<pad>": 0, "<unk>": 1} | {
    piece: index for index, piece in enumerate(pieces, start=2)
}

tokens = tokenize("Reset account now!")
max_length = 6
ids = [vocabulary.get(token, vocabulary["<unk>"]) for token in tokens][:max_length]
ids += [vocabulary["<pad>"]] * (max_length - len(ids))
attention_mask = [token_id != vocabulary["<pad>"] for token_id in ids]

print(f"tokens={tokens}")
print(f"vocabulary={vocabulary}")
print(f"input_ids={ids}")
print(f"attention_mask={attention_mask}")

Output:

tokens=['reset', 'account', 'now', '!']
vocabulary={'<pad>': 0, '<unk>': 1, 'now': 2, 'password': 3, 'reset': 4, 'worked': 5}
input_ids=[4, 1, 2, 1, 0, 0]
attention_mask=[True, True, True, True, False, False]

Subword tokenizers reduce whole-word out-of-vocabulary failures, but they do not remove versioning or domain-shift problems. Length is measured after tokenization. Record whether truncation removes the beginning or end, and audit truncation rates by label and slice.

Common mistakes​

  • Building a vocabulary before splitting, which leaks validation and test text statistics.
  • Treating IDs as ordered quantities; ID 200 is not semantically larger than ID 20.
  • Using the padding ID as an unknown ID or forgetting the attention mask.
  • Deduplicating after splitting, allowing exact or near-duplicate text into evaluation.
  • Assuming whitespace corresponds to words in every language and writing system.

Practice​

Add a maximum length of three and report which tokens are lost. Then decide whether head-only, tail-only, or task-specific truncation is defensible for a support-ticket classifier.

Next steps​

Word and Sentence Embeddings replaces discrete IDs with learned vectors.

Primary sources​

Checked 2026-10-16: Hugging Face tokenizer API, Hugging Face tokenizers guide, and PyTorch pad_sequence.