Skip to main content

Project: Build a Text Classification Pipeline

This project classifies tiny repository-owned support messages as billing or technical. A TF-IDF logistic-regression baseline is deliberately simpler than a transformer: it is cheap, inspectable, deterministic, and strong enough to expose the evaluation contract before an optional pretrained extension adds cost and artifact risk.

Prerequisites​

Complete the preceding NLP lessons, especially Text Data and Tokenization, Fine-Tuning a Text Classifier, and Evaluating NLP Systems. Use the pinned scikit-learn 1.9.1 environment.

Learning goals​

  • reject exact cross-split duplicates and fit vocabulary/IDF state on the training split only.
  • select a decision threshold on validation data and touch test labels once.
  • report confusion, macro F1, class support, probability error, length slices, and named errors.

Worked example​

The fixtures contain 12 training, 4 validation, and 6 test messages. They are balanced for teaching convenience and do not represent real support traffic. Stable IDs support qualitative error review without making row order an identity.

import json
import os
from pathlib import Path

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, brier_score_loss, confusion_matrix, f1_score

fixture_root = Path(os.environ.get("NLP_FIXTURE_ROOT", "tests/fixtures/modern-ai-nlp"))

def load(split):
    return [json.loads(line) for line in (fixture_root / f"{split}.jsonl").read_text().splitlines()]

train, validation, test = (load(split) for split in ("train", "validation", "test"))
normalize = lambda text: " ".join(text.lower().split())
split_texts = [{normalize(row["text"]) for row in rows} for rows in (train, validation, test)]
assert not (split_texts[0] & split_texts[1] or split_texts[0] & split_texts[2] or split_texts[1] & split_texts[2])

train_texts = [row["text"] for row in train]
train_labels = np.array([row["label"] == "technical" for row in train], dtype=int)
vectorizer = TfidfVectorizer(ngram_range=(1, 2), lowercase=True)
vectorizer.fit(train_texts)  # training split only
model = LogisticRegression(C=4.0, solver="liblinear", random_state=45)
model.fit(vectorizer.transform(train_texts), train_labels)

validation_labels = np.array([row["label"] == "technical" for row in validation], dtype=int)
validation_probability = model.predict_proba(vectorizer.transform([row["text"] for row in validation]))[:, 1]
candidates = [0.40, 0.50, 0.60]
threshold = max(candidates, key=lambda value: (f1_score(validation_labels, validation_probability >= value), -abs(value - 0.5)))

test_labels = np.array([row["label"] == "technical" for row in test], dtype=int)
test_probability = model.predict_proba(vectorizer.transform([row["text"] for row in test]))[:, 1]
test_predictions = (test_probability >= threshold).astype(int)  # first and only test use
confusion = confusion_matrix(test_labels, test_predictions)
lengths = np.array([len(row["text"].split()) for row in test])
short = lengths <= 4
error_ids = [row["id"] for row, truth, prediction in zip(test, test_labels, test_predictions) if truth != prediction]

print(f"splits={{'train': {len(train)}, 'validation': {len(validation)}, 'test': {len(test)}}} vocabulary={len(vectorizer.vocabulary_)} overlap=0")
print(f"threshold={threshold:.2f} validation_macro_f1={f1_score(validation_labels, validation_probability >= threshold, average='macro'):.3f}")
print(f"support={np.bincount(test_labels).tolist()} confusion={confusion.tolist()}")
print(f"accuracy={accuracy_score(test_labels, test_predictions):.3f} macro_f1={f1_score(test_labels, test_predictions, average='macro'):.3f} brier={brier_score_loss(test_labels, test_probability):.3f}")
print(f"short_accuracy={accuracy_score(test_labels[short], test_predictions[short]):.3f} long_accuracy={accuracy_score(test_labels[~short], test_predictions[~short]):.3f} error_ids={error_ids}")

Output with the pinned local environment:

splits={'train': 12, 'validation': 4, 'test': 6} vocabulary=83 overlap=0
threshold=0.50 validation_macro_f1=1.000
support=[3, 3] confusion=[[3, 0], [0, 3]]
accuracy=1.000 macro_f1=1.000 brier=0.107
short_accuracy=1.000 long_accuracy=1.000 error_ids=[]

The vectorizer's vocabulary and inverse-document-frequency values come only from training text. Validation chooses the threshold. Only then are test probabilities and labels used. A real project should group split by customer, conversation, author, template, and time as appropriate; normalized exact matching does not catch a near-duplicate paraphrase.

An optional transformer extension is manual and separate. In a reviewed environment with transformers==5.20.0, load a pinned tokenizer/model revision, tokenize each fixed split, tune only on validation, and compare errors, calibration, latency, memory, and slice behavior against this baseline. That is a manual/download boundary; CI requires no hosted API, credentials, model hub, or cached weights.

Common mistakes​

  • Fitting TF-IDF or tokenizer vocabulary before splitting.
  • Allowing messages from one thread, customer, or template family into multiple splits.
  • Treating this balanced fixture as evidence about production class imbalance.
  • Reading test errors and then changing features, threshold, or model while retaining the same test claim.
  • Replacing a competitive sparse baseline with a transformer without measuring quality and operational cost.
  • Logging private message text during qualitative error review instead of controlled IDs and redacted excerpts.

Practice​

Add a third account class with disjoint examples in every split. Declare the split unit and class costs first; then choose a multiclass metric and decide how probability calibration will be checked.

Next steps​

The NLP and transformers batch is complete. Return to the Deep Learning & Modern AI overview; the next path begins with how large language models generate text.

Primary sources​

Checked 2026-10-16: scikit-learn TfidfVectorizer, scikit-learn LogisticRegression, scikit-learn classification metrics, Hugging Face sequence classification, and Hugging Face model cards.