Debugging Training Runs

Training failure is a symptom, not a diagnosis. Begin with the smallest deterministic batch that should overfit, verify shapes and labels, then add data and complexity while logging enough evidence to distinguish optimization, generalization, and evaluation failures.

Prerequisites

Complete Batch and Layer Normalization and the earlier lessons on losses, optimizers, and regularization.

Learning goals

• separate data, forward-pass, gradient, optimizer, and evaluation checks.
• recognize overfitting, exploding gradients, and leakage from observable evidence.
• use anomaly detection and clipping as focused diagnostic tools, not blanket cures.

Worked example

A compact run log can expose multiple hypotheses. Falling training loss with rising validation loss is an overfitting signal, not proof of its cause. A rapidly growing gradient norm suggests an unstable update path. Fitting preprocessing on all splits is direct leakage.

import torch

history = [
    {"train_loss": 0.72, "validation_loss": 0.70, "gradient_norm": 1.2},
    {"train_loss": 0.31, "validation_loss": 0.44, "gradient_norm": 3.8},
    {"train_loss": 0.08, "validation_loss": 0.91, "gradient_norm": 180.0},
]
overfit_signal = (
    history[-1]["train_loss"] < history[-2]["train_loss"]
    and history[-1]["validation_loss"] > history[-2]["validation_loss"]
)
exploding_gradient = history[-1]["gradient_norm"] > 100

train = torch.tensor([0.0, 2.0])
test = torch.tensor([10.0])
train_mean = train.mean()
leaked_mean = torch.cat([train, test]).mean()

print(f"overfit_signal={overfit_signal} exploding_gradient={exploding_gradient}")
print(f"train_mean={train_mean.item():.1f} leaked_mean={leaked_mean.item():.1f}")

Output:

overfit_signal=True exploding_gradient=True
train_mean=1.0 leaked_mean=4.0

Use a layered checklist:

1. Data and split: inspect raw rows, target prevalence, duplicates, entity/time boundaries, missing values, and train-only preprocessing. Never use test behavior to tune the run.
2. Shapes and values: assert batch/output/target shapes, dtypes, devices, finite inputs, finite logits, and the loss contract.
3. Tiny-batch test: with stochastic regularizers disabled, verify that a sufficiently expressive model can drive loss down on a few examples. Failure points toward wiring or optimization; success does not prove generalization.
4. Gradient path: log norms by layer, None gradients, dead/saturated activation fractions, and finite values before clipping.
5. Update: compare parameter values before and after step(), confirm gradient clearing, and lower the learning rate as a controlled experiment.
6. Evaluation: pair model.eval() with torch.inference_mode(), preserve the untouched test set, and compare like-for-like reductions and metrics.

torch.autograd.detect_anomaly() can identify a backward operation that produces a non-finite value, but it adds overhead and should be scoped to diagnosis. clip_grad_norm_ can keep one run finite; log the unclipped norm and investigate initialization, scale, architecture, data, and learning rate rather than hiding persistent instability.

Common mistakes

• Changing data, architecture, optimizer, and learning rate simultaneously.
• Looking only at the final scalar loss and not the first failing layer or batch.
• Using validation/test rows to compute normalization statistics.
• Confusing eval() with disabled gradient recording; the two controls are orthogonal.
• Treating one successful seed or one tiny-batch overfit as evidence of robust learning.

Practice

Add a finite check for every numeric field in the sample history. Then write the first three experiments you would run if training loss stayed flat, and a different first three if training loss fell while validation loss rose.

Next steps

Apply the complete workflow in Project: MLP from Data to Evaluation.

Primary sources

Checked 2026-10-16: PyTorch autograd anomaly detection, PyTorch gradient clipping, PyTorch autograd mechanics, and PyTorch reproducibility.