Skip to main content

Project: MLP from Data to Evaluation

This project assembles the full neural-foundations workflow: local data, stratified splits, train-only preprocessing, explicit shapes/dtypes/device, a nonlinear MLP, stable logits loss, gradient state, optimizer updates, validation-based checkpoint selection, evaluation mode, and one final test measurement.

Prerequisites​

Complete Debugging Training Runs and all earlier lessons in this path. Use the repository's pinned PyTorch 2.14.1 CPU environment.

Learning goals​

  • build a complete CPU-safe classification run with no download, account, accelerator, or secret.
  • isolate training, validation, and test roles while fitting preprocessing on training data only.
  • select a checkpoint on validation loss and report held-out test loss and accuracy honestly.

Worked example​

The fixture contains 400 noisy two-dimensional points around four local centers. Opposite centers share a label, creating a nonlinear XOR-like task. Each center contributes 60 training, 20 validation, and 20 test examples, so all splits contain both labels and all four regions. The fixed model and 80-epoch budget are declared before test evaluation.

import copy
import torch
from torch import nn

torch.manual_seed(43)
torch.set_num_threads(1)
torch.use_deterministic_algorithms(True)
device = torch.device("cpu")
generator = torch.Generator().manual_seed(430)

centers = torch.tensor([[-1.0, -1.0], [-1.0, 1.0], [1.0, -1.0], [1.0, 1.0]])
center_labels = torch.tensor([0, 1, 1, 0], dtype=torch.long)
features, targets = [], []
for center, label in zip(centers, center_labels):
    features.append(center + 0.28 * torch.randn((100, 2), generator=generator))
    targets.append(torch.full((100,), label, dtype=torch.long))
X = torch.cat(features)
y = torch.cat(targets)

train_parts, validation_parts, test_parts = [], [], []
for start in range(0, 400, 100):
    order = start + torch.randperm(100, generator=generator)
    train_parts.append(order[:60])
    validation_parts.append(order[60:80])
    test_parts.append(order[80:])
train_index = torch.cat(train_parts)
validation_index = torch.cat(validation_parts)
test_index = torch.cat(test_parts)
assert len(set(train_index.tolist()) & set(test_index.tolist())) == 0

# Fit preprocessing on training features only, then reuse it unchanged.
mean = X[train_index].mean(dim=0)
std = X[train_index].std(dim=0)

def prepare(index):
    return ((X[index] - mean) / std).to(device), y[index].to(device)

X_train, y_train = prepare(train_index)
X_validation, y_validation = prepare(validation_index)
X_test, y_test = prepare(test_index)

model = nn.Sequential(
    nn.Linear(2, 8),
    nn.ReLU(),
    nn.Dropout(0.1),
    nn.Linear(8, 2),
).to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=0.03, weight_decay=0.01)
loss_function = nn.CrossEntropyLoss()

def evaluate(inputs, labels):
    model.eval()
    with torch.inference_mode():
        logits = model(inputs)
        loss = loss_function(logits, labels).item()
        accuracy = (logits.argmax(dim=1) == labels).float().mean().item()
    return loss, accuracy

best_loss, best_epoch, best_state = float("inf"), 0, None
for epoch in range(1, 81):
    model.train()
    optimizer.zero_grad(set_to_none=True)
    training_loss = loss_function(model(X_train), y_train)
    training_loss.backward()
    optimizer.step()

    validation_loss, _ = evaluate(X_validation, y_validation)
    if validation_loss < best_loss:
        best_loss = validation_loss
        best_epoch = epoch
        best_state = copy.deepcopy(model.state_dict())

model.load_state_dict(best_state)
train_loss, train_accuracy = evaluate(X_train, y_train)
validation_loss, validation_accuracy = evaluate(X_validation, y_validation)
test_loss, test_accuracy = evaluate(X_test, y_test)  # first and only test use

print(f"splits train={len(train_index)} validation={len(validation_index)} test={len(test_index)}")
print(f"shape={tuple(X_train.shape)} dtype={X_train.dtype} device={X_train.device}")
print(f"selected_epoch={best_epoch}")
print(f"train loss={train_loss:.4f} accuracy={train_accuracy:.3f}")
print(f"validation loss={validation_loss:.4f} accuracy={validation_accuracy:.3f}")
print(f"test loss={test_loss:.4f} accuracy={test_accuracy:.3f}")

Output with the pinned CPU environment:

splits train=240 validation=80 test=80
shape=(240, 2) dtype=torch.float32 device=cpu
selected_epoch=80
train loss=0.0309 accuracy=1.000
validation loss=0.0403 accuracy=0.988
test loss=0.0401 accuracy=1.000

The validation result selects among the 80 epoch checkpoints; it is therefore selection evidence, not an unbiased final estimate. The test split is evaluated only after the checkpoint is fixed. Re-running design choices after reading test performance would turn the test split into another validation set and require new untouched evidence.

Perfect test accuracy here is not a production claim. The fixture is deliberately small, balanced, synthetic, and cleanly separated. It contains no subgroup, calibration, shift, asymmetric-cost, privacy, latency, or external-validity evidence. Across other hardware or PyTorch versions, controlled seeds do not guarantee bitwise-identical floating-point results.

Common mistakes​

  • Standardizing before splitting, which leaks validation/test distribution information into training.
  • Tuning hidden width, epochs, regularization, or threshold from the test result.
  • Evaluating with dropout active or assuming inference_mode() calls eval() automatically.
  • Saving a live model object without its code/version/config/data contract instead of an explicit reviewed state and metadata.
  • Reporting accuracy without the split sizes, class coverage, loss, selection process, and fixture limitations.

Practice​

Increase the noise scale from 0.28 to 0.70 without changing the split or model. Before running, predict which metric should degrade first. Then add a confusion matrix computed after the checkpoint is fixed, and explain why choosing a new architecture from the test confusion matrix would invalidate the final test role.

Next steps​

The foundation batch is complete. Return to the Deep Learning & Modern AI overview; the computer-vision path continues with images as tensors, where spatial axes and local structure add new modeling assumptions.

Primary sources​

Checked 2026-10-16: PyTorch optimization tutorial, PyTorch CrossEntropyLoss, Autograd mechanics and evaluation mode, PyTorch reproducibility, and PyTorch numerical accuracy.