Perceptrons and Multilayer Networks

A neuron computes an affine transformation (z = xW^T + b), then usually applies a nonlinear activation. A single thresholded neuron is a linear classifier: its decision boundary is a line, plane, or hyperplane. Stacking affine layers without nonlinearities still collapses to one affine transformation, so depth becomes useful only when nonlinear operations intervene.

Prerequisites

Complete Tensors, Gradients, and Autodiff. You should be able to multiply a (batch, input_features) matrix by a transposed weight matrix.

Learning goals

• map nn.Linear(in_features, out_features) to weight and bias shapes.
• explain why one linear boundary cannot represent XOR.
• trace batch, hidden, and output shapes through a multilayer perceptron (MLP).

Worked example

The four XOR inputs have shape (4, 2). A linear rule can isolate at most one side of a line, so the displayed perceptron misses one case. The two hidden ReLU units instead form two hinges from the sum (x_1+x_2); the output combines them into the XOR pattern.

import torch
from torch import nn

x = torch.tensor(
    [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]],
    dtype=torch.float32,
)
linear_logits = x @ torch.tensor([1.0, 1.0]) - 1.5
linear_predictions = (linear_logits > 0).to(torch.int64)

model = nn.Sequential(nn.Linear(2, 2), nn.ReLU(), nn.Linear(2, 1))
with torch.no_grad():
    model[0].weight.copy_(torch.tensor([[1.0, 1.0], [1.0, 1.0]]))
    model[0].bias.copy_(torch.tensor([0.0, -1.0]))
    model[2].weight.copy_(torch.tensor([[10.0, -20.0]]))
    model[2].bias.fill_(-5.0)

mlp_logits = model(x).squeeze(1)
mlp_predictions = (mlp_logits > 0).to(torch.int64)
print(f"input_shape={tuple(x.shape)} output_shape={tuple(mlp_logits.shape)}")
print(f"linear={linear_predictions.tolist()} mlp={mlp_predictions.tolist()}")

Output:

input_shape=(4, 2) output_shape=(4,)
linear=[0, 0, 0, 1] mlp=[0, 1, 1, 0]

For a learned binary classifier, the final layer would normally return one raw logit per example and a numerically stable loss would connect that logit to the label. Do not threshold inside the training graph: thresholding is nondifferentiable and a deployment threshold is a decision rule, not a learned probability model.

Hidden width and depth change representation capacity, but they do not guarantee generalization or causal understanding. Parameter count, data coverage, inductive assumptions, regularization, and evaluation all matter. The hand-set weights above prove representational possibility, not that an optimizer will find the same solution on every dataset.

Common mistakes

• Omitting the activation between Linear layers and believing the result is nonlinear.
• Transposing the batch or weights until code runs without documenting the intended dimensions.
• Applying sigmoid before a loss that already expects logits.
• Treating a network's output score as a calibrated probability without checking calibration.
• Increasing width to memorize training examples while ignoring validation behavior.

Practice

Write down every parameter shape in the example: (2, 2), (2,), (1, 2), and (1,). Then compute the two hidden activations for input [1, 1] by hand and confirm why the final logit is negative.

Next steps

Continue with Activation Functions to compare the nonlinearities that make multilayer networks expressive and trainable.

Primary sources

Checked 2026-10-16: PyTorch nn.Linear, PyTorch nn.Sequential, and Rosenblatt's perceptron report.