Weight Initialization

Initialization chooses the network's starting point and its initial signal scale. Identical zero weights make units in the same layer learn symmetrically; arbitrary scales can shrink or amplify activations and gradients through depth. Good initialization improves the starting conditions but does not guarantee convergence.

Prerequisites

Complete Gradient Descent and Optimizers and review variance and standard deviation.

Learning goals

• recognize repeated shrinkage and amplification through depth.
• connect fan-in/fan-out and activation assumptions to Xavier and Kaiming schemes.
• separate controlled random initialization from full reproducibility.

Worked example

Twelve repeated multiplications make the scale problem visible. The same lesson motivates variance-aware random weights. For ReLU, Kaiming uniform initialization uses the declared fan mode and nonlinearity; with four inputs and fan_in, its theoretical uniform bound is $\sqrt{6/4}$.

import torch
from torch import nn

small = torch.tensor(1.0)
large = torch.tensor(1.0)
for _ in range(12):
    small = small * 0.5
    large = large * 1.5

layer = nn.Linear(4, 4, bias=False)
generator = torch.Generator().manual_seed(43)
nn.init.kaiming_uniform_(
    layer.weight,
    a=0.0,
    nonlinearity="relu",
    generator=generator,
)
bound = (6 / 4) ** 0.5

print(f"vanishing_signal={small.item():.6f} exploding_signal={large.item():.3f}")
print(f"kaiming_bound={bound:.4f} within_bound={bool((layer.weight.abs() <= bound).all())}")

Output:

vanishing_signal=0.000244 exploding_signal=129.746
kaiming_bound=1.2247 within_bound=True

Xavier/Glorot initialization balances fan-in and fan-out under assumptions suited to approximately linear or symmetric activations. Kaiming/He initialization accounts for rectifiers discarding part of the signal. In PyTorch, confirm whether a weight is used as x @ weight.T, as nn.Linear does; the initialization documentation warns that transposed use changes which dimension is fan-in.

Biases are often initialized to zero because weights already break symmetry. Output-layer initialization may deserve separate treatment when a known class prior or initial prediction is useful, but encode that choice explicitly and do not leak test prevalence.

A seed fixes the random-number stream in one declared environment. PyTorch does not promise bitwise-identical floating-point results across releases, platforms, or CPU/GPU devices. Record the version and execution envelope; use repeated seeds when conclusions depend on optimization variability.

Common mistakes

• Setting every weight to the same value and preventing hidden units from specializing.
• Pairing a ReLU-oriented initializer with a very different activation without reconsidering scale assumptions.
• Reinitializing pretrained parameters accidentally after loading them.
• Claiming a seed makes training universally deterministic.
• Treating stable initial activation variance as proof that later gradients will remain healthy.

Practice

Compute the two repeated signals after four multiplications. Then compare the Kaiming bound for fan-in 4 and fan-in 64. Explain why wider inputs require a smaller typical weight scale.

Next steps

Continue with Regularization and Dropout to control memorization after initialization.

Primary sources

Checked 2026-10-16: PyTorch initialization functions, Glorot and Bengio (2010), He et al. (2015), and PyTorch reproducibility.