Activation Functions

An activation function changes the geometry and gradient flow of a network. There is no universally best choice: the hidden activation, output parameterization, initialization, depth, and task must fit together.

Prerequisites

Complete Perceptrons and Multilayer Networks and be comfortable interpreting a derivative as local sensitivity.

Learning goals

• recognize saturation and dead activations from gradients.
• distinguish hidden-layer activations from output/loss parameterizations.
• choose a defensible default and name its failure modes.

Worked example

For sigmoid, $\sigma'(x)=\sigma(x)(1-\sigma(x))$. Far from zero the output saturates near 0 or 1 and the derivative becomes small. ReLU, $\max(0,x)$, avoids positive-side saturation but has zero derivative for negative inputs.

import torch

values = torch.tensor([-8.0, 0.0, 8.0], requires_grad=True)
sigmoid = torch.sigmoid(values)
sigmoid.sum().backward()
print(f"sigmoid={[round(value, 6) for value in sigmoid.tolist()]}")
print(f"sigmoid_grad={[round(value, 6) for value in values.grad.tolist()]}")

dead = torch.tensor([-2.0, -1.0], requires_grad=True)
torch.relu(dead).sum().backward()
print(f"relu_grad={dead.grad.tolist()}")

Output:

sigmoid=[0.000335, 0.5, 0.999665]
sigmoid_grad=[0.000335, 0.25, 0.000335]
relu_grad=[0.0, 0.0]

Repeated multiplication by derivatives smaller than one can produce a vanishing signal through many layers. Derivatives or weights consistently larger than one can instead explode. Residual connections, normalization, initialization, and architecture all influence this behavior; changing the activation alone is not a complete diagnosis.

Useful choices include:

• ReLU: a strong, inexpensive hidden-layer baseline, with an unbounded positive side and possible dead units.
• Leaky ReLU: preserves a small negative slope, reducing—but not eliminating—dead-unit risk.
• GELU/SiLU: smooth nonlinearities common in modern architectures; their extra computation is rarely the first concern in a tiny network.
• Tanh: zero-centered output but saturating tails; still useful when a bounded signed state is intentional.
• Sigmoid: appropriate for converting a binary logit at evaluation time, but usually not the default deep hidden activation.

For multiclass classification, the model can return one raw logit per class. CrossEntropyLoss handles the log-softmax calculation stably, so adding softmax before it changes the intended contract.

Common mistakes

• Choosing an activation by popularity without considering the loss and output domain.
• Calling every zero ReLU output a defect; sparsity is expected, while units that remain inactive for all relevant data are the concern.
• Diagnosing vanishing gradients from one layer or batch without logging gradients across depth and time.
• Using a bounded output for an unbounded regression target without documenting that constraint.
• Rounding tiny gradients to zero in logs and then reasoning from the rounded value.

Practice

Compute the sigmoid derivative at 0 by hand. Then predict how the gradients at (-8) and (8) change if their magnitude doubles. For ReLU, propose one data or initialization change and one activation change that could revive consistently inactive units.

Next steps

Continue with Loss Functions and Objectives to connect raw outputs to the quantity optimized during training.

Primary sources

Checked 2026-10-16: PyTorch nonlinear activations, PyTorch ReLU, and Glorot and Bengio on saturation and training difficulty.