Gradient Descent and Optimizers

An optimizer transforms current parameters and their gradients into an update. It does not compute the loss or gradients: the forward pass defines the loss, backward() populates gradients, and step() applies the optimizer rule.

Prerequisites

Complete Loss Functions and Objectives. You should understand that gradients accumulate in parameter .grad fields.

Learning goals

• order gradient clearing, forward computation, backward propagation, and parameter update.
• calculate a small stochastic-gradient-descent update by hand.
• distinguish optimizer choice from learning-rate schedule, clipping, and weight decay.

Worked example

For $L(w)=(w-3)^2$, the derivative is $2(w-3)$. Plain SGD with learning rate $0.1$ updates $w \leftarrow w-0.1\,\partial L/\partial w$.

import torch
from torch import nn

w = nn.Parameter(torch.tensor(0.0))
optimizer = torch.optim.SGD([w], lr=0.1)

for step in range(3):
    optimizer.zero_grad(set_to_none=True)
    loss = (w - 3.0).square()
    loss.backward()
    optimizer.step()
    print(f"step={step + 1} loss={loss.item():.4f} w={w.item():.4f}")

optimizer.zero_grad(set_to_none=True)
print(f"grad_is_none_after_zero={w.grad is None}")

Output:

step=1 loss=9.0000 w=0.6000
step=2 loss=5.7600 w=1.0800
step=3 loss=3.6864 w=1.4640
grad_is_none_after_zero=True

The printed loss belongs to the parameter value before that line's update. Logging it as a post-update loss would shift the learning curve and complicate debugging.

Mini-batch SGD uses a noisy estimate of the full-data gradient. Momentum smooths update direction using prior gradients. Adam rescales coordinates using moving first and second moments; AdamW applies decoupled weight decay rather than inserting an L2 term into Adam's adaptive gradient. These methods still require learning-rate selection and honest validation. A faster fall in training loss does not establish better generalization.

Schedules change the learning rate over steps or epochs. Gradient clipping limits an update signal, which can protect a run from a spike but can also hide the cause of persistent exploding gradients. Log the pre-clipping norm when diagnosing. If accumulating gradients across micro-batches intentionally, scale the loss or learning rate consistently and clear gradients only at the accumulation boundary.

Common mistakes

• Calling zero_grad() after backward() but before step(), erasing the update signal.
• Forgetting to clear gradients and accidentally accumulating across unrelated batches.
• Comparing optimizers with unequal tuning budgets or different regularization.
• Describing AdamW weight decay as identical to an L2 penalty under every optimizer.
• Reacting to one noisy batch by changing several hyperparameters at once.

Practice

Derive the first two updates without running the code. Then change the learning rate to 0.6: predict whether the sequence converges monotonically, oscillates, or diverges, and verify your answer.

Next steps

Continue with Weight Initialization to control signal and gradient scales before the first optimizer step.

Primary sources

Checked 2026-10-16: PyTorch optimizer interface, PyTorch SGD, PyTorch AdamW, and Decoupled Weight Decay Regularization.