Tensors, Gradients, and Autodiff

A tensor is an array together with a shape, dtype, device, and—when requested—gradient history. Those details are part of the model contract, not incidental bookkeeping. This path uses PyTorch 2.14.1 on CPU so every example stays local and repeatable.

Prerequisites

You should be comfortable with Python, vectors, matrices, and the chain rule. Revisit Vectors and Their Operations if shapes such as (batch, features) are unfamiliar.

Learning goals

• read a tensor's shape, dtype, device, and gradient state.
• connect a forward computation to the graph traversed by backward().
• distinguish leaf tensors, computed tensors, and accumulated .grad values.

Worked example

Let $x \in \mathbb{R}^{2 \times 2}$ and define the scalar loss

$$L = \frac{1}{4}\sum_{i,j} x_{ij}^{2}.$$

The derivative is $\partial L / \partial x = x/2$. PyTorch records the square and mean operations because the leaf tensor requests gradients:

import torch

x = torch.tensor(
    [[1.0, -2.0], [3.0, 0.5]],
    dtype=torch.float32,
    device="cpu",
    requires_grad=True,
)
loss = x.square().mean()
loss.backward()

print(f"shape={tuple(x.shape)} dtype={x.dtype} device={x.device}")
print(f"loss={loss.item():.4f}")
gradient = [[round(value, 3) for value in row] for row in x.grad.tolist()]
print(f"grad={gradient}")

Output:

shape=(2, 2) dtype=torch.float32 device=cpu
loss=3.5625
grad=[[0.5, -1.0], [1.5, 0.25]]

Calling backward() again would add into x.grad; it would not replace it. Training loops therefore clear parameter gradients before the next backward pass. A tensor created inside torch.inference_mode() does not retain a backward graph. That gradient mode is separate from a module's training/evaluation mode, which later controls layers such as dropout and batch normalization.

Shapes also constrain operations: multiplying a (batch, 3) feature matrix by a (3, 4) weight matrix produces (batch, 4). Broadcasting can make an unintended shape appear valid, so inspect both the intended and actual shape at boundaries.

Common mistakes

• Creating integer inputs and expecting gradients; differentiable neural computations normally use floating-point or complex tensors.
• Moving the model but not the batch to the same device, or mixing incompatible dtypes without an explicit reason.
• Reading .grad before backward(), when it is still None.
• Forgetting that gradients accumulate, then interpreting a sum across batches as one-batch evidence.
• Calling .item() or converting to NumPy in the middle of a loss calculation and accidentally leaving the graph.

Practice

Replace mean() with sum(). Derive the new gradient before running the code, then explain why its magnitude is four times larger. Next, change the input to shape (1, 4) and identify which properties change and which do not.

Next steps

Continue with Perceptrons and Multilayer Networks to turn tensor operations into a model with learnable parameters.

Primary sources

Checked 2026-10-16: PyTorch 2.14.1 release, Tensor attributes, Autograd mechanics, and Numerical accuracy.