Skip to main content

Sequence Models before Transformers

Recurrent neural networks process a sequence by carrying state from one step to the next. LSTMs and GRUs improve gradient flow and gating, but recurrence still constrains parallel training. This history motivates transformers; it does not make recurrent models obsolete for every small, streaming, or stateful workload.

Prerequisites​

Complete Word and Sentence Embeddings and know matrix multiplication and tanh.

Learning goals​

  • trace an input sequence through a recurrent hidden state.
  • observe that reversing tokens changes the final representation.
  • compare recurrence's streaming state with attention's parallel token interactions.

Worked example​

This two-dimensional recurrence uses fixed matrices. Each hidden state depends on the previous one, so time steps cannot all be evaluated independently.

import torch

sequence = torch.tensor([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
input_weight = torch.eye(2)
recurrent_weight = 0.5 * torch.eye(2)

def encode(tokens):
    hidden = torch.zeros(2)
    states = []
    for token in tokens:
        hidden = torch.tanh(token @ input_weight + hidden @ recurrent_weight)
        states.append(hidden)
    return torch.stack(states)

forward = encode(sequence)
reversed_states = encode(sequence.flip(0))
rounded = [[round(value, 3) for value in row] for row in forward.tolist()]

print(f"input={tuple(sequence.shape)} states={tuple(forward.shape)}")
print(f"forward_states={rounded}")
print(f"forward_final={[round(v, 3) for v in forward[-1].tolist()]}")
print(f"reversed_final={[round(v, 3) for v in reversed_states[-1].tolist()]}")

Output:

input=(3, 2) states=(3, 2)
forward_states=[[0.762, 0.0], [0.363, 0.762], [0.828, 0.881]]
forward_final=[0.828, 0.881]
reversed_final=[0.828, 0.414]

The final states differ because order changes the computation. In an RNN, padding is commonly handled with packed sequences or length-aware selection; reading the last padded position as if it were content is a bug.

Common mistakes​

  • Describing a recurrent hidden state as perfect memory; information can decay or be overwritten.
  • Ignoring exploding or vanishing gradients merely because an LSTM or GRU is used.
  • Comparing architectures without equal data, parameter, latency, and tuning budgets.
  • Claiming transformers dominate low-latency streaming or small-data workloads categorically.
  • Leaking future tokens into a task that requires causal, online prediction.

Practice​

Set recurrent_weight to zero and explain why the final state then ignores earlier tokens. Next, describe the state and latency constraints of an on-device streaming classifier.

Next steps​

Attention Step by Step replaces a single recurrent bottleneck with weighted access to token values.

Primary sources​

Checked 2026-10-16: PyTorch RNN API, PyTorch GRU API, LSTM paper, and GRU encoder-decoder paper.