Retries, Backfills, and Recovery

A retry repeats one failed attempt. A backfill intentionally creates work for historical intervals. Recovery is the broader procedure that restores trustworthy outputs after infrastructure failure, bad data, or bad code. All three depend on deterministic partitions and idempotent publication.

Prerequisites

Complete Late-Arriving Data and Watermarks and Idempotent Data Jobs.

Learning goals

• Retry transient failures with a limit and delay policy.
• Fail or quarantine permanent errors instead of retrying forever.
• Backfill explicit partitions through the same validated code path.
• Verify recovery from output invariants, not a green scheduler state alone.

Worked example

The first extraction attempt raises a simulated transient timeout; the second succeeds. The backfill then replaces each daily partition from replayable source events, so repeating it would converge on the same totals.

import json
from collections import defaultdict
from decimal import Decimal
from pathlib import Path

root = Path("fixtures/data-engineering/pipelines")
attempts = 0

def extract_with_transient_failure():
    global attempts
    attempts += 1
    if attempts == 1:
        raise TimeoutError("temporary source timeout")
    events = []
    for name in ("events-batch-1.jsonl", "events-batch-2.jsonl", "events-late.jsonl"):
        events.extend(json.loads(line) for line in (root / name).read_text().splitlines())
    return list({event["event_id"]: event for event in events}.values())

for attempt in range(1, 4):
    try:
        source_events = extract_with_transient_failure()
        break
    except TimeoutError:
        if attempt == 3:
            raise

published = {}
for day in ("2026-10-01", "2026-10-02"):
    replacement = defaultdict(Decimal)
    for event in source_events:
        if event["occurred_at"].startswith(day):
            replacement[event["customer_id"]] += Decimal(event["amount"])
    published[day] = dict(replacement)
totals = [(day, f"{sum(values.values()):.2f}") for day, values in sorted(published.items())]
print(attempts, totals)

The result is 2 [('2026-10-01', '35.50'), ('2026-10-02', '10.00')]. A real retry policy should add bounded exponential delay and jitter, but sleeping would make this verifier slower without teaching more. Never retry deterministic validation failures such as the malformed amount fixture unchanged; quarantine it with evidence or fix the source/code.

A backfill plan names the interval, code version, input snapshot, transformation version, concurrency limit, expected cost, validation query, publication method, and rollback. Backfills can compete with current workloads and overwrite newer corrections, so isolate or order them deliberately. Verify row counts, unique keys, partition coverage, totals, rejects, and downstream freshness before declaring recovery complete.

Limitations and common mistakes

• Unbounded retries turn a permanent error into load and alert noise.
• Retrying non-idempotent writes can duplicate business effects.
• Backfilling with “latest” inputs makes historical results irreproducible.
• Clearing a failed task without understanding partial output can publish mixed versions.
• Success status proves orchestration completion, not data correctness.

Practice

Write a recovery runbook for a bug affecting October 1–7. Include pause criteria, snapshot identity, backfill order, reconciliation, consumer communication, and rollback.

Next steps

Continue with Airflow DAG Fundamentals to express schedules, dependencies, retries, and historical runs in an orchestrator.

Primary sources

Checked 2026-10-15: Airflow backfill, Airflow task retry concepts, Airflow best practices, and Flink recovery semantics.