Skip to main content

Batch vs. Streaming

Batch processing consumes a bounded collection and reaches a defined end. Stream processing treats input as unbounded and continuously updates results. A scheduled job every minute is still a series of bounded batches; low frequency does not make it a streaming engine. Conversely, a streaming system may internally process micro-batches while exposing continuous semantics.

Prerequisites​

Complete ETL vs. ELT. Keep the event grain—one logical event per event_id—in view.

Learning goals​

  • Distinguish boundedness from schedule frequency.
  • Relate at-most-once, at-least-once, and end-to-end exactly-once claims to loss and duplication.
  • Choose a mode from freshness, correctness, cost, and operational capability.
  • Explain why replayable sources and idempotent or transactional sinks matter.

Worked example​

This is a bounded simulation of two arrivals, not a broker or production stream. The repeated e-101 models an at-least-once source: the second delivery must not create a second logical event.

import json
from pathlib import Path

root = Path("fixtures/data-engineering/pipelines")
micro_batches = ["events-batch-1.jsonl", "events-batch-2.jsonl"]
seen = set()
snapshots = []
for name in micro_batches:
    for line in (root / name).read_text().splitlines():
        seen.add(json.loads(line)["event_id"])
    snapshots.append(len(seen))
print(snapshots)

The snapshots are [3, 5], although six records were delivered. At-most-once delivery may lose an event after acknowledging it too early. At-least-once delivery avoids loss by retrying but can duplicate effects. An exactly-once state guarantee does not automatically imply that every external side effect happens once. End-to-end exactly-once outcomes require compatible source replay, state recovery, and transactional or idempotent sinks.

Choose batch when a bounded input, periodic freshness target, and simple recovery satisfy the product. Choose streaming when decisions truly require continuous low-latency updates and the team can operate event-time semantics, state, checkpoints, backpressure, schema changes, and continuous monitoring. A faster schedule is often the lower-risk intermediate step.

Limitations and common mistakes​

  • “Real time” is not a requirement until it has a measured latency target and consequence.
  • Processing time is not event time; network delay and outages reorder arrivals.
  • Exactly-once language must name the boundary: operator state, database transaction, or complete business effect.
  • Streaming does not eliminate backfills; bugs and schema changes still require historical replay.
  • Airflow orchestrates bounded workflows; it is not a general-purpose streaming engine.

Practice​

For an hourly finance report and a fraud decision due in 200 milliseconds, write the maximum acceptable staleness, loss policy, duplicate policy, replay source, and sink guarantee. Do not choose technology until those are explicit.

Next steps​

Continue with Pipeline Dependency Graphs. Late-Arriving Data and Watermarks develops event-time behavior with a bounded simulation.

Primary sources​

Checked 2026-10-15: Apache Flink fault tolerance, Flink source and sink guarantees, Apache Beam basics, and Airflow overview.