Skip to main content

Pipeline Dependency Graphs

A directed acyclic graph (DAG) describes precedence: an edge from validation to publication means publication may run only after validation succeeds. It does not describe where code executes, how data moves, or whether a task is idempotent. Those are separate contracts.

Prerequisites​

Complete Batch vs. Streaming. The event pipeline now has four stages: land, validate, aggregate, and publish.

Learning goals​

  • Distinguish control dependencies from data dependencies.
  • Detect cycles and derive a valid execution order.
  • Choose task boundaries around observable, retryable outputs.
  • Avoid graphs that pass large data through an orchestrator's metadata channel.

Worked example​

The graph below names dependencies only. The topological sort is deterministic because ready tasks are sorted; a production scheduler may run independent tasks concurrently.

dependencies = {
    "land": set(),
    "validate": {"land"},
    "aggregate": {"validate"},
    "publish": {"aggregate"},
    "quality_report": {"validate"},
}
remaining = {task: set(needs) for task, needs in dependencies.items()}
order = []
while remaining:
    ready = sorted(task for task, needs in remaining.items() if not needs)
    if not ready:
        raise ValueError("cycle detected")
    order.extend(ready)
    for task in ready:
        remaining.pop(task)
    for needs in remaining.values():
        needs.difference_update(ready)
print(order)

The order is ['land', 'validate', 'aggregate', 'quality_report', 'publish']. quality_report could run after validation without waiting for aggregation. A graph edge should represent a real prerequisite, not a preference for how the diagram looks.

Each node needs an input contract, output contract, partition identity, owner, timeout, retry policy, and success condition. Prefer tasks that publish one complete artifact or transaction. Very large tasks hide which stage failed; tiny tasks create scheduling overhead and complicated partial states. Put boundaries where work can be independently observed and safely retried.

Data should move through durable storage or a well-defined service. Orchestrator metadata is suitable for small control values such as a partition name or manifest location, not an entire dataset.

Limitations and common mistakes​

  • Acyclic does not mean correct: the graph can omit a necessary dependency.
  • File existence alone may not prove completeness; publish a manifest, checksum, or transactional marker.
  • Dynamic task generation must remain bounded and observable.
  • Cross-workflow dependencies can create hidden coupling and difficult recovery.
  • A DAG is a model of ordering, not a streaming transport or transformation engine.

Practice​

Add a quarantine task for invalid rows. Decide whether publish should wait for it, and justify the choice from the publication contract rather than diagram symmetry.

Next steps​

Continue with Idempotent Data Jobs so every node can survive a retry. Later, Airflow DAG Fundamentals maps these concepts to a maintained orchestration interface.

Primary sources​

Checked 2026-10-15: Airflow DAG concepts, Airflow task concepts, and Airflow best practices.