Airflow DAG Fundamentals

Apache Airflow programmatically authors, schedules, and monitors bounded workflows. Airflow 3.3.2 is the stable release checked for this lesson, and airflow.sdk is the public DAG-authoring interface for Airflow 3.x. Airflow coordinates work; it is not a streaming engine, data transport, or substitute for idempotent transformation code.

Prerequisites

Complete Retries, Backfills, and Recovery and Pipeline Dependency Graphs.

Learning goals

• Define a DAG and TaskFlow tasks with maintained public imports.
• Connect logical intervals to deterministic partitions.
• Configure bounded retries without hiding permanent failures.
• Keep large data out of orchestration metadata.

Worked example

This DAG passes only artifact locations between tasks. In a project, validate_partition and publish_partition should be ordinary tested functions or commands; Airflow supplies context, scheduling, retries, state, and observability around them.

from datetime import datetime, timedelta, timezone

from airflow.sdk import dag, get_current_context, task


@dag(
    dag_id="local_event_pipeline",
    schedule="@daily",
    start_date=datetime(2026, 10, 1, tzinfo=timezone.utc),
    catchup=False,
    tags=["data-engineering"],
)
def local_event_pipeline():
    @task(retries=2, retry_delay=timedelta(minutes=5))
    def locate_input() -> str:
        context = get_current_context()
        day = context["data_interval_start"].date().isoformat()
        return f"/data/raw/occurred_date={day}/events.jsonl"

    @task
    def validate(input_path: str) -> str:
        # Call project-owned validation code; return a small artifact reference.
        return input_path.replace("/raw/", "/validated/")

    @task
    def publish(validated_path: str) -> str:
        # Project code atomically replaces this interval's output.
        return validated_path.replace("/validated/", "/published/")

    publish(validate(locate_input()))


local_event_pipeline()

The repository verifier parses this example, checks the airflow.sdk public import, rejects legacy/internal authoring imports, and confirms its task dependency chain. It does not install or start Airflow: a production deployment also needs a supported Python version, constraints-selected dependencies, metadata services, an executor, secrets, monitoring, and operational ownership.

catchup=False prevents automatic creation of all past scheduled runs when this DAG is first enabled; it does not disable deliberate backfills. For a backfill, select explicit dates and Airflow's reprocessing behavior, then rely on the partition-specific idempotence established in earlier lessons. A task retry repeats the same logical interval, so task code should use data_interval_start or another explicit partition rather than wall-clock “now.”

Keep DAG files lightweight. Avoid network calls, database queries, and expensive computation at module import time because the scheduler parses them repeatedly. Pass small control values or artifact references between tasks, not full datasets.

Limitations and common mistakes

• Airflow is designed for finite batch workflows, not continuously running stream processing.
• A dependency arrow controls order; it does not move data or guarantee data quality.
• catchup=False is not a recovery strategy and does not make tasks idempotent.
• Internal model or metadata-database imports are not stable DAG-author interfaces.
• A syntactically valid DAG still needs deployment-level integration tests and operational configuration.

Practice

Add a quality-report task after validation and decide whether publication must wait for it. Then describe how you would create a two-day backfill without changing the DAG's schedule.

Next steps

Continue with Pipeline Observability to define the evidence needed to operate this workflow.

Primary sources

Checked 2026-10-15: Apache Airflow 3.3.2 release metadata, Airflow 3 public interface, Task SDK, DAG concepts, backfill, and best practices.