Workflow Orchestration for ML

An orchestrator coordinates when independently testable work runs, with dependencies, retries, concurrency, and visibility. It should call the same Study Sprint commands used locally and in CI. Training logic hidden inside scheduler callbacks is harder to reproduce, test, and migrate.

Prerequisites

Complete CI for Machine Learning and the Data Engineering lesson on Airflow DAG Fundamentals.

Learning goals

• separate orchestration policy from training, evaluation, and promotion logic.
• choose retry, backfill, concurrency, and idempotency behavior per task.
• recognize when a CI job, script, or scheduler is sufficient.

Worked example

Apache Airflow 3.3.2 is the stable release checked for this lesson; 3.4.0b1 is a prerelease and is not the baseline. Airflow 3's public authoring interface is airflow.sdk:

from airflow.sdk import dag, task

@dag(dag_id="study_sprint_candidate", schedule=None, catchup=False)
def study_sprint_candidate():
    @task
    def train_and_evaluate() -> str:
        # Invoke the versioned project CLI and return an immutable evidence URI.
        return "artifacts/run-identity/lifecycle-report.json"

    @task
    def request_promotion(evidence_uri: str) -> None:
        # Record a request; do not silently move a production alias.
        print(evidence_uri)

    request_promotion(train_and_evaluate())

study_sprint_candidate()

This is a manual boundary: Airflow is not installed by this batch and the snippet is statically reviewed rather than scheduled. The repository already documents and checks Airflow 3.3.2 for Data Engineering; installing the scheduler, API server, metadata database, and providers would not improve the local capstone.

Define task inputs and outputs with immutable identities. Make reruns idempotent, cap retries, distinguish transient infrastructure failures from invalid data, bound backfills, prevent overlapping promotions, and surface cancellation. Keep promotion criteria in versioned project code, then make alias movement an explicit authorized task with rollback evidence.

A script is sufficient for one manually triggered lifecycle. CI scheduling is sufficient when the work is short, repository-scoped, and needs no complex backfill. An orchestrator becomes useful for many dependent workflows, calendars, queues, retries, cross-system ownership, and operational visibility—provided a team owns it.

Common mistakes

• Importing private Airflow internals instead of airflow.sdk.
• Retrying deterministic validation failures until they waste capacity.
• Passing large model bytes through task metadata instead of immutable storage.
• Allowing concurrent runs to race on one mutable production alias.
• Treating scheduler success as proof that promotion criteria passed.

Practice

Write retry and idempotency rules for train, register, deploy-check, monitor, promote, and rollback tasks. Mark which task requires human authorization and which can be replayed safely.

Next steps

Continue with Model Cards and Operational Documentation to package decisions and limitations with the artifact.

Primary sources

Checked 2026-10-15: Airflow public interface, Airflow Task SDK API, Airflow best practices, and PyPI Apache Airflow 3.3.2.