Skip to main content

CI for Machine Learning

ML CI must test ordinary software behavior and the statistical contracts that ordinary unit tests miss. It should reject broken lineage, leakage, changed fixture outputs, incompatible inference schemas, and missing monitoring evidence before merge. It should not silently promote a model from untrusted pull-request code.

Prerequisites​

Complete Feature Stores and Feast and Testing Data and ML Code.

Learning goals​

  • layer fast schema and code checks before integration and build checks.
  • isolate untrusted changes from secrets and write permissions.
  • keep promotion criteria separate from successful CI execution.

Worked example​

The Study Sprint public checks are ordinary commands and therefore fit a clean CI environment:

permissions:
  contents: read

jobs:
  ml-contracts:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          persist-credentials: false
      - name: Install pinned verification environment
        run: python3 -m pip install --requirement requirements-content.txt
      - name: Verify the complete local lifecycle
        run: |
          npm run test:ml-engineering-reproducibility
          npm run test:ml-engineering-serving
          npm run test:ml-engineering-monitoring
          npm run test:ml-engineering-platform

This is a manual boundary: the workflow fragment is reviewed against the repository's CI contract, but this page does not create a second CI run or grant deployment credentials. The local verifier executes the four listed public commands instead.

The repository's real workflow also locks Node/npm, uses a clean npm install, pins actions to full commit SHAs, and grants read-only repository permission. Python dependencies used by executed examples are exact pins. Cache only public dependency downloads; model or data caches need separate integrity, privacy, poisoning, and retention decisions.

Run cheap deterministic checks first, then integration and production-build checks. Report promotion evidence as an artifact, but require a distinct governed step to change a registry alias or deployment. Pull requests from forks or unreviewed branches must have no secret and no production write path. Environments, reviewers, short-lived credentials, and least privilege constrain a later release job.

A local pre-merge script can be sufficient for one repository if a protected branch enforces it. A CI service becomes valuable for clean-environment evidence, review visibility, concurrency, and policy enforcement; it still needs owners, budgets, retention, and incident handling.

Common mistakes​

  • Giving a test job deployment credentials “for convenience.”
  • Passing because code tests run while data, model, and serving contracts do not.
  • Downloading mutable datasets or models during a supposedly repeatable gate.
  • Pinning package names but not lockfiles, action SHAs, or runtime versions.
  • Equating a green check with approval to promote.

Practice​

Classify every current quality command as fast, integration, content, build, or browser evidence. Decide the earliest safe failure point and which jobs must never receive secrets.

Next steps​

Continue with Workflow Orchestration for ML to place the same idempotent commands on a schedule or event boundary.

Primary sources​

Checked 2026-10-15: GitHub Actions secure use, GitHub token permissions, SLSA provenance 1.2, and NIST Secure Software Development Framework 1.1.