Skip to main content

Configuration Without Hidden State

Configuration is the set of choices that may vary while the training algorithm remains the same: input path, schema, feature list, split, seed, estimator, and hyperparameters. Hidden configuration lives in edited notebook cells, implicit defaults, environment variables, or filenames with meanings only one person knows.

Prerequisites​

Complete Reproducible Environments. Be able to read JSON and identify model hyperparameters.

Learning goals​

  • separate code, configuration, secrets, and observed results.
  • validate configuration before expensive work starts.
  • preserve the effective configuration in Study Sprint lineage.

One source of truth per choice​

The Study Sprint configuration names every feature and eight holdout IDs. The split is therefore stable even if CSV row order changes. random_state is explicit for stochastic estimator behavior, although the explicit IDs—not the seed—define this holdout.

{
  "data_path": "fixtures/ml-engineering/study-sprint-model.csv",
  "numeric_features": ["prior_score", "practice_minutes"],
  "categorical_features": ["learning_track"],
  "test_ids": ["s04", "s09", "s14", "s19", "s24", "s29", "s34", "s39"],
  "random_state": 39,
  "model": {
    "type": "logistic_regression",
    "C": 1.0,
    "max_iter": 500,
    "solver": "liblinear"
  }
}

Defaults are still choices. A training program should either serialize the resolved default or reject its omission. Otherwise a library upgrade can silently change the effective run.

Worked example​

The public command receives paths and code identity as CLI arguments, then reads modeling choices from JSON:

.venv-content/bin/python examples/ml-engineering/train.py \
  --config examples/ml-engineering/training-config.json \
  --output-dir /tmp/study-sprint-run \
  --code-version ticket-39-fixture-v1

Before fitting, it checks required columns, unique non-empty IDs, numeric and categorical values, binary targets, declared holdout IDs, and both target classes in both partitions. Invalid input exits with status 2 and an explanation; it does not publish a partial successful run.

Secrets are different from configuration. Credentials may enter through a secret manager or protected environment boundary, but they must never be copied into a run file. Record the non-secret resource identity and access policy instead of the secret value.

Configuration precedence​

If a system supports files, environment variables, and CLI overrides, define and log a precedence order. Preserve the resolved values, not merely the base file. This lesson avoids merging layers: JSON owns modeling choices; three required CLI arguments own the config path, output path, and code version. The narrow contract is easier to audit.

Configuration validation cannot prove that a feature is ethical, available at inference time, or free of leakage. Those are data and system-design questions.

Common mistakes​

  • Using environment variables for ordinary model parameters, then forgetting to record them.
  • Letting library defaults change without appearing in a diff.
  • Storing API keys beside hyperparameters.
  • Generating a fresh random seed inside the training process.
  • Defining a test fraction but not retaining which entities entered the holdout.

Practice​

Propose a new decision threshold for Study Sprint. Decide whether it belongs in training configuration, the serialized model contract, or a later serving policy. Explain how a reviewer discovers its effective value for one prediction.

Next steps​

Data and Code Versioning gives declared inputs stable identities rather than mutable names.

Primary sources​

Checked 2026-10-15: Python argparse documentation, the JSON standard, RFC 8259, The Twelve-Factor App configuration principle, and scikit-learn guidance on controlling randomness.