Combining Predictive Models with Optimization
Predict-then-optimize systems use estimates—demand, travel time, risk, or response—as coefficients or right-hand sides in a downstream decision model. Prediction quality matters through the decisions it changes, not only through a generic error score.
Prerequisites
Complete validating optimization models and review from prediction to decision.
Learning goals
- Define the prediction-to-parameter interface with units and timing.
- Prevent data leakage across training, tuning, decision selection, and final evaluation.
- Compare predictive error with realized decision cost and decision regret.
- Stress-test prediction error, model uncertainty, and optimization assumptions separately.
The predictive-prescriptive boundary
Suppose a model predicts east and west demand before route runs are chosen. The optimizer maps that prediction to a plan. On a held-out case with realized cost ,
The oracle term is a retrospective benchmark, not an implementable policy: it uses actual demand that was unavailable at decision time. Decision regret can reveal asymmetric consequences that mean absolute error hides.
Use temporal or grouped splits that match deployment. Fit preprocessing and predictive models only on training data, tune the full predictive-plus-optimization pipeline on validation data, freeze it, and report once on untouched final cases. Selecting a model because it minimizes final-test regret is data leakage.
Worked example
from itertools import product
plans = [(a, b) for a, b in product(range(4), repeat=2)
if a + b <= 3 and 5 * a + 4 * b <= 14]
def cost(plan, demand):
a, b = plan
shortage = max(0, demand[0] - (4 * a + b)) + max(0, demand[1] - (a + 4 * b))
return 80 * a + 70 * b + 50 * shortage
def choose(prediction):
return min(plans, key=lambda plan: (cost(plan, prediction), plan))
actual = (9, 8)
oracle_cost = min(cost(plan, actual) for plan in plans)
for name, prediction in [("model_a", (8, 8)), ("model_b", (10, 6))]:
plan = choose(prediction)
mae = sum(abs(prediction[i] - actual[i]) for i in range(2)) / 2
realized = cost(plan, actual)
print(f"{name} mae_packages={mae:.1f} plan={plan} realized_cost_eur={realized} regret_eur={realized - oracle_cost}")
print("holdout_only=True lower_prediction_error_wins_decision=False")
Expected output:
model_a mae_packages=0.5 plan=(1, 2) realized_cost_eur=370 regret_eur=40
model_b mae_packages=1.5 plan=(2, 1) realized_cost_eur=330 regret_eur=0
holdout_only=True lower_prediction_error_wins_decision=False
Model A has lower MAE but crosses a decision boundary in the costly direction. This single synthetic case demonstrates the possibility; it is not evidence that Model B generalizes better.
Limitations and common mistakes
- Decision-focused metrics do not eliminate the need for calibration, subgroup checks, and predictive diagnostics.
- Optimizing against a biased predictor can amplify harm where constraints or penalties are asymmetric.
- The oracle benchmark may itself omit real costs and constraints.
- Retraining frequency, feedback loops, intervention effects, and changing demand can invalidate historical error distributions.
Practice
Enumerate a grid of east/west predictions and draw the regions that select each plan. Identify where a one-package prediction change causes a discontinuous decision change.
Next steps
Integrate planning, solver evidence, invariant checks, stress tests, and fallback reporting in the constrained-delivery project.
Primary sources
Checked 2026-10-19: Elmachtoub and Grigas, Smart “Predict, then Optimize” and the Management Science article; scikit-learn 1.9, cross-validation guidance and common pitfalls including data leakage.