Skip to main content

Simulation-Based Optimization

Simulation is useful when a policy can be evaluated but its expected performance is inconvenient to express analytically. The simulation protocol becomes part of the experiment: random streams, replication budgets, stopping rules, and reported uncertainty must be fixed or disclosed.

Prerequisites​

Complete robust optimization.

Learning goals​

  • Use fixed random streams for reproducibility.
  • Use common random numbers when comparing policies on paired simulated conditions.
  • Keep optimization search, final evaluation, and out-of-sample validation separate.
  • Separate sampling error from model uncertainty and solver gap.

A fair comparison protocol​

The example draws one planning scenario and one recorded prediction-error pair per replication. Both candidate policies see the same 256 pairs. This pairing removes avoidable variation from the difference between policies; it does not make either policy correct.

The seed identifies the pseudo-random stream, while the scenario table and residual table define the simulated world. Changing either is a model change. Repeating more times reduces Monte Carlo sampling error, but cannot reveal omitted traffic, invalid probabilities, or a structural demand shift.

Worked example​

import numpy as np

scenarios = np.array([[4, 4], [7, 6], [9, 8]])
probabilities = np.array([0.2, 0.5, 0.3])
errors = np.array([[-1, 0], [0, -1], [0, 0], [1, 0], [0, 1], [1, 1]])
policies = {"sample_average": (2, 1), "box_robust": (1, 2)}
rng = np.random.default_rng(60)
scenario_draws = rng.choice(3, size=256, p=probabilities)
error_draws = rng.integers(0, len(errors), size=256)
demands = np.maximum(0, scenarios[scenario_draws] + errors[error_draws])

print("seed=60 replications=256 common_draws=True")
for name, (route_a, route_b) in policies.items():
capacity = np.array([4 * route_a + route_b, route_a + 4 * route_b])
shortages = np.maximum(0, demands - capacity).sum(axis=1)
costs = 80 * route_a + 70 * route_b + 50 * shortages
print(f"{name} mean_eur={costs.mean():.2f} p95_eur={np.quantile(costs, 0.95):.2f}")

Expected output:

seed=60 replications=256 common_draws=True
sample_average mean_eur=274.73 p95_eur=380.00
box_robust mean_eur=297.34 p95_eur=420.00

This sample favors the sample-average policy on both summaries. It does not prove dominance in the real operation. Report the seed and all policy results, not only the most favorable run.

Limitations and common mistakes​

  • Reusing the final evaluation stream while tuning policies overfits simulation noise.
  • One seed is reproducible but does not quantify sampling error; repeat independent streams or construct an appropriate interval when inference is required.
  • Common random numbers help differences only under suitable dependence; they are not a blanket variance guarantee.
  • A low simulated cost cannot close a solver gap or validate the simulator's assumptions.

Practice​

Run 20 independent outer seeds, retaining 256 paired replications per seed. Summarize the distribution of the paired mean-cost difference without selecting the best seed.

Next steps​

When exact search is too expensive, define honest budgets and evidence for heuristics and metaheuristics.

Primary sources​

Checked 2026-10-19: Kleinman, Spall, and Naiman, Simulation-Based Optimization with Stochastic Approximation Using Common Random Numbers and the author-hosted paper; NumPy 2.5, Generator and parallel reproducibility guidance.