Machine Learning: Operations Research
Every other section of this arc stops at the forecast. This one carries on to the decision the forecast exists to support, and grades the model on the money rather than on the loss.
The thesis is one sentence: a point forecast is the wrong object whenever being wrong in one direction costs more than being wrong in the other — which is nearly always. Stocking one unit short is a lost sale; stocking one unit over is a write-off, and the two rarely cost the same. Rostering one nurse short means a queue; one nurse over means an idle shift. Committing one generator short means shedding load; one over means burning fuel for nothing. In each case the quantity worth forecasting is the whole distribution, because the optimizer needs a shape, not a centre.
Two numbers recur through sub-arc A, and they are its spine. The value of the stochastic solution (VSS) is what optimizing against the distribution saves over optimizing against its mean — the payoff for admitting uncertainty exists. The expected value of perfect information (EVPI) is what a flawless forecast would save over the stochastic solution — the ceiling on every further modelling effort. Their ratio is the practical question: is the remaining gain in modelling the uncertainty, or in reducing it? On staffing the answer is 4.3% against 0.7%; on the grid, 42.3% against 4.0%. Both say the same thing, and it is not what a forecasting-accuracy mindset expects.
The later sub-arcs each break the frame in a different way, and both breakages are the interesting part. In B the classical answer is not wrong so much as self-concealing: an Erlang-C staffing plan (the textbook queueing formula) hits its 20% wait target exactly on a daily average while breaching it in 11 hours out of 24, because a stationary formula applied hour by hour assumes each hour starts with an empty queue and a three-hour service time means the small hours never do. The two B notebooks also check each other: the queue assumes service times are memoryless, and the survival model of length of stay returns a Weibull shape of 1.71, which says they are not. In C the failure moves into the objective itself. Delivery drivers look 15% wasteful against a distance-minimising route until you count zone switches, where the optimizer makes 41 and the driver 23 — they are optimizing something real that was left out of the objective. And in fraud, the entire gain comes from the decision layer: calibrating the score, deriving the threshold from costs and deferring the ambiguous cases to a human cuts realized cost 30% with no change whatsoever to the classifier.
Also worth stating plainly: this is where the honest negative results live. The hierarchical Bayesian forecaster loses to gradient boosting on data-rich products and wins only where data is scarce. Pooling across the grid's 20 zones is worse than fitting them independently on median accuracy, and earns its place on one zone that has never seen a hot day. The neural forecaster over-covers at every quantile. Further along, a hierarchical Bayesian calibrator fails to beat plain isotonic regression on the fraud scores, and a readmission model reaches an AUC of only 0.639 — reported as weak rather than dressed up, because a model that uncertain can inform triage but cannot carry a discharge decision. None of that weakens the thesis, because the thesis is about what you do with a distribution rather than about which model produces it.
Sub-arc A establishes that thesis on three problems of increasing scale, where the decision reduces to a quantity and the grading is in money. Sub-arc B changes the constraint to a queue: the decision becomes how much service capacity to build, and the grading moves from money to a service level and a bed count. Sub-arc C then breaks the pattern deliberately: the decision becomes a route or a threshold rather than a quantity, and in both cases the interesting failure turns out to be the objective rather than the model.
Sub-arc A — three problems, one question: what is the distribution worth?
Every value is committed notebook output. A comes from the inventory backtest, B from the two stochastic programs, C from the grid’s unit-commitment solve. Each panel carries its own units.
A is the thesis in one chart, and the ordering is the part worth pausing on. All three policies use the same fitted model on the same 16,800 product-days; they differ only in which part of its output they act on. Ordering the critical-fractile quantile costs £273,949, the point forecast £327,126, and the median £359,802 — worst of the three, despite being the estimator that minimises absolute error. Accuracy and usefulness are not the same axis.
B is why the section reports two numbers rather than one. VSS is what optimizing against the distribution saves over optimizing against its mean; EVPI is what a perfect forecast would save over that. On staffing the split is 4.3% against 0.7%, on the grid 42.3% against 4.0% — six-to-one and ten-to-one in favour of modelling the uncertainty rather than reducing it. Effort spent sharpening the forecast is bidding for the smaller bar in both cases, which is the opposite of where forecasting practice usually points.
C needs a log axis, and that is itself the finding. Committing generators for the bare point forecast costs $5.72M a day once the real demand distribution arrives, because it sheds load whenever demand lands high — an order of magnitude worse than anything else on the chart, and the reason no operator does it. The 15% reserve rule that operators do use lands at $873,890; optimizing over scenarios reaches $504,444 with zero shed, against a perfect-information floor of $484,182. The gap between the last two is small because the plan is already close to what perfect knowledge would buy.
How the examples relate — three sub-arcs
One sentence is true of all seven: turn an uncertain forecast into a decision, and measure its value in the units that matter. What changes across the sub-arcs is the kind of decision — a quantity to order, a queue to staff against, a route or a classification threshold — and therefore what “value” is denominated in. All three sub-arcs are built.
A · Forecast a distribution, then optimize
The optimizer needs a shape, not a centre. What is the distribution worth?
1 · Predict-then-Optimizeinventory: a quantile forecast into a newsvendor, and a backtest that prices the difference 2 · Stochastic Allocationstaffing: the canonical two-stage program, and VSS/EVPI as the pair that grades it 3 · Energy Gridhierarchical Bayesian load forecasting into unit commitment — scheduling generators that switch on or off whole, never fractionallyB · Capacity & congestion
When the constraint is a queue, the decision is how much service to build.
4 · Queueing — ED capacityErlang-C sized hour by hour, and a simulation showing it hits the daily average while failing 11 hours of 24 5 · Healthcare capacitylength of stay as a survival problem, and the arithmetic from that distribution to beds requiredC · Combinatorial & classification decisions
Where the decision is a route or a threshold rather than a quantity.
6 · Vehicle routingan exact mixed-integer program through OR-Tools to a learned search, then 6,112 real Amazon routes where the objective turns out to be wrong 7 · Fraud detectioncalibration, a cost-derived threshold and uncertainty triage — 30% saved with no change to the classifierRead in order, sub-arc A establishes the thesis on decisions that reduce to a quantity, and does so on problems where the optimizer is small enough to inspect: a single quantile, then a two-stage linear program, then a mixed-integer one. B tests the framing where the decision is how much service to build, and finds the classical formula failing in a way its own average hides. C is the sceptical case, since neither routing nor classification is obviously “forecast a distribution, then optimize against it” — and the thesis survives in an altered form: what has to be got right is still the objective, not the search or the score.
Predict-then-Optimize — Inventory
The full chain, end to end, on 1,067,371 real transactions from a UK gift wholesaler. Cleaning keeps 89.4% of rows and leaves 4,862 products, of which the Syntetos–Boylan scheme — which sorts demand by how often it is non-zero and how variable it is when it comes — finds 0 smooth and 4,015 lumpy: a catalogue with essentially no product that sells steadily, and therefore no product a mean describes well. One notebook runs the whole pipeline at that catalogue scale; the figures below come from a 200-product subset small enough to inspect end to end. Nine quantiles are fit under the pinball loss — the asymmetric penalty that makes a model estimate a chosen quantile instead of a mean — and land at 0.977 coverage against a nominal 0.975, exactly where the decision lives. The newsvendor turns them into an order at the critical fractile, the quantile fixed by how much an unsold unit costs against a missed sale, which spans 0.38 to 0.98 across products because real prices drive it. The backtest is the point: over 16,800 product-days, ordering the fractile quantile costs £273,949 against £327,126 for the point forecast, a 16.3% cut, cheapest in 3 of 3 folds. Ordering the median is worst of all — the best point forecast under absolute error, and the wrong quantity to stock.
View example →Stochastic Allocation — Staffing
The same idea in its canonical form: a two-stage stochastic program on real emergency-department arrivals, where the roster is committed before the day's demand is known and the only recourse left is expensive. Scenarios are drawn from a Bayesian posterior predictive — the model’s own distribution over days it has not yet seen — rather than a rule of thumb, and the arrival series carries three nested seasonalities at once. Over 41 held-out weeks the stochastic plan is at least as cheap as the expected-value plan in 39, with VSS $2,006 (4.3%) against EVPI $324 (0.7%). That gap is the finding: modelling the uncertainty is worth six times more than eliminating it, so a team pouring effort into forecast accuracy while solving the deterministic problem has the priority backwards. The three costs satisfy WS ≤ RP ≤ EEV — perfect foresight (wait-and-see), the stochastic plan (recourse problem), and the plan built on the mean forecast (expected value of the EV solution). That ordering is a theorem, so it doubles as a check on the code.
View example →Energy Grid — Load Forecasting into Unit Commitment
The technical peak of the sub-arc, on GEFCom2012: 38,712 hourly observations, 20 grid zones and 11 weather stations. The real file is awkward in two ways the notebook states before modelling — zones 2, 3 and 7 are the same series to a correlation of 1.00000, and the zones sum to 1,647 GW against a metered total of 1,033. A hierarchical Bayesian model over the whole grid does not beat independent fits on median accuracy (9.99% against 8.95%), and earns its place on the zone that has never seen a hot day, where borrowing the grid's cooling response cuts hot-day error from 35.8% to 19.6%. Reconciliation then learns weights of 0.00 on three of the four duplicated feeds and 0.03 on the last, rediscovering the redundancy by itself. Feeding those scenarios into the commitment problem — which is hard precisely because a generator is on or off, never 40% on — gives VSS $369,446/day — 42.3% of the cost of the 15% reserve rule operators actually use, against an EVPI of 4.0%.
View example →Queueing — Emergency Department Capacity
The same real ED arrivals as the staffing example, now with the queue itself as the constraint. Erlang-C — the classical formula for how many servers hold the chance of waiting below a target — sizes the evening peak at 23 servers, five more than the 18 Erlangs of offered load would suggest. (An Erlang is the bare work arriving: arrivals per hour times hours per case, so 18 Erlangs is 18 cases’ worth of service demanded every hour.) Applying it hour by hour gives a 410 server-hour day. Then a from-scratch discrete-event simulation shows what the formula cannot: the plan lands exactly on its target in aggregate — the chance an arrival has to wait, P(wait), is 0.20 — while breaching it in 11 hours of 24, worst at 01:00, where P(wait) hits 0.73 and the mean wait is 110 minutes. The cause is structural — a three-hour service time means beds are still full of the evening surge after midnight, but a stationary formula sized those hours on their own low arrival rate, assuming each period starts empty.
View example →Healthcare Capacity — Survival Meets Bed Planning
The service side of the same loop, earning the number the queueing example assumed. Length of stay on 101,766 encounters is treated as time-to-event: Kaplan–Meier for the discharge curve, Cox for who leaves sooner (concordance 0.689), and competing risks to separate discharge from death rather than treating a death as a longer stay. A Bayesian Weibull returns shape k = 1.71, so the discharge hazard rises with days already spent — length of stay is not memoryless, which is exactly what the M/M/c queue next door assumes — the c-server model whose exponential service times are what make Erlang-C solvable in closed form. The capacity link follows: 53 beds on average, 65 for a 95% service level, and a 10% cut in length of stay saves 6 beds. The readmission model is reported with its weak AUC of 0.639 rather than dressed up.
View example →Vehicle Routing — From Exact Optimization To Real Drivers
A methods ladder, then a reality check. An exact mixed-integer program returns 217.2 with an optimality certificate on nine customers and does not scale; Clarke–Wright savings plus 2-opt — build routes by merging the pairs that save most, then repeatedly uncross any two edges — matches it there and does 100 customers in 4 ms; OR-Tools is 1.1% shorter and 2,000× slower; and a learned edge classifier prunes the graph to 14% of pairs while covering 92% of the solution arcs, finding 1,164 in 3 ms. Then 6,112 real Amazon routes: drivers run a median 1.15× the shortest tour that visits every stop — the travelling-salesman optimum — and look 15% wasteful — until you count zone switches, where the optimizer makes 41 and the driver 23. A zone-aware objective costs 1.06× the distance and switches 21 times, beating the drivers at their own criterion. The objective was wrong, not the drivers.
View example →Fraud Detection — Cost-Sensitive Decisions
At 0.173% prevalence a do-nothing model is 99.83% accurate and both a booster and a logistic regression post ROC-AUC 0.98 — the chance the model ranks a random fraud above a random legitimate charge — so the scoreboard has to go. Calibration first — isotonic regression, which re-maps the scores through the best non-decreasing curve, cuts expected calibration error (ECE, the average gap between a predicted probability and the rate actually seen) from 0.183 to 0.042, and a hierarchical Bayesian calibrator does not beat it, earning its place only through the credible interval it attaches to each transaction. Then a threshold derived from two costs rather than convention, landing at 0.278, and finally triage that sends the 0.08% of cases where the posterior straddles the threshold to a human. The ladder is the point: $4,193 → $2,922, a 30% reduction, with no change whatsoever to the classifier. The companion notebook takes the same layer to IEEE-CIS, where a drifting relative-timedelta field and 13,553 cards resolving into 199,070 entities are the actual work.
View example →