Energy Grid — Load Forecasting into Unit Commitment

Commit The Fleet Before You Know The Weather

The same chain as inventory and staffing, at the scale where it is a real industry's daily practice. A grid operator must decide by mid-afternoon which generating units run tomorrow — the unit commitment problem — before knowing what the weather, and therefore the load, will do. Units have minimum up and down times, so the decision is a mixed-integer program and genuinely cannot be revised.

The data is GEFCom2012: 38,712 hourly observations from 2004 to May 2008, 20 grid zones plus the metered system total, and 11 weather stations, with no gaps and no missing values.

Two properties of the real file shape everything after it, and the notebook states both before modelling. The zones are not distinct — zones 2, 3 and 7 are the same series to a pairwise correlation of 1.00000, and zone 6 nearly so at 0.9994. The parts do not sum to the whole: the naive sum of the 20 zones averages 1,647 GW against a metered system total of 1,033 GW, a ratio of 0.63, precisely because the redundant feeds are counted repeatedly.

Load is not linear in anything. Mean demand is 1,279 GW below 40°F and 1,586 GW above 85°F, against 800 GW in the mild 55–65°F band — a U shape, because heating and cooling both consume power and the minimum sits at the balance point between them. The daily profile changes shape by season too: a single afternoon peak in summer, twin morning and evening peaks in winter.

Pooling Across The Grid — And Where It Does Not Help

The forecast is a hierarchical Bayesian model over the whole grid, and the honest reading of it is not the one a section like this usually offers. With roughly 35,000 hourly observations per zone, pooling does not beat independent fits: median WAPE is 9.99% pooled against 8.95% independent. WAPE is the weighted absolute percentage error — total absolute error divided by total actual load — which is the right accuracy measure here because, unlike a per-hour percentage, a quiet overnight hour with a small denominator cannot dominate the average. The data speak loudly enough that the prior barely moves them.

Where pooling earns its place is the case that breaks an independent fit. Zone 16 has never seen a hot day; its own cubic temperature response extrapolates wildly into weather it has no evidence about, giving a hot-day WAPE of 35.8%. Borrowing the grid's shared cooling response cuts that to 19.6%. That is the argument for hierarchy — not average accuracy, but not being catastrophically wrong where a zone is uninformative about itself. The single coherent posterior is also what makes the scenarios in the next section possible at all.

Because the zones are redundant, a bottom-up forecast is unusable: summing them naively gives a system WAPE of 49.0%. Learning reconciliation weights fixes it — 9.69% — and the learned weights on the redundant zones 2, 3, 6, 7 are 0.00, 0.00, 0.03, 0.00, which is the method discovering the duplication that section 1 flagged by hand.

The Money

Scenarios are drawn from the model's posterior predictive for one commitment day — several hundred complete versions of tomorrow, each one internally consistent across all 24 hours, rather than a single forecast curve with a band around it. The peak distribution is P50 1,992.7 GW, P95 2,204.1, P99 2,297.0 — the median, 95th and 99th percentiles of tomorrow's peak across the scenarios — and the distance between the median and the tail is the reserve the optimizer has to buy. The source matters: the Bayesian peak spread is 368 GW against a bootstrap's 699 GW, because weather-driven load moves together while resampled residuals do not.

Then the optimizer, over a fleet of 10 units with 1,662 MW of capacity. Committing for the point forecast alone is cheapest on paper and a disaster in expectation — it sheds 525 MWh and costs $5.72M once the true distribution arrives. A 15% reserve rule, which is what the rule of thumb prescribes, brings that to $873,890. Optimizing against the scenarios directly costs $504,444 and sheds nothing.

VSS — the value of the stochastic solution, what optimizing against the whole distribution saves over optimizing against its mean — is $369,446 per day, 42.3% of the reserve-rule cost. EVPI, the expected value of perfect information, is what a flawless forecast would save on top of that: $20,262, or 4.0%. The pair is the section's standard grading, introduced on the staffing example. The pattern from staffing repeats and hardens: modelling the uncertainty is worth eighteen times more than removing it. And the mechanism is visible in the commitment itself — the three plans commit 5, 7 and 9 of 10 units at the peak hour, so the stochastic plan is not simply buying more reserve than the rule, it is placing it where the scenarios say the risk actually is.

Notebook

References

• Hong, T., Pinson, P. & Fan, S. (2014). Global Energy Forecasting Competition 2012. International Journal of Forecasting 30(2), 357–363. — the dataset
• Hong, T. & Fan, S. (2016). Probabilistic electric load forecasting: a tutorial review. International Journal of Forecasting 32(3), 914–938. — the forecasting side
• Hyndman, R. J., Ahmed, R. A., Athanasopoulos, G. & Shang, H. L. (2011). Optimal combination forecasts for hierarchical time series. Computational Statistics & Data Analysis 55(9), 2579–2589. — reconciliation
• Takriti, S., Birge, J. R. & Long, E. (1996). A stochastic model for the unit commitment problem. IEEE Transactions on Power Systems 11(3), 1497–1508. — stochastic unit commitment
• Carrión, M. & Arroyo, J. M. (2006). A computationally efficient mixed-integer linear formulation for the thermal unit commitment problem. IEEE Transactions on Power Systems 21(3), 1371–1378. — the MILP formulation
• Birge, J. R. & Louveaux, F. (2011). Introduction to Stochastic Programming (2nd ed.). Springer. — VSS and EVPI again, at grid scale