← All examples

Machine Learning: Time-Series — ML vs Econometrics

Most comparisons of this kind are rigged by accident, and the interesting part of the exercise is refusing to rig them. A forecast origin is a draw: run the same models on a window ending a year earlier and the ranking can reorder completely, so a single holdout supports far less than it appears to. Features that look innocuous can be inert — hand a gradient-boosted tree a raw trend index and it contributes exactly nothing beyond the last training observation, because a tree emits a constant past its final split. And a recursive multi-step forecast compounds its own errors in a way a model with a trend state does not.

So each example here scores multiple origins, states what the classical models get for free that the machine-learning ones must be handed, and reports where the answer flips. The through-line is that the advantage is conditional, and the conditions are legible: how many series, how much data per series, whether exogenous information exists, and how much signal is there at all.

The first example sets the bar deliberately high — one clean, strongly seasonal series, exactly what SARIMA (seasonal ARIMA) and exponential smoothing were designed for. The later ones move to the regimes where that bar comes down.

The failure mode all of this is guarding against. It has a name that never gets used above: look-ahead bias, and its general form, leakage — letting a model see, at fit time, information that would not have existed at the moment the forecast was due. It is rarely as blatant as training on the test set. It arrives through standardising with a mean computed over the whole sample, through a feature built from a revised data vintage rather than the first release, through picking hyperparameters by looking at the holdout, or simply through the analyst having read the news for the period being forecast. Every design choice below — refitting at each step rather than once, choosing the origin before seeing results, scoring several origins — exists to close one of these doors. The reason to name it is that leakage does not announce itself: it shows up as a model that looks unusually good, which is exactly the outcome nobody investigates.

What “out of sample” has to mean when time is involved. Ordinary cross-validation shuffles rows, which for a time series means training on the future to predict the past. The replacement is a walk-forward or expanding-window evaluation: fit on everything up to time t, predict t+1, move forward, refit, repeat. The point at which the fitting stops is the forecast origin. Two further distinctions decide what a number means. A one-step forecast is handed the true lags at every step, so nothing compounds; a multi-step path feeds the model its own predictions, and errors accumulate. And a multi-step forecast can be built two ways — recursively, iterating the one-step model forward, or directly, fitting a separate model per horizon, which avoids the compounding at the cost of estimating more. These are not bookkeeping details. On the panel example the same exponential-smoothing model scores 0.478 as a one-step forecaster and 0.757 fitted once and extrapolated as a single 40-step path — a 58% swing from the protocol alone, wider than the gap between any two models on that page.

Stationarity, differencing, and why the baselines change shape. Most of these methods assume the series is stationary — that its mean, variance and autocorrelations do not drift with time. Real macro and price series are not, so they are transformed until they roughly are, usually by differencing: modelling the change from one period to the next instead of the level. That transformation quietly redefines the naive benchmark, which is the single most common way these tables get misread. A persistence forecast repeats the last observed value, with nothing estimated at all. On a series in levels that means “expect no change.” On an already-differenced series the last observed value is last period’s change, so persistence now asserts that the change repeats — a real prediction, and usually a worse one. Both appear in the macro table, and they are 0.7207 and 0.6291: the same word would have covered two different baselines nine hundredths apart. The seasonal analogue, seasonal-naive, repeats the value from one full season ago, and on strongly seasonal data it is a far harder benchmark than it sounds.

Local, global, and the two competitions everyone cites. Given many related series there are two strategies. A local approach fits a separate model to each series, which sees only its own history. A global approach trains one model across all of them pooled together, sharing parameters — also called cross-learning, on the argument that it borrows strength, learning patterns from every series to help each one. The evidence usually cited is a pair of forecasting competitions run by Spyros Makridakis. M4 (2018) ran 100,000 largely unrelated series and did not favour machine learning: several pure ML entries finished below a seasonal-naive benchmark and the winner was a hybrid. M5 (2020) ran roughly 42,840 Walmart sales series in one hierarchy and flipped it completely — all fifty top entries used machine learning. The panel example below tests which regime is responsible, and finds that on a panel of long series the gain comes from model flexibility rather than from pooling at all.

Reading the numbers. Accuracy here is RMSE, the root mean squared error, in the units of whatever is being forecast; the M-competitions instead use scale-free measures such as MASE (mean absolute scaled error, the error divided by the naive forecast’s), which matter when averaging across series measured in different units and matter less here, where each comparison is within one series or one standardised panel. A difference in RMSE is not automatically a finding, so the comparisons are tested: the Diebold–Mariano test takes the two models’ squared errors on the same dates, forms the difference series, and asks whether its mean is distinguishable from zero once autocorrelation is allowed for. It behaves like a t-statistic — beyond about ±1.96 the ordering is real, inside it the two models are tied however different the RMSEs look. For direction rather than magnitude the metric is AUC, the area under the receiver-operating-characteristic curve, which is the probability a randomly chosen up-day is scored above a randomly chosen down-day; 0.5 is a coin flip. And the recurring econometric benchmark on volatility is HAR, the heterogeneous autoregressive model — a regression on yesterday, last week and last month, three terms, which turns out to be remarkably hard to beat.

The cast. SARIMA (seasonal autoregressive integrated moving average) and ETS (exponential smoothing, in its error-trend-seasonal form) are the classical univariate workhorses. A VAR (vector autoregression) extends autoregression to several series that influence each other, and a BVAR (Bayesian VAR) is the same system with a shrinkage prior on its coefficients. On the machine-learning side sit gradient-boosted trees and an LSTM (long short-term memory), a recurrent network with gates designed to carry information across long gaps. Each is treated in depth elsewhere in the collection; here they are contestants.