Financial Returns — the Honesty Capstone
Python · scikit-learn · XGBoost · Download S&P data
What Is and Is Not Predictable
The hardest and most consequential case in quantitative finance, and the one where the discipline of the previous three examples pays off: can machine learning predict asset returns? Markets are close to efficient — if tomorrow's direction were forecastable, the trade would already have been made — so the honest expectation is that nothing extracts much signal. Testing that rigorously, and resisting the several ways one fools oneself, is the actual skill on display.
The starting fact: on S&P 500 daily data, returns have autocorrelation of −0.087 while log-volatility has +0.784. Autocorrelation is how much today's value tells you about tomorrow's, on a scale where 0 is nothing at all and 1 is perfect — so returns are close to white noise, a series with no usable structure left in its own history, while volatility is strongly self-predicting, with calm days following calm days and violent ones following violent ones. Any honest return predictor begins from that asymmetry.
Logistic regression, XGBoost and a random forest, all tasked with the sign of tomorrow's return, land at AUC — area under the receiver-operating-characteristic curve, the probability a randomly chosen up-day is ranked above a randomly chosen down-day, so 0.5 is a coin flip — of 0.480, 0.503 and 0.487. That is chance. More tellingly, none clears the "always predict up" baseline: the market rises on 56.3% of test days, and the best model manages 52.5% accuracy. This is not a modelling failure; it is the efficient market.
| next-day return direction, out of sample | AUC | accuracy |
|---|---|---|
| logistic regression | 0.480 | 0.525 |
| XGBoost | 0.503 | 0.522 |
| random forest | 0.487 | 0.504 |
| "always predict up" | 0.500 | 0.563 — the bar none of them clears |
The in-sample mirage
The trap that has ended countless strategies, in two numbers. Give a flexible model enough capacity and it fits the training returns perfectly — in-sample AUC 1.000 — and out of sample it is back at 0.499. The in-sample "signal" is memorised noise. With enough features, models and hyperparameter tries, one configuration will look excellent by chance, which is why the defences exist: purged cross-validation, which drops the training observations sitting close enough in time to the test set to share information with it, and the deflated Sharpe ratio, which discounts a result by how many configurations were tried before it appeared.
Where the Predictability Actually Lives
The productive contrast: the same features and model that explain none of next-day returns (R² −0.128) explain a large share of next-day volatility (+0.458). R² is the fraction of the variation a model accounts for: 1 is perfect, 0 is no better than always guessing the average, and a negative value means the model is worse than that constant guess — which is where the return model lands. That split is the general truth about machine learning in finance. What is predictable is how much prices move rather than which way (the second moment — variance — rather than the mean), along with relative performance across many assets, execution, and shifts in market regime. Not tomorrow's direction.
Having spent two sections insisting that models be judged against the right baseline, the booster faces the same test here. On the identical split: random walk +0.306, AR(1) +0.393, and a three-term HAR regression at +0.518 (HAR is heterogeneous autoregressive: a linear fit on yesterday's, last week's and last month's realised volatility, on the theory that traders operate on those three horizons). The contrast that matters survives intact — +0.458 against −0.128 is the difference between a target with structure and one without — but the booster is not the best way to exploit that structure, and HAR beats it by 0.060, matching what the LSTM example found on this same series. "Point ML at the right target" is necessary advice, not sufficient advice.
| out-of-sample R², same split | next-day return | next-day volatility |
|---|---|---|
| XGBoost, 10 features | −0.128 | +0.458 |
| random walk | — | +0.306 |
| AR(1) | — | +0.393 |
| HAR (day + week + month, 3 coefficients) | — | +0.518 |
The Strategy That Looks Like It Works
Finally the test that decides it. Turn the best direction model into a strategy — long when it predicts up, short when down — and compare with buy-and-hold. The comparison is in Sharpe ratios — annualised return divided by annualised volatility, so the return earned per unit of risk taken. It is the standard yardstick because it refuses to reward a strategy for simply taking bigger positions; roughly speaking 0 is nothing, around 0.5 is what holding the market has paid historically, and sustained values above 1 are rare. Gross Sharpe 0.74 against buy-and-hold's 0.72 reads like a faint edge. It is not one.
The model is long on 65% of days, and a position that is long most of the time simply inherits the market's drift — the tendency of equities to rise on average over long periods, which any mostly-long position collects without predicting anything: "always long" scores 0.72 by construction. A gross Sharpe sitting on top of buy-and-hold is exactly what no skill produces when the strategy is mostly long. What then destroys it is turnover — 430 position changes in 1,031 days, a cumulative cost drag of 43% of capital at 5 basis points a side (a basis point is one hundredth of a percent, so five of them is 0.05% of the traded value each time the position turns over — a deliberately optimistic assumption), taking net Sharpe to 0.13.
| annualised Sharpe, test period | value | |
|---|---|---|
| ML strategy, gross | 0.74 | looks like an edge |
| buy and hold | 0.72 | |
| "always long" | 0.72 | identical to buy-and-hold by construction |
| ML strategy, net of 5bps | 0.13 | 430 position changes, 43% cost drag |
Two separate lessons, and the second is the easier to miss. Costs bury a faint edge — everyone knows that. But before costs there was no edge to bury: the gross return came from being long in a rising market, not from prediction. A backtest reporting only the gross Sharpe would have looked like a working strategy.
The Section's Answer
That closes the section's question with a conditional answer rather than a verdict, which is the honest form of it.
| setting | what wins | and the caveat found here |
|---|---|---|
| one clean seasonal series | classical | on average — three models win across five origins |
| multivariate macro | shrinkage; ML ties | only at h=1; the random walk wins at h=12 |
| wide panel | global ML | from flexibility, not pooling — pooling pays only for short series |
| financial returns | nobody | and a mostly-long strategy will imitate an edge |
Notebook
Downloads
References
- Fama, E. F. (1970). Efficient capital markets: a review of theory and empirical work. Journal of Finance 25(2), 383–417. — why the expected answer is "no"
- Bailey, D. H., Borwein, J., López de Prado, M. & Zhu, Q. J. (2014). Pseudo-mathematics and financial charlatanism. Notices of the AMS 61(5), 458–471. — backtest overfitting, and the in-sample mirage
- López de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. — purged CV, deflated Sharpe, trial counting
- Corsi, F. (2009). A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics 7(2), 174–196. — the three coefficients that beat the booster
- Gu, S., Kelly, B. & Xiu, D. (2020). Empirical asset pricing via machine learning. Review of Financial Studies 33(5), 2223–2273. — where ML does pay in finance: the cross-section, not the index