Time-Series ML IV — Financial returns: the honesty capstone¶
Can ML predict returns? Mostly no — and knowing that is the skill¶
The subsection has moved from where classical wins (one clean series) to where ML wins (wide panels). This final notebook takes the hardest and most consequential case in quantitative finance: predicting asset returns. Markets are close to efficient — if next-day returns were easily forecastable, the trade would already have been made — so the honest expectation is that no model, however sophisticated, extracts much signal. Testing that rigorously, and resisting the ways one fools oneself, is the actual quant skill.
We do four things, all on real S&P 500 daily data:
- establish the fundamental contrast — returns are essentially unpredictable, volatility is highly predictable;
- throw the full ML roster at return-direction prediction and watch it fail to beat a coin flip — or even the "market usually goes up" baseline;
- expose the in-sample mirage — how a flexible model manufactures fake signal that evaporates out of sample (the data-snooping trap);
- run an honest trading-strategy check — the faint edge vanishes after transaction costs.
The conclusion is not defeatist: it locates where ML does pay off in finance (volatility, risk, execution, the cross-section) and sets up the Financial-ML subsection's tools for not deceiving yourself. Python-lead.
1. The fundamental contrast — returns vs volatility¶
The single most important fact for a would-be return predictor: daily returns have almost no autocorrelation (the efficient-market baseline), while volatility is strongly autocorrelated (it clusters). We measure both on the S&P series and preview the punchline with out-of-sample $R^2$: a model predicting next-day volatility explains a large share of the variance; a model predicting next-day returns explains essentially none (indeed negative $R^2$ — worse than guessing the mean). ML cannot manufacture signal that is not there.
import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
from sklearn.metrics import roc_auc_score, r2_score
import xgboost as xgb
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
d=pd.read_csv("spx_rv_ret.csv"); dates=pd.to_datetime(d["date"]); ret=d["ret"].values; lvol=np.log(np.sqrt(d["rv"].values)*100); n=len(ret)
print(f"S&P 500 daily, {n} days {d['date'].iloc[0]} to {d['date'].iloc[-1]}")
lags=range(1,26)
fig,ax=plt.subplots(figsize=(11,4))
ax.bar([l-0.2 for l in lags],[np.corrcoef(ret[l:],ret[:-l])[0,1] for l in lags],width=.4,color=GREY,label="returns")
ax.bar([l+0.2 for l in lags],[np.corrcoef(lvol[l:],lvol[:-l])[0,1] for l in lags],width=.4,color=BLUE,label="log-volatility")
ax.axhline(0,color="k",lw=.5); ax.set_xlabel("lag (days)"); ax.set_ylabel("autocorrelation"); ax.set_title("Returns have ~no memory; volatility has lots")
ax.legend(); plt.tight_layout(); plt.show()
print(f"ACF(1): returns {np.corrcoef(ret[1:],ret[:-1])[0,1]:+.3f} | log-vol {np.corrcoef(lvol[1:],lvol[:-1])[0,1]:+.3f}")
print("Returns are ~white noise; volatility is highly persistent. Any honest return predictor starts from this fact.")
S&P 500 daily, 3459 days 2000-01-03 to 2013-11-12
ACF(1): returns -0.087 | log-vol +0.784 Returns are ~white noise; volatility is highly persistent. Any honest return predictor starts from this fact.
2. The full ML roster fails to beat a coin flip¶
We build the usual predictive features — lagged returns, lagged absolute returns, current volatility, short/long return averages — and task logistic regression, XGBoost, and a random forest with predicting the sign of tomorrow's return, scored out of sample. All three land at AUC ≈ 0.50 (chance). Worse, their accuracy fails to beat the trivial "always predict up" baseline: the market rises about 56% of days, so a model must clear 56% accuracy to add anything — and none does. This is not a modelling failure; it is the efficient market.
def build(target):
X=[];Y=[]
for i in range(22,n-1):
X.append([ret[i-l] for l in range(1,6)]+[abs(ret[i-l]) for l in range(1,4)]+[lvol[i],np.mean(ret[i-5:i]),np.mean(ret[i-22:i])])
Y.append(target(i))
return np.array(X),np.array(Y)
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
Xd,yd=build(lambda i:int(ret[i+1]>0)); sp=int(0.7*len(yd)); base=max(yd[sp:].mean(),1-yd[sp:].mean())
models={"logistic":LogisticRegression(max_iter=1000),
"XGBoost":xgb.XGBClassifier(n_estimators=200,max_depth=3,learning_rate=0.05,verbosity=0),
"random forest":RandomForestClassifier(300,min_samples_leaf=20,random_state=0)}
au={}; ac={}
for nm,m in models.items():
m.fit(Xd[:sp],yd[:sp]); p=m.predict_proba(Xd[sp:])[:,1]; au[nm]=roc_auc_score(yd[sp:],p); ac[nm]=((p>0.5).astype(int)==yd[sp:]).mean()
fig,ax=plt.subplots(1,2,figsize=(13,4.2)); nm=list(models)
ax[0].bar(nm,[au[k] for k in nm],color=BLUE); ax[0].axhline(0.5,color=RED,ls="--",label="coin flip (0.5)"); ax[0].set_ylim(0.45,0.56); ax[0].set_ylabel("OOS AUC"); ax[0].set_title("Return-direction AUC \u2248 chance"); ax[0].legend()
ax[1].bar(nm,[ac[k] for k in nm],color=GREEN); ax[1].axhline(base,color=RED,ls="--",label=f'"always up" ({base:.3f})'); ax[1].set_ylim(0.45,0.6); ax[1].set_ylabel("OOS accuracy"); ax[1].set_title("...and cannot beat the majority-class baseline"); ax[1].legend()
plt.tight_layout(); plt.show()
for k in nm: print(f" {k:14s} OOS AUC {au[k]:.3f} accuracy {ac[k]:.3f} (must beat {base:.3f} to add value)")
print("Every model is at chance and below the 'market usually rises' baseline -- direction is not forecastable here.")
logistic OOS AUC 0.480 accuracy 0.525 (must beat 0.563 to add value) XGBoost OOS AUC 0.503 accuracy 0.522 (must beat 0.563 to add value) random forest OOS AUC 0.487 accuracy 0.504 (must beat 0.563 to add value) Every model is at chance and below the 'market usually rises' baseline -- direction is not forecastable here.
3. The in-sample mirage — how you fool yourself¶
Here is the trap that has ended countless strategies. Give a flexible model enough capacity and it will fit the training returns perfectly — in-sample AUC near 1.0 — and that fit is pure memorised noise: out of sample the same model is back at 0.50. With enough features, models, and hyperparameter tries, one will find a configuration that looks great in-sample or even on a single test split, purely by chance (multiple testing). The discipline the Financial-ML subsection formalises — purged/embargoed cross-validation, deflated Sharpe ratios, controlling the number of trials — exists precisely to stop this self-deception. The bar below is the whole cautionary tale in two numbers.
deep=xgb.XGBClassifier(n_estimators=500,max_depth=6,learning_rate=0.1,verbosity=0).fit(Xd[:sp],yd[:sp])
is_auc=roc_auc_score(yd[:sp],deep.predict_proba(Xd[:sp])[:,1]); oos_auc=roc_auc_score(yd[sp:],deep.predict_proba(Xd[sp:])[:,1])
fig,ax=plt.subplots(figsize=(6,4.2))
ax.bar(["in-sample","out-of-sample"],[is_auc,oos_auc],color=[GREY,RED]); ax.axhline(0.5,color="k",ls=":")
for i,v in enumerate([is_auc,oos_auc]): ax.text(i,v+0.01,f"{v:.3f}",ha="center")
ax.set_ylabel("AUC"); ax.set_title("A deep model on returns: perfect in-sample, chance out-of-sample")
plt.tight_layout(); plt.show()
print(f"Deep XGBoost on returns: in-sample AUC {is_auc:.3f} -> out-of-sample AUC {oos_auc:.3f}.")
print("The in-sample 'signal' is memorised noise. This gap is why a backtest that isn't leakage-proof and trial-adjusted lies.")
Deep XGBoost on returns: in-sample AUC 1.000 -> out-of-sample AUC 0.499. The in-sample 'signal' is memorised noise. This gap is why a backtest that isn't leakage-proof and trial-adjusted lies.
4. Where ML does help — and the R² that proves it¶
The contrast, quantified: the same feature set and model that explain none of next-day returns explain a large fraction of next-day volatility. With one caveat this notebook is obliged to apply to itself — having just insisted that models be judged against the right baseline, the booster has to face the same test on volatility, and a three-term HAR regression beats it. That is the general truth for ML in finance — the predictable objects are second moments and higher-order structure (volatility, correlation, tail risk), the cross-section (relative winners vs losers), execution, and regime/anomaly detection — not the direction of the market tomorrow. ML is a powerful tool pointed at the right target; pointed at naive return prediction it has nothing to work with.
Xr,yr=build(lambda i:ret[i+1]); Xv,yv=build(lambda i:lvol[i+1])
gr=xgb.XGBRegressor(n_estimators=200,max_depth=3,learning_rate=0.05,verbosity=0).fit(Xr[:sp],yr[:sp])
gv=xgb.XGBRegressor(n_estimators=200,max_depth=3,learning_rate=0.05,verbosity=0).fit(Xv[:sp],yv[:sp])
r2r=r2_score(yr[sp:],gr.predict(Xr[sp:])); r2v=r2_score(yv[sp:],gv.predict(Xv[sp:]))
fig,ax=plt.subplots(figsize=(6.5,4.2))
b=ax.bar(["next-day RETURN","next-day VOLATILITY"],[r2r,r2v],color=[RED,BLUE]); ax.axhline(0,color="k",lw=.6)
for i,v in enumerate([r2r,r2v]): ax.text(i,v+(0.02 if v>0 else -0.04),f"{v:+.3f}",ha="center")
ax.set_ylabel("out-of-sample $R^2$"); ax.set_title("Same model, same features: returns unpredictable, volatility predictable")
plt.tight_layout(); plt.show()
print(f"OOS R^2: next-day return {r2r:+.3f} (negative -> worse than the mean) | next-day volatility {r2v:+.3f} (strong).")
print("The predictability in markets lives in volatility/risk and the cross-section -- where the vol, GARCH and portfolio arcs operate.")
# The notebook has just spent two cells insisting that a model be judged against the right baseline.
# That standard applies here too: is the booster actually the best way to exploit volatility's persistence?
_lv=Xv[:,8] # today's log-vol, already a feature
_r2rw=r2_score(yv[sp:],_lv[sp:])
_A=np.column_stack([np.ones(sp),_lv[:sp]]); _b=np.linalg.lstsq(_A,yv[:sp],rcond=None)[0]
_r2ar=r2_score(yv[sp:],_b[0]+_b[1]*_lv[sp:])
def _har(idx):
return np.column_stack([np.ones(len(idx)),_lv[idx],
np.array([lvol[22+k-4:22+k+1].mean() for k in idx]),
np.array([lvol[22+k-21:22+k+1].mean() for k in idx])])
_tr=np.arange(sp); _te=np.arange(sp,len(yv))
_bh=np.linalg.lstsq(_har(_tr),yv[:sp],rcond=None)[0]; _r2har=r2_score(yv[sp:],_har(_te)@_bh)
print(f"\nBut the same standard applied two cells ago has to apply here. Against simple baselines on the SAME split:")
print(f" XGBoost, 10 features R^2 {r2v:+.4f}")
print(f" random walk (today's log-vol) R^2 {_r2rw:+.4f}")
print(f" AR(1) on log-vol R^2 {_r2ar:+.4f}")
print(f" HAR (day + week + month) R^2 {_r2har:+.4f} <- three coefficients")
print(f"\nThe contrast that matters survives intact and is the point of this section: {r2v:+.3f} against {r2r:+.3f} is the")
print("difference between a target with structure and one without. But the booster is NOT the best way to exploit")
print(f"that structure -- a three-term HAR regression beats it by {_r2har-r2v:+.3f}, which is the same result the LSTM")
print("notebook reached on this identical series. Volatility is where the predictability lives; classical models")
print("built around its long memory remain the way to capture it. 'Point ML at the right target' is necessary")
print("advice, not sufficient advice.")
OOS R^2: next-day return -0.128 (negative -> worse than the mean) | next-day volatility +0.458 (strong). The predictability in markets lives in volatility/risk and the cross-section -- where the vol, GARCH and portfolio arcs operate. But the same standard applied two cells ago has to apply here. Against simple baselines on the SAME split: XGBoost, 10 features R^2 +0.4576 random walk (today's log-vol) R^2 +0.3061 AR(1) on log-vol R^2 +0.3933 HAR (day + week + month) R^2 +0.5180 <- three coefficients The contrast that matters survives intact and is the point of this section: +0.458 against -0.128 is the difference between a target with structure and one without. But the booster is NOT the best way to exploit that structure -- a three-term HAR regression beats it by +0.060, which is the same result the LSTM notebook reached on this identical series. Volatility is where the predictability lives; classical models built around its long memory remain the way to capture it. 'Point ML at the right target' is necessary advice, not sufficient advice.
5. The strategy reality check — costs kill the faint edge¶
Finally, the test that matters: turn the best direction model into a trading strategy (go long when it predicts up, short when down) and compare to buy-and-hold, before and after realistic transaction costs (5 bps per trade). Two things go wrong, and they are worth separating. The obvious one is that costs bury the apparent edge. The subtler one — and the one a gross-only backtest would hide — is that there was no edge to bury in the first place: the model is long most of the time, so its gross return is the market's drift rather than any prediction. This is why an unadjusted backtest is dangerous, and why the López de Prado toolkit (next subsection) treats cost, leakage, and trial-count as first-class.
m=xgb.XGBClassifier(n_estimators=200,max_depth=3,learning_rate=0.05,verbosity=0).fit(Xd[:sp],yd[:sp])
pos=np.where(m.predict_proba(Xd[sp:])[:,1]>0.5,1,-1); r=ret[23+sp:23+sp+len(pos)]/100
cost=0.0005; turn=np.abs(np.diff(np.r_[0,pos])); gross=pos*r; net=gross-turn*cost
def sharpe(x): return np.sqrt(252)*np.mean(x)/np.std(x)
dd=dates.values[23+sp:23+sp+len(pos)]
fig,ax=plt.subplots(figsize=(11,4.4))
ax.plot(dd,np.cumprod(1+r),color="black",lw=1.6,label=f"buy & hold (Sharpe {sharpe(r):.2f})")
ax.plot(dd,np.cumprod(1+gross),color=ORANGE,lw=1.2,label=f"ML strategy, gross (Sharpe {sharpe(gross):.2f})")
ax.plot(dd,np.cumprod(1+net),color=RED,lw=1.2,label=f"ML strategy, net of costs (Sharpe {sharpe(net):.2f})")
ax.set_ylabel("cumulative growth of $1"); ax.set_title("Return-prediction strategy vs buy-and-hold"); ax.legend()
plt.tight_layout(); plt.show()
_long=(pos==1).mean(); _flips=int(np.abs(np.diff(np.r_[0,pos])).sum()/2)
print(f"Buy-hold Sharpe {sharpe(r):.2f}; ML strategy gross {sharpe(gross):.2f}, net of 5bps costs {sharpe(net):.2f}.")
print(f"\nThe gross figure needs reading carefully, because it is the most seductive number on the page. It is NOT")
print(f"a sliver of edge. The model is long on {100*_long:.0f}% of days, and a position that is long most of the time simply")
print(f"inherits the market's drift: 'always long' scores {sharpe(np.ones_like(pos)*r):.2f}, which is buy-and-hold by construction. A gross")
print("Sharpe sitting on top of buy-and-hold is exactly what NO skill produces when the strategy is mostly long.")
print(f"\nWhat destroys it is turnover. The model changes position on {_flips} of {len(pos)} days ({100*_flips/len(pos):.0f}%), and at 5bps a side")
print(f"that is a cumulative drag of {turn.sum()*cost*100:.0f}% of capital over the period. The net Sharpe of {sharpe(net):.2f} is the honest")
print("number, and it is far below simply holding the index and doing nothing.")
print("\nSo there are two separate lessons here, and the second is the easier one to miss. Trading costs bury a faint")
print("edge -- everyone knows that. But before costs there was no edge to bury: the gross return came from being")
print("long in a rising market, not from prediction, and a backtest that reported only the gross Sharpe would have")
print("looked like a working strategy.")
Buy-hold Sharpe 0.72; ML strategy gross 0.74, net of 5bps costs 0.13. The gross figure needs reading carefully, because it is the most seductive number on the page. It is NOT a sliver of edge. The model is long on 65% of days, and a position that is long most of the time simply inherits the market's drift: 'always long' scores 0.72, which is buy-and-hold by construction. A gross Sharpe sitting on top of buy-and-hold is exactly what NO skill produces when the strategy is mostly long. What destroys it is turnover. The model changes position on 430 of 1031 days (42%), and at 5bps a side that is a cumulative drag of 43% of capital over the period. The net Sharpe of 0.13 is the honest number, and it is far below simply holding the index and doing nothing. So there are two separate lessons here, and the second is the easier one to miss. Trading costs bury a faint edge -- everyone knows that. But before costs there was no edge to bury: the gross return came from being long in a rising market, not from prediction, and a backtest that reported only the gross Sharpe would have looked like a working strategy.
6. Summary — and the end of the Time-Series ML subsection¶
Machine learning cannot predict what is not predictable, and financial returns are close to unpredictable. On real S&P data: returns have ~zero autocorrelation while volatility clusters; the full ML roster forecasts return direction at chance and below the "market usually rises" baseline; a flexible model achieves a perfect in-sample fit that is pure noise out of sample; and the same features that explain none of next-day returns explain a large share of next-day volatility. A naive return-prediction strategy has no edge and loses to buy-and-hold after costs. The value of ML in finance is real but aimed — at volatility, risk, the cross-section, execution — not at guessing tomorrow's market move.
This closes the Time-Series ML subsection and its central question — do ML/NN beat traditional econometrics on time series? — with a complete, honest, conditional answer:
| setting | winner |
|---|---|
| one clean series (ex1) | classical dominates |
| multivariate macro (ex2) | shrinkage wins; ML ties |
| wide panel (ex3) | global ML wins |
| financial returns (ex4) | nobody — the market is efficient |
The unifying lesson is the arc's throughout: match the method to the structure of the problem, and be relentlessly honest about out-of-sample performance. The last piece — the tools that enforce that honesty in finance (purged/embargoed CV, meta-labeling, deflated Sharpe, backtest-overfitting control) — is the Financial-ML methodology subsection, which follows directly from the traps exposed here.