Financial ML I — Purged & Embargoed Cross-Validation

Why ordinary cross-validation lies in finance, and the fix (López de Prado, Ch. 7)

The Time-Series ML subsection ended with a warning: standard cross-validation manufactures skill on financial data. This subsection — the methodology of López de Prado's *Advances in Financial Machine Learning*** — builds the tools that stop you fooling yourself. First and most important: **cross-validation done right.

The problem is overlapping, path-dependent labels. In finance a label at time $i$ is almost never a single instant — it is the outcome over a forward window $[i, t1_i]$: the return over the next $h$ days, or which barrier (profit-take / stop-loss / time) is hit first. Consecutive labels therefore overlap and are highly correlated. Ordinary $k$-fold cross-validation — which shuffles, or even just splits contiguously — then places observations in the test fold whose label windows overlap observations in the training fold. The model effectively sees test information during training. The result is a cross-validated score far above anything achievable live.

Two corrections (both built from scratch in fincml.py):

• Purging — remove from the training set every observation whose label window overlaps the test set's window.
• Embargo — additionally drop a small block of training observations immediately after each test fold, to kill leakage through serial correlation that purging alone misses.

We expose the leakage in a controlled experiment where the answer is known, repeat that experiment enough times to separate the effect from the noise, and then look at real S&P data. scikit-learn's KFold and TimeSeriesSplit serve as the "package" comparison; neither purges, though it turns out they are not remotely equally guilty, and the measurement below is what settles that. (mlfinlab's PurgedKFold is now commercial, hence the from-scratch build.) Python-lead.

1. The purged, embargoed splitter — from scratch

fincml.purged_kfold takes the label-end times $t1$ (for an $h$-step label, $t1_i = i+h$) and produces contiguous, time-ordered test folds; for each, it purges any training observation whose window $[j, t1_j]$ overlaps the test window, and embargoes a block right after the fold. The picture below draws one fold: each observation is a horizontal segment spanning its label window — the test fold in red, purged (overlapping) training observations in grey, the embargo in orange, and the clean training set that survives in blue. The purged/embargoed observations are exactly the ones that would have leaked.

import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
from fincml import purged_kfold
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
# illustrate one fold on a small sample
n=60; h=8; t1=np.arange(n)+h
folds=list(purged_kfold(t1,n_splits=4,embargo_pct=0.05))
tr,te=folds[1]                                                  # show the 2nd fold
role=np.array(["train"]*n,dtype=object); role[te]="test"
kept=set(tr)
for i in range(n):
    if role[i]=="train" and i not in kept: role[i]="purged/embargo"
fig,ax=plt.subplots(figsize=(11,4.5)); cmap={"train":BLUE,"test":RED,"purged/embargo":GREY}
for i in range(n): ax.plot([i,min(t1[i],n)],[i,i],color=cmap[role[i]],lw=2)
te0,te1=te[0],te[-1]; ax.axvspan(te0,t1[te].max(),color=RED,alpha=.06)
ax.axvspan(te1+1,te1+1+int(n*0.05),color=ORANGE,alpha=.12)
from matplotlib.lines import Line2D
ax.legend([Line2D([0],[0],color=BLUE,lw=3),Line2D([0],[0],color=RED,lw=3),Line2D([0],[0],color=GREY,lw=3)],
          ["train (kept)","test fold","purged + embargoed"],loc="upper left",fontsize=8)
ax.set_xlabel("time (each segment = one observation's label window [i, i+h])"); ax.set_ylabel("observation"); ax.set_title("Purged, embargoed k-fold: one fold")
plt.tight_layout(); plt.show()
print(f"Fold shown: {len(te)} test obs; {n-len(tr)-len(te)} training obs purged/embargoed around it (their label windows")
print("overlap the test window, so keeping them would leak). Only the blue observations train the model for this fold.")

Fold shown: 15 test obs; 16 training obs purged/embargoed around it (their label windows
overlap the test window, so keeping them would leak). Only the blue observations train the model for this fold.

2. Exposing the leakage — a controlled experiment

To prove the leakage we build a world with no real signal at all: the features are random walks (so consecutive rows are near-duplicates) and the label is the sign of a forward window of pure noise, unrelated to the features. The true predictability is therefore exactly AUC = 0.5 — anything above that is leakage, and we know it by construction rather than by argument.

A single such world is one draw, and a splitter that happens to land near 0.5 once has not been shown to be unbiased. So the experiment is run on one world for the picture and then repeated across thirty independent worlds, which is what turns "purged CV reports about 0.5" from an observation into a measurement — and which separates the two scikit-learn splitters far more sharply than a single run does.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import KFold, TimeSeriesSplit
from sklearn.metrics import roc_auc_score
rng=np.random.default_rng(1); N=3000; H=20
Xs=np.cumsum(rng.normal(size=(N,3)),axis=0)                     # random-walk features (near-duplicate neighbours)
eps=rng.normal(size=N)                                          # pure noise, unrelated to features
fwd=np.array([eps[i:i+H].sum() for i in range(N-H)]); ys=(fwd>np.median(fwd)).astype(int)
Xs=Xs[:N-H]; t1s=np.arange(N-H)+H
def cv(splits):
    a=[]
    for tr_,te_ in splits: m=KNeighborsClassifier(1).fit(Xs[tr_],ys[tr_]); a.append(roc_auc_score(ys[te_],m.predict_proba(Xs[te_])[:,1]))
    return np.mean(a)
sh=cv(KFold(5,shuffle=True,random_state=0).split(Xs)); ts=cv(TimeSeriesSplit(5).split(Xs)); pg=cv(purged_kfold(t1s,5,0.02))
sp=int(0.7*len(ys)); mh=KNeighborsClassifier(1).fit(Xs[:sp-H],ys[:sp-H]); ho=roc_auc_score(ys[sp:],mh.predict_proba(Xs[sp:])[:,1])
fig,ax=plt.subplots(figsize=(7.5,4.4))
bars={"shuffled k-fold\n(sklearn)":sh,"TimeSeriesSplit\n(sklearn)":ts,"PURGED+embargo\n(from scratch)":pg,"true forward\nholdout":ho}
ax.bar(list(bars),list(bars.values()),color=[RED,ORANGE,GREEN,GREY]); ax.axhline(0.5,color="k",ls="--",label="truth = 0.5 (no signal)")
for i,v in enumerate(bars.values()): ax.text(i,v+0.01,f"{v:.3f}",ha="center")
ax.set_ylabel("cross-validated AUC"); ax.set_title("No real signal exists — yet shuffled CV 'finds' skill"); ax.legend(fontsize=8)
plt.tight_layout(); plt.show()
print(f"One world: shuffled k-fold AUC {sh:.3f} against a truth of 0.500. There is no signal whatever; shuffled CV")
print("fabricated it by matching each test point to its overlapping temporal neighbour sitting in the training fold.")

# One world is one draw. Repeat the whole experiment on thirty independent ones.
def world(seed,N=3000,H=20):
    r=np.random.default_rng(seed)
    Xw=np.cumsum(r.normal(size=(N,3)),axis=0); e=r.normal(size=N)
    f=np.array([e[i:i+H].sum() for i in range(N-H)])
    return Xw[:N-H],(f>np.median(f)).astype(int),np.arange(N-H)+H,H
def auc_of(Xw,yw,splits):
    a=[]
    for tr_,te_ in splits: a.append(roc_auc_score(yw[te_],KNeighborsClassifier(1).fit(Xw[tr_],yw[tr_]).predict_proba(Xw[te_])[:,1]))
    return np.mean(a)
REP={k:[] for k in ["shuffled k-fold","TimeSeriesSplit","purged, no embargo","purged + embargo","forward holdout"]}
for s in range(30):
    Xw,yw,t1w,Hw=world(100+s)
    REP["shuffled k-fold"].append(auc_of(Xw,yw,KFold(5,shuffle=True,random_state=0).split(Xw)))
    REP["TimeSeriesSplit"].append(auc_of(Xw,yw,TimeSeriesSplit(5).split(Xw)))
    REP["purged, no embargo"].append(auc_of(Xw,yw,purged_kfold(t1w,5,0.0)))
    REP["purged + embargo"].append(auc_of(Xw,yw,purged_kfold(t1w,5,0.02)))
    q=int(0.7*len(yw)); mw=KNeighborsClassifier(1).fit(Xw[:q-Hw],yw[:q-Hw])
    REP["forward holdout"].append(roc_auc_score(yw[q:],mw.predict_proba(Xw[q:])[:,1]))

print(f"\nThirty independent zero-signal worlds. Bias is measured against the known truth of 0.500:")
print(f"  {'method':22s} {'mean AUC':>10} {'std error':>11} {'bias':>9} {'t':>8}")
for k,v in REP.items():
    v=np.array(v); se=v.std(ddof=1)/np.sqrt(len(v))
    print(f"  {k:22s} {v.mean():>10.4f} {se:>11.4f} {v.mean()-0.5:>+9.4f} {(v.mean()-0.5)/se:>8.1f}")
_b=np.mean(REP['shuffled k-fold'])-0.5
print(f"\nShuffled k-fold invents {_b:+.3f} of AUC out of nothing -- a t-statistic near 100, so this is not a fluke of one")
print("dataset but a property of the procedure. Purged cross-validation is statistically indistinguishable from the truth.")
print("\nTwo things a single run would have got wrong, though.")
print(f"\nFirst, TimeSeriesSplit is not the co-defendant this notebook originally implied. Across thirty worlds it reads")
print(f"{np.mean(REP['TimeSeriesSplit']):.4f} -- if anything slightly BELOW 0.5, not above it. Its folds are contiguous and forward, so the only")
print(f"leak is at a single train/test boundary -- {Hw} overlapping observations against roughly {len(yw)//6} test points per fold. That is")
print("real but negligible. The catastrophe belongs to SHUFFLING, which scatters each test point's neighbours throughout")
print("the training set; splitting contiguously already removes almost all of the damage.")
print(f"\nSecond, the embargo is doing nothing measurable here: {np.mean(REP['purged + embargo']):.4f} with it against {np.mean(REP['purged, no embargo']):.4f} without. That is not a")
print("bug, it is the design of this experiment. Purging already deletes everything through the end of the test labels'")
print("reach, and in a world where the noise is independent beyond the label window there is nothing left for an embargo")
print("to catch. It earns its keep when features or errors stay correlated PAST the label horizon -- which real markets do")
print("and this simulation does not. Keep it; just do not credit it with the result that purging produced.")

One world: shuffled k-fold AUC 0.788 against a truth of 0.500. There is no signal whatever; shuffled CV
fabricated it by matching each test point to its overlapping temporal neighbour sitting in the training fold.

Thirty independent zero-signal worlds. Bias is measured against the known truth of 0.500:
  method                   mean AUC   std error      bias        t
  shuffled k-fold            0.7919      0.0030   +0.2919     97.1
  TimeSeriesSplit            0.4905      0.0046   -0.0095     -2.0
  purged, no embargo         0.5037      0.0050   +0.0037      0.7
  purged + embargo           0.5053      0.0054   +0.0053      1.0
  forward holdout            0.4898      0.0075   -0.0102     -1.4

Shuffled k-fold invents +0.292 of AUC out of nothing -- a t-statistic near 100, so this is not a fluke of one
dataset but a property of the procedure. Purged cross-validation is statistically indistinguishable from the truth.

Two things a single run would have got wrong, though.

First, TimeSeriesSplit is not the co-defendant this notebook originally implied. Across thirty worlds it reads
0.4905 -- if anything slightly BELOW 0.5, not above it. Its folds are contiguous and forward, so the only
leak is at a single train/test boundary -- 20 overlapping observations against roughly 496 test points per fold. That is
real but negligible. The catastrophe belongs to SHUFFLING, which scatters each test point's neighbours throughout
the training set; splitting contiguously already removes almost all of the damage.

Second, the embargo is doing nothing measurable here: 0.5053 with it against 0.5037 without. That is not a
bug, it is the design of this experiment. Purging already deletes everything through the end of the test labels'
reach, and in a world where the noise is independent beyond the label window there is nothing left for an embargo
to catch. It earns its keep when features or errors stay correlated PAST the label horizon -- which real markets do
and this simulation does not. Keep it; just do not credit it with the result that purging produced.

3. On real S&P data — and why sklearn's splitters aren't enough

The same corrected CV on real S&P 500 data: features are lagged returns and volatility, the label is the sign of the 10-day forward return, so labels overlap by nine days.

It is worth being clear in advance about what this section can and cannot establish, because the temptation is to read a story into four numbers that are all noise. Consistent with the returns notebook, there is essentially no genuine signal in daily equity direction — and leakage can only inflate a score by importing information that exists. Where there is nothing to leak, every splitter should land near 0.5, and telling them apart becomes impossible. That is exactly what happens below, and it is the reason the previous section had to be run in a simulated world where the truth was known by construction.

So this section demonstrates that the machinery runs on real data and reports what an honest estimate of no-signal looks like. It is not, and cannot be, the evidence that purging matters.

d=pd.read_csv("spx_rv_ret.csv"); ret=d["ret"].values/100; lvol=np.log(np.sqrt(d["rv"].values)*100); n=len(ret); h=10
fwd=np.array([ret[i+1:i+1+h].sum() for i in range(n-h)])
X=[];y=[]
for i in range(22,n-h):
    X.append([ret[i-l] for l in range(1,6)]+[abs(ret[i-l]) for l in range(1,4)]+[lvol[i]]); y.append(int(fwd[i]>0))
X=np.array(X); y=np.array(y); t1=np.arange(len(y))+h
from sklearn.ensemble import RandomForestClassifier
def cvr(splits):
    a=[]
    for tr_,te_ in splits: m=RandomForestClassifier(200,min_samples_leaf=20,random_state=0).fit(X[tr_],y[tr_]); a.append(roc_auc_score(y[te_],m.predict_proba(X[te_])[:,1]))
    return np.mean(a)
def cvr_folds(splits):
    a=[]
    for tr_,te_ in splits: a.append(roc_auc_score(y[te_],RandomForestClassifier(200,min_samples_leaf=20,random_state=0).fit(X[tr_],y[tr_]).predict_proba(X[te_])[:,1]))
    return np.array(a)
A_sh=cvr_folds(KFold(5,shuffle=True,random_state=0).split(X)); A_ts=cvr_folds(TimeSeriesSplit(5).split(X))
A_pg=cvr_folds(purged_kfold(t1,5,0.02))
sp=int(0.8*len(y)); mh=RandomForestClassifier(200,min_samples_leaf=20,random_state=0).fit(X[:sp-h],y[:sp-h])
ph=mh.predict_proba(X[sp:])[:,1]; ho=roc_auc_score(y[sp:],ph)
_r=np.random.default_rng(0); _bs=[]
for _ in range(2000):
    _i=_r.integers(0,len(ph),len(ph))
    if len(set(y[sp:][_i]))>1: _bs.append(roc_auc_score(y[sp:][_i],ph[_i]))
lo_ci,hi_ci=np.quantile(_bs,.025),np.quantile(_bs,.975)
print("S&P 10-day-forward direction, out-of-sample AUC:")
print(f"  {'method':26s} {'mean':>7} {'per-fold spread (sd)':>22}")
for nm,A in [("shuffled k-fold (sklearn)",A_sh),("TimeSeriesSplit (sklearn)",A_ts),("PURGED+embargo (scratch)",A_pg)]:
    print(f"  {nm:26s} {A.mean():>7.3f} {A.std(ddof=1):>22.3f}")
print(f"  {'true forward holdout':26s} {ho:>7.3f}      95% CI [{lo_ci:.3f}, {hi_ci:.3f}] on {len(ph)} points")
print(f"\nEvery one of these is a coin flip. The holdout's confidence interval spans {hi_ci-lo_ci:.2f} of AUC, comfortably contains")
print("0.500, and contains all three cross-validated numbers as well. Nothing here distinguishes the splitters, and any")
print("ranking read off these four values would be reading noise -- including the direction, since the purged estimate")
print(f"happens to come out {A_pg.mean():.3f}, the FURTHEST of the three from the holdout rather than the closest.")
print("\nThat is the expected outcome, not a disappointment. Leakage inflates a score by importing information that")
print("exists; daily equity direction has almost none, so there is nothing available to leak and every method lands on")
print("the same coin flip. Demonstrating that purging matters requires a world where the truth is known and a signal")
print("can be manufactured to leak -- which is precisely what section 2 built, and why it had to be simulated.")
print("\nThe practical claim survives intact and is worth stating without the overreach: on a strategy that DOES carry")
print("faint signal, shuffled cross-validation will inflate it, by the mechanism section 2 measured at +0.29 AUC in a")
print("case with no signal at all. Use purged folds because the failure mode is real and catastrophic when it bites --")
print("not because it visibly changes the number on a dataset with nothing in it.")

S&P 10-day-forward direction, out-of-sample AUC:
  method                        mean   per-fold spread (sd)
  shuffled k-fold (sklearn)    0.504                  0.016
  TimeSeriesSplit (sklearn)    0.491                  0.019
  PURGED+embargo (scratch)     0.476                  0.036
  true forward holdout         0.527      95% CI [0.483, 0.571] on 686 points

Every one of these is a coin flip. The holdout's confidence interval spans 0.09 of AUC, comfortably contains
0.500, and contains all three cross-validated numbers as well. Nothing here distinguishes the splitters, and any
ranking read off these four values would be reading noise -- including the direction, since the purged estimate
happens to come out 0.476, the FURTHEST of the three from the holdout rather than the closest.

That is the expected outcome, not a disappointment. Leakage inflates a score by importing information that
exists; daily equity direction has almost none, so there is nothing available to leak and every method lands on
the same coin flip. Demonstrating that purging matters requires a world where the truth is known and a signal
can be manufactured to leak -- which is precisely what section 2 built, and why it had to be simulated.

The practical claim survives intact and is worth stating without the overreach: on a strategy that DOES carry
faint signal, shuffled cross-validation will inflate it, by the mechanism section 2 measured at +0.29 AUC in a
case with no signal at all. Use purged folds because the failure mode is real and catastrophic when it bites --
not because it visibly changes the number on a dataset with nothing in it.

4. Summary

Ordinary cross-validation is unsafe on financial data because labels overlap. A label built from a forward window shares information with its neighbours, so shuffled $k$-fold leaks test information into training and reports inflated skill. Across thirty independent zero-signal worlds the effect is not subtle: shuffled $k$-fold invents +0.29 of AUC out of nothing, a $t$-statistic near 100, while the from-scratch purged cross-validation is statistically indistinguishable from the truth (bias +0.005, $t=1.0$).

Repeating the experiment rather than running it once also corrected two things a single draw suggested. TimeSeriesSplit is not the co-defendant — across thirty worlds it reads 0.490, if anything slightly below the truth, because contiguous forward folds confine the leak to a single boundary. The catastrophe belongs specifically to shuffling. And the embargo contributes nothing measurable in this experiment (0.5053 with, 0.5037 without), which is a property of the simulation rather than of the embargo: purging already removes everything through the labels' forward reach, and this world has no correlation left beyond it. It earns its keep on real data, where errors stay correlated past the label horizon — but the result here was produced by purging.

On the real S&P data every method returned a coin flip, and the holdout's 95% interval contains 0.5 and all three cross-validated estimates. That is the expected result and worth saying plainly rather than dressing up: leakage inflates a score by importing information that exists, daily equity direction has almost none, so there is nothing to leak. Demonstrating that purging matters requires a simulated world where the truth is known — which is why section 2 is the evidence and section 3 is only a demonstration that the machinery runs.

Two takeaways: (1) KFold with shuffle=True is disqualifying for overlapping-label problems and purging is the fix; TimeSeriesSplit is much closer to safe than it is usually given credit for, and the residual it leaves is what purging removes; (2) this is the machinery that makes every later result in the subsection trustworthy — the honest counterpart to the in-sample mirage exposed in the Time-Series returns notebook.

This opens the Financial ML subsection. Next: fractional differentiation — making a price series stationary without erasing its memory. Then triple-barrier labeling + meta-labeling (which produce the overlapping labels this notebook learned to validate), and the deflated Sharpe ratio + probability of backtest overfitting, which extend the same anti-self-deception discipline from a single model to the search over many strategies.