Purged & Embargoed Cross-Validation — Why Ordinary CV Lies

Overlapping Labels

In finance a label is almost never an instant. It is the outcome over a forward window $[i,\ t1_i]$ — the return over the next ten days, or which barrier a trade hits first. Consecutive labels therefore overlap and are strongly correlated, and an ordinary k-fold puts observations in the training set whose windows overlap the test fold. The model is handed its own test set through the back door, and the cross-validated score comes back higher than anything achievable live.

Two corrections, both built from scratch. Purging removes from training every observation whose label window overlaps the test window — intervals $[j,\ t1_j]$ and $[t_{\text{start}},\ t_{\max}]$ intersect exactly when $j \le t_{\max}$ and $t1_j \ge t_{\text{start}}$. Embargo additionally drops a block immediately after each test fold, to catch leakage through serial correlation that purging alone misses.

A World With Nothing In It

Proving that this matters requires a world where the answer is known, so the experiment is built to have no signal whatsoever: random-walk features, so consecutive rows are near-duplicates, and labels drawn from a forward window of pure noise unrelated to those features. True predictability is exactly AUC = 0.500 — the area under the receiver-operating-characteristic curve, which is the probability the model ranks a randomly chosen positive case above a randomly chosen negative one, so 0.5 is a coin flip. Anything above it is leakage, by construction rather than by argument.

One such world is one draw, and a splitter that happens to land near 0.5 once has not been shown to be unbiased. Repeated across thirty independent worlds, the result is unambiguous — and sharper than the single run suggested.

Shuffled k-fold invents +0.29 of AUC out of nothing, at a t-statistic near 100 — the bias measured in units of its own standard error, so 100 means the gap is a hundred times larger than the noise around it. Not a fluke of one dataset, then, but a property of the procedure. Purged cross-validation is statistically indistinguishable from the truth.

Two things a single run got wrong

Replication also corrected two things a single run implied. TimeSeriesSplit is not the co-defendant. Across thirty worlds it reads 0.490 — if anything slightly below the truth, not above it. Its folds are contiguous and forward, so the only leak is at a single train/test boundary: twenty overlapping observations against roughly 500 test points per fold. Real, but negligible. The catastrophe belongs specifically to shuffling, which scatters each test point's neighbours throughout the training set; splitting contiguously already removes almost all of the damage.

And the embargo does nothing measurable here — 0.5053 with it against 0.5037 without. That is a property of the simulation, not a defect in the embargo. Purging already deletes everything through the forward reach of the test labels, and in a world whose noise is independent beyond the label window there is nothing left to catch. It earns its keep where errors stay correlated past the label horizon, which real markets do and this simulation does not. Keep it — just don't credit it with the result purging produced.

And On Real Data, Nothing

On real S&P data the honest outcome is that nothing is distinguishable. Shuffled reads 0.504, TimeSeriesSplit 0.491, purged 0.476, and a true forward holdout 0.527 with a 95% interval of [0.483, 0.571] that contains 0.5 and all three cross-validated numbers. Any ranking read off those four values is reading noise — including the direction, since the purged estimate happens to land furthest from the holdout rather than closest.

That is the expected result rather than a disappointment, and it is worth stating plainly. Leakage inflates a score by importing information that exists; daily equity direction has almost none, so there is nothing available to leak and every method lands on the same coin flip. Demonstrating that purging matters requires a simulated world where a signal can be manufactured to leak — which is why the controlled experiment is the evidence and the real-data section is only a demonstration that the machinery runs. The practical claim survives without the overreach: on a strategy that does carry faint signal, shuffled cross-validation will inflate it by the mechanism measured above.

Where this sits

This is the validation machinery behind Financial Returns Predictability, where a gross Sharpe of 0.74 turned out to be market drift. Fold construction is the same question raised in Model Selection, where unshuffled folds on sorted data quietly measured extrapolation instead — the mirror image of the problem here, since dependent data is exactly when you want contiguous folds. The habit of testing a method in a world with nothing in it runs through the evaluation subsection: the noise control in t-SNE & UMAP and the simulated ECE floor in Calibration are the same move.

Notebook

Downloads

Splitter — Source Code

"""From-scratch financial-ML cross-validation tools (Lopez de Prado, "Advances
in Financial Machine Learning", ch. 7).

The problem: in finance a label at time i is usually built from a FORWARD window
-- e.g. the return (or triple-barrier outcome) over [i, t1[i]].  Labels of
nearby observations therefore overlap and are highly correlated, so an ordinary
(shuffled or contiguous) k-fold puts information about a test observation into
the training set through its overlapping neighbours.  That leakage inflates the
cross-validated score far above anything achievable live.

Two fixes:
  * PURGING - drop from the training set any observation whose label window
    [j, t1[j]] overlaps the test set's combined label window.
  * EMBARGO - additionally drop a small block of training observations
    immediately AFTER each test fold, to kill leakage through serial
    correlation that purging (which only removes overlaps) misses.

`purged_kfold` yields (train_idx, test_idx) pairs implementing both.
"""
import numpy as np


def purged_kfold(t1, n_splits=5, embargo_pct=0.01):
    """Purged, embargoed k-fold splits.

    Parameters
    ----------
    t1 : array, t1[i] = index at which observation i's label window ends
         (label spans [i, t1[i]]).  For an h-step-ahead label, t1[i] = i + h.
    n_splits : number of contiguous test folds (NOT shuffled -- time order kept).
    embargo_pct : fraction of the sample embargoed after each test fold.

    Yields (train_idx, test_idx) integer arrays.
    """
    t1 = np.asarray(t1)
    n = len(t1)
    idx = np.arange(n)
    embargo = int(n * embargo_pct)
    for test_idx in np.array_split(idx, n_splits):          # contiguous folds, in time
        t_start = test_idx[0]
        t_end = test_idx[-1]
        t_max = int(t1[test_idx].max())                     # furthest reach of any test label
        # purge: a train obs j leaks if its label window [j, t1[j]] overlaps
        # the test window [t_start, t_max]  <=>  j <= t_max AND t1[j] >= t_start
        leaks = (idx <= t_max) & (t1 >= t_start)
        keep = ~leaks
        # embargo: also drop the block just after the test fold
        if embargo > 0:
            keep[t_end + 1: t_end + 1 + embargo] = False
        train_idx = idx[keep]
        yield train_idx, test_idx


def cv_score(clf, X, y, t1, n_splits=5, embargo_pct=0.01, scorer=None):
    """Average a scorer over purged, embargoed folds. scorer(y_true, proba)->float."""
    from sklearn.metrics import roc_auc_score
    scorer = scorer or roc_auc_score
    out = []
    for tr, te in purged_kfold(t1, n_splits, embargo_pct):
        clf.fit(X[tr], y[tr])
        p = clf.predict_proba(X[te])[:, 1]
        out.append(scorer(y[te], p))
    return np.array(out)

References

• López de Prado, M. (2018). Advances in Financial Machine Learning, ch. 7. Wiley. — purging and embargoing
• Bergmeir, C. & Benítez, J. M. (2012). On the use of cross-validation for time series predictor evaluation. Information Sciences 191, 192–213. — when k-fold is and is not safe on dependent data
• Arlot, S. & Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys 4, 40–79.
• Hansen, P. R. & Timmermann, A. (2012). Choice of sample split in out-of-sample forecast evaluation. EUI Working Paper. — the holdout's own uncertainty