Bailey et al. (2017) The Probability of Backtest Overfitting

Summary

Investment firms select strategies and allocate capital using backtests — simulations of performance on historical data — but standard overfitting safeguards (like a single holdout split) are unreliable in this setting. Bailey, Borwein, López de Prado and Zhu propose a general framework to assess the probability of backtest overfitting (PBO): the probability that the strategy chosen as best in-sample will underperform the median of the alternatives out-of-sample. Their model-free, nonparametric implementation, combinatorially symmetric cross-validation (CSCV), produces reasonable PBO estimates across examples.

Key Claims

• PBO defined. Backtest overfitting occurs when the configuration that looks best in-sample is selected, but its out-of-sample performance is no better than a coin flip among the candidates. PBO is the probability that the in-sample optimal strategy ranks below the median out-of-sample.
• CSCV. Partition the performance matrix (strategies × time) into $S$ equal time blocks; over all $\binom{S}{S/2}$ symmetric train/test splits, pick the in-sample-best strategy and record its relative out-of-sample rank $\bar\omega_c$. Using both halves of each split as train and test ("symmetric") makes the estimate efficient and unbiased.
• Logit statistic. Map each split's OOS rank to a logit $\lambda_c=\ln\frac{\bar\omega_c}{1-\bar\omega_c}$; high logits mean in-sample and out-of-sample rankings agree (low overfitting). $\text{PBO}=\Pr(\lambda<0)$, the fraction of splits where the IS-best is below-median OOS.
• Beyond PBO. The framework also yields performance degradation (OOS Sharpe declining as IS Sharpe rises — the overfitting signature), the probability of loss, and stochastic-dominance diagnostics.
• Why holdout fails. A single holdout is high-variance and easily gamed by trying enough configurations; the combinatorial approach averages over many train/test partitions.

Concepts Introduced or Extended

• Backtest Overfitting

Entities Mentioned

• David H. Bailey
• Jonathan M. Borwein
• Marcos López de Prado
• Qiji Jim Zhu

Quotes

"Standard statistical techniques designed to prevent regression overfitting, such as holdout, tend to be unreliable and inaccurate in the context of investment backtests. We propose a general framework to assess the probability of backtest overfitting (PBO) … combinatorially symmetric cross-validation (CSCV)."

My Take

Where the deflated Sharpe ratio corrects a reported number for the number of trials, PBO tackles the harder, more honest question: given the whole set of configurations you searched, how likely is it that your "winner" is a fluke? CSCV's cleverness is using every block as both train and test across all symmetric partitions, which turns the noisy single-holdout into a distribution of in-sample-vs-out-of-sample rank agreements — and the resulting PBO and performance-degradation plots are unusually legible for a statistical diagnostic. It is the rigorous backbone of the same authors' polemical "pseudo-mathematics" essay. The practical limitation is shared with the deflated SR: it needs the full menu of trials, so a researcher who quietly discards configurations before applying CSCV understates their own overfitting.