Deflated Sharpe & Backtest Overfitting

The Maximum-Sharpe Illusion

Try enough configurations and the best backtest will look spectacular even if none has any real skill, because you cherry-picked the luckiest of many random outcomes. On 100 pure-noise strategies the winner posts an annualised Sharpe of 1.00, a number that would pass most desks' smell test, against an expected maximum under pure luck of 1.03. (Annualised means rescaled from daily figures to what a year would deliver, the convention that lets strategies of different frequencies be compared on one axis.) The winner is exactly what chance delivers.

Two corrections, both built from scratch. The Deflated Sharpe Ratio asks for the probability that the true Sharpe exceeds not zero but the expected maximum from $N$ trials, adjusted for the length of the record and for the returns' skew and kurtosis — how lopsided the distribution is, and how much of its weight sits in extreme moves. Both matter because the Sharpe ratio only summarises a return stream honestly when that stream is roughly symmetric and thin-tailed, which financial returns are not. The Probability of Backtest Overfitting needs no trial count: combinatorially symmetric cross-validation splits the timeline every balanced way, picks the in-sample best, and records its out-of-sample rank.

Both are worth calibrating before being trusted, which is the question this example is really about: what do these numbers report when the answer is known, and how often are they right?

What the Two Tools Can Actually Detect

The DSR is a stringent test with modest power. Swept against a known injected edge, it never clears 0.95 at a true annualised Sharpe of 0.63 — the size used in the illustration — reaches a one-in-three detection rate only near 1.6, and becomes reliable around 3. It is correctly calibrated under the null, sitting at 0.5 with zero edge and never producing a false positive.

That reframes the illustration's "genuinely-skilled" case. Its DSR of 0.66 is not the test succeeding — it is the test failing to detect a real edge, because 1,250 days and 100 trials cannot establish a Sharpe that size. The useful conclusion is that a strategy failing the DSR has not been shown to be worthless; it has been shown to be unproven, and after a wide search only very large edges are provable at all.

PBO's null is 0.50, not zero. With independent no-skill strategies the in-sample winner is a random pick out of sample, so its rank is uniform and PBO must centre on a half — which replication confirms at 0.52. A single draw of 0.68 is sampling variation, not a detected pathology. Genuine skill does move it down, to 0.34 on average, but the two sampling distributions overlap heavily and a single PBO near 0.4 could have come from either.

Judging Real Strategies

On the real S&P moving-average crossovers, one correction is needed before either tool applies, and it cuts in the strategy's favour. The expected-maximum formula assumes independent trials, and crossovers on the same price series are nothing of the kind — a 30/100 and a 40/120 hold the same position most days. Their return streams correlate at 0.75 on average, and the participation ratio of their correlation matrix puts the effective number of distinct strategies at 1.7 of 35. That ratio is a standard way of asking how many genuinely different directions a set of correlated series spans: it returns 35 if the strategies are independent, and 1 if they are all the same strategy in disguise.

Using the nominal count therefore over-deflates. Correcting to the effective $N$ moves the DSR from 0.68 to 0.91 — a large move, because a search over two effectively distinct ideas sets a much lower luck bar than a search over 35. Neither clears 0.95, so the verdict is unchanged and worth phrasing precisely: a raw annualised Sharpe of 0.44 is unremarkable to begin with, and after accounting for the search this backtest is not established as skill. That is not the same as worthless — it is insufficient evidence, which is the honest end state for most backtests.

When the Package Cross-Check Fails

The R companion runs the CRAN pbo package on the same data, and this is where a cross-check earns its keep by failing. On the controlled cases all three implementations agree. On the real strategies the package reports a PBO of 0.012 where two independently written from-scratch implementations — one Python, one R — both report 0.31.

Tracing it through the package's own per-split output localises the difference. The two agree on the in-sample selection in 94% of splits, the remainder being ties between near-duplicate strategies; what they disagree on is the out-of-sample rank given to that selection — a mean of 31.1 of 35 from the package against 20.9 from a direct recomputation of out-of-sample Sharpe. A rank near the top makes the log-odds of the winner being in the bottom half large and negative, which drives the estimated probability of overfitting toward zero. Beyond that the package's internals are not recoverable from what it returns, so the disagreement is reported rather than resolved. It surfaces only where the ranking step carries the most weight — 35 strategies at 0.75 correlation — and not on the controlled cases where every implementation looks alike.

Where this sits

This closes the Financial ML subsection, which runs purged cross-validation → fractional differentiation → triple-barrier labelling → selection bias. The Sharpe ratio itself is the scorecard used throughout High-Dimensional Portfolios and Financial Returns Predictability, where a gross Sharpe of 0.74 turned out to be market drift. The habit of calibrating a diagnostic against a known answer before trusting it is the same one applied in Calibration and t-SNE & UMAP.

Notebooks

Downloads

Module — Source Code

"""From-scratch deflated Sharpe ratio (Bailey & Lopez de Prado 2014) and the
Probability of Backtest Overfitting via Combinatorial Symmetric CV (CSCV)."""
import numpy as np
from itertools import combinations
from scipy.stats import norm, skew, kurtosis

def prob_sharpe_ratio(sr, T, g3, g4, sr_star=0.0):
    """PSR: probability the true (per-period) Sharpe exceeds sr_star, adjusting
    for skew g3 and kurtosis g4 (non-normal returns)."""
    denom = np.sqrt(1 - g3*sr + (g4-1)/4*sr**2)
    return float(norm.cdf((sr - sr_star)*np.sqrt(T-1)/denom))

def expected_max_sharpe(sr_std, N):
    """Expected maximum of N independent Sharpe estimates under the null (skill=0)."""
    g = 0.5772156649015329                                  # Euler-Mascheroni
    return sr_std*((1-g)*norm.ppf(1-1.0/N) + g*norm.ppf(1-1.0/(N*np.e)))

def deflated_sharpe(returns_best, all_sr, T=None):
    """Deflated Sharpe = PSR of the selected strategy against the expected-max
    Sharpe implied by having tried len(all_sr) strategies."""
    x = np.asarray(returns_best); T = T or len(x)
    sr = x.mean()/x.std()
    sr0 = expected_max_sharpe(np.std(all_sr, ddof=1), len(all_sr))
    return prob_sharpe_ratio(sr, T, skew(x), kurtosis(x, fisher=False), sr0), sr, sr0

def pbo_cscv(M, S=8):
    """Probability of Backtest Overfitting via CSCV. M is (T, N) of per-period
    strategy returns. Returns (pbo, logits)."""
    T, N = M.shape
    blocks = np.array_split(np.arange(T), S)
    logits = []
    for tr in combinations(range(S), S//2):
        te = [b for b in range(S) if b not in tr]
        i_tr = np.concatenate([blocks[b] for b in tr]); i_te = np.concatenate([blocks[b] for b in te])
        sr_is = M[i_tr].mean(0)/(M[i_tr].std(0)+1e-12)
        n_star = int(np.argmax(sr_is))                     # best strategy in-sample
        sr_oos = M[i_te].mean(0)/(M[i_te].std(0)+1e-12)
        rank = (sr_oos.argsort().argsort()[n_star] + 1)/(N + 1)   # OOS relative rank of the IS-best
        rank = min(max(rank, 1e-6), 1-1e-6)
        logits.append(np.log(rank/(1-rank)))
    logits = np.array(logits)
    return float(np.mean(logits <= 0)), logits              # PBO = P(IS-best underperforms OOS median)

References

• Bailey, D. H. & López de Prado, M. (2014). The deflated Sharpe ratio: correcting for selection bias, backtest overfitting and non-normality. Journal of Portfolio Management 40(5), 94–107.
• Bailey, D. H. & López de Prado, M. (2012). The Sharpe ratio efficient frontier. Journal of Risk 15(2), 3–44. — the probabilistic Sharpe ratio
• Bailey, D. H., Borwein, J., López de Prado, M. & Zhu, Q. J. (2017). The probability of backtest overfitting. Journal of Computational Finance 20(4), 39–69. — CSCV
• Harvey, C. R. & Liu, Y. (2015). Backtesting. Journal of Portfolio Management 42(1), 13–28. — multiple-testing adjustment with correlated trials
• Barlow, M. (2016). pbo: Probability of Backtest Overfitting. CRAN R package. — the reference implementation cross-checked here