Harvey-Liu (2015) Backtesting

Summary

Replaces the industry rule of thumb — "discount a reported Sharpe ratio by 50%" — with an analytical haircut derived from the statistics of multiple testing. A backtested strategy is one of many tried (by the researcher, the firm, or the profession), so its apparent significance must be judged against the number of trials. Harvey and Liu convert a strategy's Sharpe ratio to a $t$-ratio, apply a multiple-testing $p$-value adjustment, and convert the adjusted $t$-ratio back to a haircut Sharpe ratio. The haircut is non-linear: genuinely high Sharpe ratios are discounted little, while marginal ones are discounted heavily — often by far more than 50%.

Key Claims

• The 50% haircut is only a rule of thumb. A principled discount should depend on the strategy's own statistics and on how many strategies were searched.
• Multiple testing is the right lens. With $N$ candidate strategies/factors, some will show $t\approx 2$ by chance; the significance cutoff must rise with $N$.
• Three adjustments. The paper applies Bonferroni, Holm, and Benjamini-Hochberg-Yekutieli (BHY) procedures — controlling family-wise error (Bonferroni/Holm) or the false discovery rate (BHY) — to the strategy's $p$-value.
• Sharpe ↔ t-ratio round trip. A Sharpe ratio maps to a $t$-ratio given the sample length; the multiple-testing-adjusted $p$-value maps back to a haircut Sharpe ratio, giving a concrete, defensible discount.
• Non-linear haircut. The proportional haircut shrinks as the true Sharpe rises: strong strategies survive nearly intact, weak ones are penalized heavily — the opposite of a flat 50% cut.

Concepts Introduced or Extended

• Backtest Overfitting

Entities Mentioned

• Campbell R. Harvey
• Yan Liu

Quotes

"A common practice in evaluating backtests of trading strategies is to discount the reported Sharpe ratios by 50%. ... The 50% haircut is only a rule of thumb. Our article's goal is to develop an analytical way to determine the haircut's magnitude. Our framework relies on the statistical concept of multiple testing."

My Take

The practitioner-facing companion to the factor-zoo and backtest-overfitting literatures: it takes the abstract "control for multiple testing" imperative and turns it into a number a portfolio manager can apply to a single reported Sharpe ratio. The round trip — Sharpe → $t$ → multiple-testing-adjusted $t$ → haircut Sharpe — is simple enough to be used, and the non-linearity is the right message (a flat 50% cut over-penalizes genuinely good strategies and under-penalizes marginal ones). It complements the CSCV/PBO machinery of Bailey-López de Prado (which resamples in-sample vs. out-of-sample performance) with a p-value-adjustment route to the same end, and it is the methodological sibling of the authors' cross-sectional factor-zoo work in Harvey-Liu-Zhu (2016). Its main practical caveat is the same as everyone else's: you must know how many strategies were really tried.