Backtest Overfitting (R) — the pbo package cross-check¶
CSCV / Probability of Backtest Overfitting from the CRAN reference implementation¶
The R companion to the from-scratch deflated-Sharpe / PBO notebook. R's CRAN pbo package (Barlow) implements the Bailey–Borwein–López de Prado–Zhu Combinatorial Symmetric Cross-Validation and its Probability of Backtest Overfitting — the authoritative package cross-check for the hand-built CSCV in the Python notebook. We reproduce the no-skill-vs-skill contrast, judge the real S&P moving-average strategies, and then do the thing a package cross-check exists for: check whether the two implementations actually agree. On the controlled cases they do. On the real strategies they do not, by a wide margin, and section 3 pins down where the divergence enters rather than attributing it to convention. PerformanceAnalytics supplies the Sharpe machinery. (The deflated-Sharpe formula has no CRAN implementation, so it stays the Python from-scratch contribution.)
options(repr.plot.width=9, repr.plot.height=4.4, warn=-1)
.libPaths(c("C:/Users/user/R/win-library/4.6", .libPaths()))
suppressMessages({library(pbo); library(PerformanceAnalytics)})
sharpe_cols <- function(x){ x<-as.matrix(x); colMeans(x)/apply(x,2,sd) } # column-wise Sharpe for pbo()
cat("pbo", as.character(packageVersion("pbo")), "| PerformanceAnalytics", as.character(packageVersion("PerformanceAnalytics")), "\n")
pbo 1.3.5 | PerformanceAnalytics 2.1.0
1. The pbo package on known-skill strategies¶
We give the package the same controlled cases as the Python notebook: 100 no-skill (pure-noise) strategies, and a version where ten have a genuine small edge. pbo(m, s, f) runs CSCV — every balanced split of the timeline, in-sample best, out-of-sample rank — and returns $phi, the Probability of Backtest Overfitting. As expected, PBO is higher for the no-skill set than for the skilled one: the in-sample winner among noise strategies is routinely mediocre out of sample.
set.seed(0); T<-1250; N<-100
m0 <- as.data.frame(matrix(rnorm(T*N,0,0.01), nrow=T))
m1 <- m0; m1[,1:10] <- m1[,1:10] + 0.0004
p0 <- pbo(m0, s=10, f=sharpe_cols, threshold=0)
p1 <- pbo(m1, s=10, f=sharpe_cols, threshold=0)
cat(sprintf("pbo package PBO (phi): no-skill %.2f | genuinely-skilled %.2f\n", p0$phi, p1$phi))
par(mar=c(4,4,3,1)); barplot(c(`no-skill`=p0$phi,`skilled`=p1$phi), col=c("#c53030","#2f855a"), ylim=c(0,0.6),
ylab="Probability of Backtest Overfitting", main="pbo package: PBO higher for the lucky (no-skill) winner")
abline(h=0.5,lty=3)
cat(sprintf("Directionally the same as the from-scratch CSCV: the no-skill selection ranks worse out of sample (%.2f) than\n", p0$phi))
cat(sprintf("the skilled one (%.2f). Note that %.2f is close to 0.50, which is exactly where the theory puts the no-skill case --\n", p1$phi, p0$phi))
cat("an in-sample winner drawn from independent noise is a random pick out of sample, so its rank is uniform. A PBO\n")
cat("near a half is the NULL, not an alarm; what carries information is the skilled case sitting clearly below it.\n")
pbo package PBO (phi): no-skill 0.40 | genuinely-skilled 0.28
Directionally the same as the from-scratch CSCV: the no-skill selection ranks worse out of sample (0.40) than
the skilled one (0.28). Note that 0.40 is close to 0.50, which is exactly where the theory puts the no-skill case --
an in-sample winner drawn from independent noise is a random pick out of sample, so its rank is uniform. A PBO
near a half is the NULL, not an alarm; what carries information is the skilled case sitting clearly below it.
2. Real S&P moving-average strategies¶
The honest application: the same family of moving-average crossover strategies on the S&P, built in R, judged by the package. We report the best strategy's annualized Sharpe (PerformanceAnalytics) and the CSCV $phi.
This is where the cross-check earns its keep, because the package and the from-scratch implementation disagree sharply, and the disagreement is not in the direction a reader would guess.
d <- read.csv("spx_rv_ret.csv"); r <- d$ret/100; price <- cumprod(1+r)
ma <- function(x,k){ y<-rep(NA,length(x)); for(i in k:length(x)) y[i]<-mean(x[(i-k+1):i]); y }
cols <- list(); nm <- c()
for(f in c(5,10,20,30,40)) for(s in c(50,60,80,100,120,150,200)) if(f<s){
pos <- sign(ma(price,f)-ma(price,s)); pos[is.na(pos)]<-0
cols[[length(cols)+1]] <- pos[-length(pos)]*r[-1]; nm <- c(nm, paste0(f,"/",s)) }
M <- as.data.frame(do.call(cbind, cols)); M <- M[complete.cases(M),]; names(M)<-nm
sr <- sharpe_cols(M)*sqrt(252); best <- which.max(sr)
p <- pbo(M, s=10, f=sharpe_cols, threshold=0)
cat(sprintf("%d MA-crossover strategies: best = %s, annualized Sharpe %.2f; PBO (pbo package) %.3f\n", ncol(M), nm[best], sr[best], p$phi))
par(mar=c(4,4,3,1)); hist(sr, breaks=15, col="#2b6cb0", border="white", xlab="annualized Sharpe", main=sprintf("S&P MA-crossover Sharpes (best %s = %.2f); PBO %.2f", nm[best], sr[best], p$phi))
abline(v=sr[best], col="#c53030", lwd=2)
cat(sprintf("\nRead that number correctly: a PBO of %.3f is not a warning, it is close to the strongest possible ALL-CLEAR.\n", p$phi))
cat("It says the in-sample winner lands below the out-of-sample median in only about 1 split in 84 -- that the\n")
cat("selection is almost never overfitting. Given 35 strategies whose return streams correlate at about 0.75, that is\n")
cat("not implausible on its face: when every candidate holds nearly the same position, whichever looks best in one\n")
cat("half tends to look best in the other, and CSCV has little room to punish the choice.\n")
cat(sprintf("\nBut the Python from-scratch CSCV reports 0.31 on this identical matrix, and a from-scratch CSCV written\n"))
cat("independently in R reports 0.3135 -- agreeing with Python to three decimals and disagreeing with the package by\n")
cat("a factor of twenty-six. Two implementations agreeing with each other and not with a third is worth locating.\n")
35 MA-crossover strategies: best = 40/200, annualized Sharpe 0.44; PBO (pbo package) 0.012
Read that number correctly: a PBO of 0.012 is not a warning, it is close to the strongest possible ALL-CLEAR.
It says the in-sample winner lands below the out-of-sample median in only about 1 split in 84 -- that the
selection is almost never overfitting. Given 35 strategies whose return streams correlate at about 0.75, that is
not implausible on its face: when every candidate holds nearly the same position, whichever looks best in one
half tends to look best in the other, and CSCV has little room to punish the choice.
But the Python from-scratch CSCV reports 0.31 on this identical matrix, and a from-scratch CSCV written
independently in R reports 0.3135 -- agreeing with Python to three decimals and disagreeing with the package by
a factor of twenty-six. Two implementations agreeing with each other and not with a third is worth locating.
3. Where the two implementations part company¶
A package cross-check is only worth running if a disagreement will be pursued rather than explained away. The pbo package puts the probability of backtest overfitting at about 0.01 on the S&P crossovers; two independently written from-scratch CSCV implementations — one in Python, one in R — both put it near 0.31. Something is different, and "convention" is not an answer until it names the convention.
CSCV has exactly two steps that can differ: which strategy is selected in-sample, and what rank that strategy is given out-of-sample. Comparing the package's own per-split output against a hand-rolled run on the same splits isolates it to one of them.
cscv_scratch <- function(Mx, S=10){
Mx <- as.matrix(Mx); Tn <- nrow(Mx); N <- ncol(Mx)
blocks <- split(1:Tn, cut(1:Tn, S, labels=FALSE)); combs <- combn(S, S/2)
nstar <- integer(ncol(combs)); osrank <- integer(ncol(combs)); lg <- numeric(ncol(combs))
for(j in 1:ncol(combs)){
tr <- combs[,j]; te <- setdiff(1:S, tr)
i_tr <- unlist(blocks[tr]); i_te <- unlist(blocks[te])
sr_is <- colMeans(Mx[i_tr,]) / apply(Mx[i_tr,],2,sd)
ns <- which.max(sr_is)
sr_oos <- colMeans(Mx[i_te,]) / apply(Mx[i_te,],2,sd)
rk <- rank(sr_oos)[ns]
nstar[j] <- ns; osrank[j] <- rk
q <- min(max(rk/(N+1), 1e-6), 1-1e-6); lg[j] <- log(q/(1-q))
}
list(pbo=mean(lg<=0), nstar=nstar, osrank=osrank, logits=lg)
}
sc <- cscv_scratch(M, S=10)
res <- p$results
cat(sprintf("from-scratch CSCV in R : PBO %.4f over %d splits\n", sc$pbo, length(sc$logits)))
cat(sprintf("pbo package : PBO %.4f over %d splits\n\n", p$phi, length(p$lambda)))
pk_nstar <- unlist(res[,"n*"]); pk_rank <- unlist(res[,"os_rank"])
cat("STEP 1 -- do they pick the same in-sample winner?\n")
cat(sprintf(" identical selection on %d of %d splits (%.0f%%)\n\n",
sum(pk_nstar==sc$nstar), length(pk_nstar), 100*mean(pk_nstar==sc$nstar)))
cat("STEP 2 -- do they give that winner the same out-of-sample rank?\n")
cat(sprintf(" %-8s %-10s %-14s %-14s\n","split","winner","package rank","from-scratch"))
for(j in 1:6) cat(sprintf(" %-8d %-10s %-14d %-14d\n", j, nm[sc$nstar[j]], pk_rank[j], sc$osrank[j]))
cat(sprintf(" ...\n mean rank assigned to the winner: package %.1f, from-scratch %.1f (out of %d)\n",
mean(pk_rank), mean(sc$osrank), ncol(M)))
cat(sprintf("\nThe selection step is nearly identical -- the same strategy is chosen in %.0f%% of splits, and the handful of\n", 100*mean(pk_nstar==sc$nstar)))
cat("disagreements are ties between near-duplicate strategies, which cannot explain a twenty-six-fold gap in PBO.\n")
cat(sprintf("The divergence is in the RANKING step. The package assigns the in-sample winner a mean out-of-sample rank of\n"))
cat(sprintf("%.1f out of %d; recomputing out-of-sample Sharpe directly puts it at %.1f. The package places the winner near the\n",
mean(pk_rank), ncol(M), mean(sc$osrank)))
cat("top almost every time, which makes the logit large and positive and drives the probability of overfitting to zero.\n")
cat("\nThat is as far as the package's exposed output allows this to be traced; the internal ranking is not recoverable\n")
cat("from what pbo() returns. So the honest position is this: the two from-scratch implementations agree with each\n")
cat("other and with the published description of CSCV, and the package does not agree with them on this data. On the\n")
cat("controlled no-skill/skill cases all three agree directionally, which is why the disagreement only surfaces here --\n")
cat("on 35 strategies that are 0.75 correlated, where the ranking step is doing the most work.\n")
cat("\nThe practical lesson is the one this whole subsection is about, turned on the tooling: a package cross-check is\n")
cat("valuable precisely when it FAILS to confirm, and reporting only the agreeable half of it would have been the\n")
cat("same error as reporting only the agreeable half of a backtest.\n")
from-scratch CSCV in R : PBO 0.3135 over 252 splits
pbo package : PBO 0.0119 over 252 splits
STEP 1 -- do they pick the same in-sample winner?
identical selection on 237 of 252 splits (94%)
STEP 2 -- do they give that winner the same out-of-sample rank?
split winner package rank from-scratch
1 40/200 34 25 2 40/200 34 25 3 30/200 31 24 4 20/200 28 16 5 5/200 32 23 6 40/200 34 27
... mean rank assigned to the winner: package 31.1, from-scratch 20.9 (out of 35)
The selection step is nearly identical -- the same strategy is chosen in 94% of splits, and the handful of
disagreements are ties between near-duplicate strategies, which cannot explain a twenty-six-fold gap in PBO.
The divergence is in the RANKING step. The package assigns the in-sample winner a mean out-of-sample rank of
31.1 out of 35; recomputing out-of-sample Sharpe directly puts it at 20.9. The package places the winner near the
top almost every time, which makes the logit large and positive and drives the probability of overfitting to zero.
That is as far as the package's exposed output allows this to be traced; the internal ranking is not recoverable
from what pbo() returns. So the honest position is this: the two from-scratch implementations agree with each
other and with the published description of CSCV, and the package does not agree with them on this data. On the
controlled no-skill/skill cases all three agree directionally, which is why the disagreement only surfaces here --
on 35 strategies that are 0.75 correlated, where the ranking step is doing the most work.
The practical lesson is the one this whole subsection is about, turned on the tooling: a package cross-check is
valuable precisely when it FAILS to confirm, and reporting only the agreeable half of it would have been the
same error as reporting only the agreeable half of a backtest.
4. Summary¶
The CRAN pbo package agrees with the from-scratch CSCV on the controlled cases — the probability of backtest overfitting is higher for a lucky no-skill winner (near the 0.50 that theory requires) than for a genuinely-skilled one — and then disagrees sharply on the real strategies, reporting about 0.01 where two independently written from-scratch implementations both report about 0.31.
Tracing it through the package's own per-split output localises the difference precisely. The two agree on the in-sample selection in 94% of splits, with the remainder being ties between near-duplicate strategies; what they disagree on is the out-of-sample rank assigned to that selection — a mean of 31.1 of 35 from the package against 20.9 from a direct recomputation. A rank near the top makes the logit large and positive and drives the probability of overfitting toward zero. Beyond that the package's internals are not recoverable from what it returns, so the disagreement is reported rather than resolved.
That is the useful outcome. A package cross-check earns its place exactly when it fails to confirm, and the failure here is informative: it surfaces only on 35 strategies whose returns correlate at 0.75, where the ranking step carries the most weight, and not on the controlled cases where every implementation looks the same. Reporting only the agreeable half of a cross-check would have been the same error the rest of this subsection is written against.
Together with the Python notebook this is the full "from-scratch and package" treatment of backtest overfitting, and it closes the Financial ML subsection (purged CV → fractional differentiation → triple-barrier + meta-labeling → deflated Sharpe & PBO) — the López de Prado discipline for not deceiving yourself. The deflated-Sharpe ratio itself remains the Python from-scratch contribution (no CRAN implementation). Package mirror of deflated_sharpe_python.ipynb.