import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings
warnings.filterwarnings("ignore")
from fincml_bt import deflated_sharpe, pbo_cscv, expected_max_sharpe
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
rng=np.random.default_rng(0); T=1250; N=100; ann=np.sqrt(252)
M0=rng.normal(0,0.01,(T,N))                                     # N pure-noise strategies
sr0=M0.mean(0)/M0.std(0); best0=int(np.argmax(sr0))
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
ax[0].hist(sr0*ann,bins=25,color=GREY,edgecolor="white"); ax[0].axvline(sr0[best0]*ann,color=RED,lw=2,label=f"best = {sr0[best0]*ann:.2f} (looks great!)")
ax[0].axvline(0,color="k",lw=1,ls=":"); ax[0].set_xlabel("annualized Sharpe"); ax[0].set_ylabel("# strategies"); ax[0].set_title("100 NO-SKILL strategies: the best looks skilled"); ax[0].legend(fontsize=8)
Ns=np.arange(2,501); emax=[expected_max_sharpe(np.std(sr0,ddof=1),k)*ann for k in Ns]
ax[1].plot(Ns,emax,color=BLUE,lw=2); ax[1].axhline(sr0[best0]*ann,color=RED,ls="--",label="our best of 100")
ax[1].set_xlabel("number of strategies tried (N)"); ax[1].set_ylabel("expected MAX Sharpe under null"); ax[1].set_title("The luck bar rises with N"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print(f"Best of {N} no-skill strategies: annualized Sharpe {sr0[best0]*ann:.2f} -- but the EXPECTED max under pure luck is")
print(f"{expected_max_sharpe(np.std(sr0,ddof=1),N)*ann:.2f}. The 'winner' is exactly what chance delivers. Raw Sharpe is meaningless after a search.")

Best of 100 no-skill strategies: annualized Sharpe 1.00 -- but the EXPECTED max under pure luck is
1.03. The 'winner' is exactly what chance delivers. Raw Sharpe is meaningless after a search.

dsr0,sr_b0,srmax0=deflated_sharpe(M0[:,best0],sr0,T)
M1=rng.normal(0,0.01,(T,N)); M1[:,:10]+=0.0004                 # 10 strategies with a real small edge
sr1=M1.mean(0)/M1.std(0); best1=int(np.argmax(sr1)); dsr1,sr_b1,_=deflated_sharpe(M1[:,best1],sr1,T)
fig,ax=plt.subplots(figsize=(7,4.2))
ax.bar(["no-skill\nwinner","genuinely-skilled\nwinner"],[dsr0,dsr1],color=[RED,GREEN])
ax.axhline(0.5,color="k",ls=":",label="0.5 = no better than best-of-luck"); ax.axhline(0.95,color=GREY,ls="--",label="0.95 significance")
for i,v in enumerate([dsr0,dsr1]): ax.text(i,v+0.02,f"{v:.3f}",ha="center")
ax.set_ylim(0,1); ax.set_ylabel("Deflated Sharpe Ratio (P[true SR>0])"); ax.set_title("DSR sees through the illusion"); ax.legend(fontsize=8)
plt.tight_layout(); plt.show()
print(f"no-skill winner: raw annualized Sharpe {sr_b0*ann:.2f} but DSR {dsr0:.3f} -- a coin flip, NOT a discovery.")
print(f"skilled winner:  raw annualized Sharpe {sr_b1*ann:.2f}, DSR {dsr1:.3f}.")
print(f"\nThe DSR does its job on the first case: a Sharpe of {sr_b0*ann:.2f} that would pass most desks' smell test is correctly")
print("valued at nothing once the 100 trials behind it are accounted for.")
print(f"\nBut read the second case carefully rather than as a success. Those ten strategies carry a genuine edge of")
print(f"{0.0004/0.01*ann:.2f} annualized Sharpe, and the DSR still comes out at {dsr1:.3f} -- above the no-skill case, and nowhere near the")
print("0.95 line the chart draws. The honest reading is not 'higher confidence of a genuine edge'; it is that this")
print("sample CANNOT establish an edge of that size after a search of this width. The DSR is not being conservative")
print("by mistake. Section 3 measures exactly how large an edge has to be before it can be certified.")

no-skill winner: raw annualized Sharpe 1.00 but DSR 0.470 -- a coin flip, NOT a discovery.
skilled winner:  raw annualized Sharpe 1.32, DSR 0.655.

The DSR does its job on the first case: a Sharpe of 1.00 that would pass most desks' smell test is correctly
valued at nothing once the 100 trials behind it are accounted for.

But read the second case carefully rather than as a success. Those ten strategies carry a genuine edge of
0.63 annualized Sharpe, and the DSR still comes out at 0.655 -- above the no-skill case, and nowhere near the
0.95 line the chart draws. The honest reading is not 'higher confidence of a genuine edge'; it is that this
sample CANNOT establish an edge of that size after a search of this width. The DSR is not being conservative
by mistake. Section 3 measures exactly how large an edge has to be before it can be certified.

p0,lg0=pbo_cscv(M0,S=10); p1,lg1=pbo_cscv(M1,S=10)
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
ax[0].hist(lg0,bins=20,color=RED,alpha=.6,label=f"no-skill (PBO {p0:.2f})"); ax[0].hist(lg1,bins=20,color=GREEN,alpha=.6,label=f"skilled (PBO {p1:.2f})")
ax[0].axvline(0,color="k",lw=1.5); ax[0].set_xlabel("logit(OOS rank of in-sample best)"); ax[0].set_ylabel("# CSCV splits"); ax[0].set_title("Mass left of 0 = overfitting"); ax[0].legend(fontsize=8)
ax[1].bar(["no-skill","genuinely\nskilled"],[p0,p1],color=[RED,GREEN]); ax[1].axhline(0.5,color="k",ls=":")
for i,v in enumerate([p0,p1]): ax[1].text(i,v+0.02,f"{v:.2f}",ha="center")
ax[1].set_ylabel("Probability of Backtest Overfitting"); ax[1].set_title("PBO: high for a lucky winner, low for a real one")
plt.tight_layout(); plt.show()
print(f"PBO: no-skill {p0:.2f} (the in-sample best is routinely mediocre OOS -> overfit) vs skilled {p1:.2f} (holds up OOS).")

PBO: no-skill 0.68 (the in-sample best is routinely mediocre OOS -> overfit) vs skilled 0.32 (holds up OOS).

REPS=30
print(f"DSR power -- how large must a genuine edge be before the test can certify it? ({REPS} worlds per row)")
print(f"  {'injected edge':24s} {'true ann. Sharpe':>17} {'mean DSR':>10} {'P(DSR>0.95)':>13} {'P(DSR>0.50)':>13}")
rp=np.random.default_rng(7)
for e_,tag in [(0.0,"none"),(0.0004,"the one used above"),(0.001,"strong"),(0.002,"very strong")]:
    ds=[]
    for _ in range(REPS):
        Mx=rp.normal(0,0.01,(T,N))
        if e_>0: Mx[:,:10]+=e_
        sx=Mx.mean(0)/Mx.std(0); bx=int(np.argmax(sx))
        ds.append(deflated_sharpe(Mx[:,bx],sx,T)[0])
    ds=np.array(ds)
    print(f"  {tag:24s} {e_/0.01*ann:>17.2f} {ds.mean():>10.3f} {np.mean(ds>0.95):>13.2f} {np.mean(ds>0.50):>13.2f}")
print("\nAt zero edge the mean DSR sits at about 0.5 and it never clears 0.95 -- correctly calibrated under the null.")
print("At the edge used in section 2 it STILL never clears 0.95. The test needs roughly an annualized Sharpe of 1.6")
print("before it has even a one-in-three chance, and around 3 before it is reliable. That is the real message of the")
print("DSR and it is more useful than any single verdict: after a wide search, only very large edges are certifiable,")
print("and a strategy that fails the test has not been shown to be worthless -- it has been shown to be unproven.")

print(f"\n\nPBO calibration -- the sampling distribution rather than one draw ({REPS} worlds each):")
pn=[];pk=[]
for _ in range(REPS):
    Mx=rp.normal(0,0.01,(T,N)); pn.append(pbo_cscv(Mx,S=10)[0])
    My=rp.normal(0,0.01,(T,N)); My[:,:10]+=0.0004; pk.append(pbo_cscv(My,S=10)[0])
pn=np.array(pn); pk=np.array(pk)
print(f"  {'strategies':22s} {'mean PBO':>10} {'sd':>7} {'range':>16}")
print(f"  {'no skill':22s} {pn.mean():>10.3f} {pn.std(ddof=1):>7.3f} {f'[{pn.min():.2f}, {pn.max():.2f}]':>16}")
print(f"  {'genuine small edge':22s} {pk.mean():>10.3f} {pk.std(ddof=1):>7.3f} {f'[{pk.min():.2f}, {pk.max():.2f}]':>16}")
print(f"\nThe no-skill case centres on {pn.mean():.2f} -- essentially 0.50, exactly as the theory says it must, because an")
print(f"in-sample winner chosen from independent noise is a random pick out of sample. The {p0:.2f} printed above came from")
print("one draw and sits high in that distribution; it is not evidence of anything beyond sampling variation.")
print(f"\nThe skilled case does sit lower ({pk.mean():.2f}), so PBO carries real signal -- but the two distributions overlap")
print(f"heavily, {f'[{pn.min():.2f}, {pn.max():.2f}]'} against {f'[{pk.min():.2f}, {pk.max():.2f}]'}, and a single PBO value near 0.4 could have come from either.")
print("Use PBO as one reading among several, not as a threshold to be crossed.")

fig,ax=plt.subplots(1,2,figsize=(13,4.2))
ax[0].hist(pn,bins=12,color=RED,alpha=.6,label=f"no skill (mean {pn.mean():.2f})")
ax[0].hist(pk,bins=12,color=GREEN,alpha=.6,label=f"genuine edge (mean {pk.mean():.2f})")
ax[0].axvline(0.5,color="k",ls=":",lw=1.5,label="0.50 = the null, not a red flag")
ax[0].set_xlabel("PBO"); ax[0].set_ylabel(f"# simulated worlds"); ax[0].set_title("PBO sampling distributions overlap"); ax[0].legend(fontsize=7)
es=[0.0,0.0005,0.001,0.0015,0.002,0.003]; pw=[]
for e_ in es:
    c=0
    for _ in range(REPS):
        Mx=rp.normal(0,0.01,(T,N))
        if e_>0: Mx[:,:10]+=e_
        sx=Mx.mean(0)/Mx.std(0); bx=int(np.argmax(sx))
        c+=deflated_sharpe(Mx[:,bx],sx,T)[0]>0.95
    pw.append(c/REPS)
ax[1].plot(np.array(es)/0.01*ann,pw,"o-",color=PURP,lw=2)
ax[1].axhline(0.8,color=GREY,ls=":",label="80% power"); ax[1].axvline(0.0004/0.01*ann,color=RED,ls="--",label="edge used in section 2")
ax[1].set_xlabel("true annualized Sharpe of the skilled strategies"); ax[1].set_ylabel("P(DSR clears 0.95)")
ax[1].set_ylim(-0.03,1.03); ax[1].set_title("DSR power curve"); ax[1].legend(fontsize=7)
plt.tight_layout(); plt.show()

DSR power -- how large must a genuine edge be before the test can certify it? (30 worlds per row)
  injected edge             true ann. Sharpe   mean DSR   P(DSR>0.95)   P(DSR>0.50)
  none                                  0.00      0.505          0.00          0.43
  the one used above                    0.63      0.574          0.00          0.63
  strong                                1.59      0.900          0.37          1.00

  very strong                           3.17      0.992          0.97          1.00

At zero edge the mean DSR sits at about 0.5 and it never clears 0.95 -- correctly calibrated under the null.
At the edge used in section 2 it STILL never clears 0.95. The test needs roughly an annualized Sharpe of 1.6
before it has even a one-in-three chance, and around 3 before it is reliable. That is the real message of the
DSR and it is more useful than any single verdict: after a wide search, only very large edges are certifiable,
and a strategy that fails the test has not been shown to be worthless -- it has been shown to be unproven.


PBO calibration -- the sampling distribution rather than one draw (30 worlds each):

  strategies               mean PBO      sd            range
  no skill                    0.515   0.147     [0.26, 0.90]
  genuine small edge          0.337   0.146     [0.04, 0.67]

The no-skill case centres on 0.52 -- essentially 0.50, exactly as the theory says it must, because an
in-sample winner chosen from independent noise is a random pick out of sample. The 0.68 printed above came from
one draw and sits high in that distribution; it is not evidence of anything beyond sampling variation.

The skilled case does sit lower (0.34), so PBO carries real signal -- but the two distributions overlap
heavily, [0.26, 0.90] against [0.04, 0.67], and a single PBO value near 0.4 could have come from either.
Use PBO as one reading among several, not as a threshold to be crossed.

d=pd.read_csv("spx_rv_ret.csv"); r=d["ret"].values/100; price=np.cumprod(1+r); cols=[]; names=[]
for f in [5,10,20,30,40]:
    for s in [50,60,80,100,120,150,200]:
        if f<s:
            fast=pd.Series(price).rolling(f).mean().values; slow=pd.Series(price).rolling(s).mean().values
            pos=np.nan_to_num(np.sign(fast-slow)); cols.append(pos[:-1]*r[1:]); names.append(f"{f}/{s}")
Mr=np.column_stack(cols); Mr=Mr[~np.isnan(Mr).any(1)]
srr=Mr.mean(0)/Mr.std(0); best=int(np.argmax(srr)); dsrr,srb,srm=deflated_sharpe(Mr[:,best],srr,len(Mr)); pr,lgr=pbo_cscv(Mr,S=10)
fig,ax=plt.subplots(1,2,figsize=(13,4.2))
ax[0].hist(srr*ann,bins=15,color=BLUE,edgecolor="white"); ax[0].axvline(srb*ann,color=RED,lw=2,label=f"best {names[best]}: {srb*ann:.2f}")
ax[0].set_xlabel("annualized Sharpe"); ax[0].set_ylabel("# strategies"); ax[0].set_title(f"{Mr.shape[1]} MA-crossover strategies on S&P"); ax[0].legend(fontsize=8)
ax[1].bar(["Deflated\nSharpe","PBO"],[dsrr,pr],color=[GREEN,RED]); ax[1].axhline(0.5,color="k",ls=":")
for i,v in enumerate([dsrr,pr]): ax[1].text(i,v+0.02,f"{v:.2f}",ha="center")
ax[1].set_ylim(0,1); ax[1].set_title("The verdict on the best backtest")
plt.tight_layout(); plt.show()
Cm=np.corrcoef(Mr.T); offd=Cm[np.triu_indices(Mr.shape[1],1)]
evs=np.linalg.eigvalsh(Cm)[::-1]; n_eff=(evs.sum()**2)/(evs**2).sum()
from fincml_bt import prob_sharpe_ratio, expected_max_sharpe as _ems
from scipy.stats import skew as _sk, kurtosis as _ku
xb=Mr[:,best]
sr_eff=_ems(np.std(srr,ddof=1),max(int(round(n_eff)),2))
dsr_eff=prob_sharpe_ratio(xb.mean()/xb.std(),len(xb),_sk(xb),_ku(xb,fisher=False),sr_eff)
print(f"Best MA-crossover ({names[best]}): raw annualized Sharpe {srb*ann:.3f}; PBO {pr:.2f}.")
print(f"\nThese {Mr.shape[1]} strategies are not {Mr.shape[1]} independent trials. Mean pairwise correlation between their return")
print(f"streams is {offd.mean():.2f} (range {offd.min():.2f} to {offd.max():.2f}), and the participation ratio of the correlation matrix puts the")
print(f"effective number of independent strategies at {n_eff:.1f}. The search was far narrower than it looks.")
print(f"\n  {'luck bar (expected max Sharpe, annualized)':46s} {'DSR':>7}")
print(f"  {f'using the nominal N = {Mr.shape[1]}':46s} {dsrr:>7.3f}    (bar {srm*ann:.3f})")
print(f"  {f'using the effective N = {int(round(n_eff))}':46s} {dsr_eff:>7.3f}    (bar {sr_eff*ann:.3f})")
print(f"\nCorrecting for the correlation moves the DSR from {dsrr:.2f} to {dsr_eff:.2f} -- a large move, and in the strategy's favour,")
print("because a search over two effectively distinct ideas sets a much lower luck bar than a search over 35.")
print(f"Neither figure clears 0.95. The verdict is the same either way and it is worth phrasing precisely: this backtest")
print(f"is NOT established as skill. A raw annualized Sharpe of {srb*ann:.2f} is unremarkable to begin with, and after accounting")
print("for the search it cannot be distinguished from the best of what luck provides. That is not the same as saying")
print("the strategy is worthless -- it is saying the evidence is insufficient, which is the honest end state for most")
print("backtests and the reason this subsection exists.")

Best MA-crossover (40/200): raw annualized Sharpe 0.437; PBO 0.31.

These 35 strategies are not 35 independent trials. Mean pairwise correlation between their return
streams is 0.75 (range 0.49 to 0.97), and the participation ratio of the correlation matrix puts the
effective number of independent strategies at 1.7. The search was far narrower than it looks.

  luck bar (expected max Sharpe, annualized)         DSR
  using the nominal N = 35                         0.676    (bar 0.314)
  using the effective N = 2                        0.909    (bar 0.076)

Correcting for the correlation moves the DSR from 0.68 to 0.91 -- a large move, and in the strategy's favour,
because a search over two effectively distinct ideas sets a much lower luck bar than a search over 35.
Neither figure clears 0.95. The verdict is the same either way and it is worth phrasing precisely: this backtest
is NOT established as skill. A raw annualized Sharpe of 0.44 is unremarkable to begin with, and after accounting
for the search it cannot be distinguished from the best of what luck provides. That is not the same as saying
the strategy is worthless -- it is saying the evidence is insufficient, which is the honest end state for most
backtests and the reason this subsection exists.

Financial ML IV — Deflated Sharpe & Backtest Overfitting (capstone of the subsection)¶

The best of many strategies is inflated by luck — and how to catch it (Bailey & López de Prado)¶

1. The maximum-Sharpe illusion¶

2. The Deflated Sharpe Ratio¶

3. Probability of Backtest Overfitting (CSCV)¶

3. What these two tools can actually detect¶

5. Judging real strategies — S&P moving-average crossovers¶

6. Summary — and the end of the Financial ML subsection¶