Finance — High-Dimensional Portfolios

When the Optimiser Becomes an Error-Maximiser

Shrinking the Factor Zoo ended on an honest cliffhanger: with 20 diversified factors over 60 years, penalisation bought parsimony but not performance, because that problem is not high-dimensional. This is the regime where shrinkage stops being optional — and it runs on the Risk & Asset Allocation section's own data, 48 US stocks at weekly frequency.

The task is the minimum-variance portfolio, $\min_w w'\Sigma w$ subject to $\mathbf 1'w = 1$, solved by $w \propto \Sigma^{-1}\mathbf 1$. It is chosen deliberately: it depends only on the covariance, never on expected returns, so it isolates the covariance-estimation problem cleanly. The catch is $\Sigma$. With 48 assets it holds 1,176 free parameters, and a one-year estimation window supplies only 52 observations. The sample covariance is then barely invertible, $\Sigma^{-1}$ amplifies whatever noise it contains, and the "optimal" portfolio takes enormous offsetting long/short bets that have nothing to do with real risk — Markowitz's optimiser turned into an error-maximiser (Michaud, 1989).

Where the estimate stops existing

One boundary is worth stating precisely, because it is easy to dramatise incorrectly. A covariance estimated from $T$ observations has rank at most $T-1$, so for $T \le N$ it is exactly singular and the minimum-variance portfolio does not exist. Solvers do not always say so: at a 30-week window the matrix has rank 29 and a condition number around $10^{18}$, and solve() returns a confident-looking answer built from floating-point noise. Those windows are therefore excluded here rather than plotted. The genuine deterioration is dramatic enough without them — out-of-sample volatility runs from 12% at a long window to 61% as $T$ approaches $N$, while shrinkage stays flat near 13% throughout.

The Horse Race

A rolling backtest — one-year window, re-estimated weekly, everything strictly out of sample — over five portfolios: sample minimum-variance, Ledoit–Wolf shrinkage, a ridge (L2) covariance, the long-only portfolio whose no-short constraint acts as implicit L1 (Jagannathan & Ma, 2003), and equal-weight 1/N.

The sample portfolio carries 15× gross leverage and 46% out-of-sample volatility — three times the shrunk estimators, at ten times the leverage and twenty times the turnover. Every form of shrinkage or constraint collapses both to sane levels. The long-only portfolio reaches the lowest volatility while holding only about a dozen of the 48 stocks, at the lowest turnover of any optimised method: the no-short constraint is not merely stabilising, it is selecting.

Is the edge over 1/N real?

The result that needed testing rather than asserting is the comparison with 1/N. On Sharpe the sparse portfolio's 0.73 against 1/N's 0.70 looks like a win. A paired bootstrap over the 259 weeks puts the gap at +0.028 with a 95% interval of [−0.58, +0.61], and it comes out ahead in 53% of resamples — a coin flip. DeMiguel, Garlappi & Uppal's finding survives the test: with zero estimation and no optimisation whatsoever, 1/N matches every one of these estimators on risk-adjusted return.

But Sharpe is not what these portfolios optimise — they minimise variance and never estimate a single expected return. Judged on their own objective the verdict reverses and is unambiguous: against 1/N the sparse portfolio's volatility advantage is −5.5 percentage points with a 95% interval of [−8.3, −2.8], comfortably clear of zero, and its tie with Ledoit–Wolf (−0.5pp, interval straddling zero) is a genuine tie. Shrinkage buys lower risk, not higher return per unit of it — and an estimator should be judged on the quantity it was built to control.

Two estimators, one number

A cross-engine detail that looks like a bug and is not. In R, corpcor::cov.shrink and the ridge covariance report the same volatility to two decimals — 16.2554 against 16.2616 — yet they are different portfolios, their weekly returns correlating 0.979. Both are linear shrinkage toward a diagonal target, so converging is unsurprising. scikit-learn's Ledoit–Wolf, with a different target, reaches 15.5%. The choice of shrinkage target matters more than the choice of package.

Where this sits

This closes the loop the factor example opened: shrinkage's payoff scales with dimensionality, and portfolio construction is where it becomes indispensable. It is the frequentist twin of Shrinkage Estimation of Mean and Covariance, Bayesian Estimation & Estimation Risk and The Black–Litterman Model, which treat this same $\Sigma^{-1}$ instability with a prior rather than a penalty — and, since the long-only constraint selects assets exactly as the Lasso selects factors, of the Variable Selection arc.

Notebooks

Downloads

References

• Ledoit, O. & Wolf, M. (2004). A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis 88(2), 365–411. — the shrinkage estimator used here
• Michaud, R. O. (1989). The Markowitz optimization enigma: is 'optimized' optimal? Financial Analysts Journal 45(1), 31–42. — the error-maximiser diagnosis
• Jagannathan, R. & Ma, T. (2003). Risk reduction in large portfolios: why imposing the wrong constraints helps. Journal of Finance 58(4), 1651–1683. — the no-short constraint as implicit shrinkage
• DeMiguel, V., Garlappi, L. & Uppal, R. (2009). Optimal versus naive diversification. Review of Financial Studies 22(5), 1915–1953. — the 1/N benchmark tested above
• Brodie, J., Daubechies, I., De Mol, C., Giannone, D. & Loris, I. (2009). Sparse and stable Markowitz portfolios. PNAS 106(30), 12267–12272. — L1 in portfolio construction
• Schäfer, J. & Strimmer, K. (2005). A shrinkage approach to large-scale covariance matrix estimation. Statistical Applications in Genetics and Molecular Biology 4(1). — the estimator behind R's corpcor