Robust Bayesian Allocation

Model

Estimation risk established that portfolio weights are really distributions, most of them straddling zero. This project acts on that. Rather than optimising against a single point estimate, robust allocation optimises against the worst case within an uncertainty set around it — asking not "what is best if my estimate is right?" but "what is best if my estimate is as wrong as it plausibly could be?" The aversion parameter $q$ sets how large that set is, and the whole method lives between two familiar limits.

Two limits, both verified

Those limits are verified rather than claimed, which is the right way to check a new optimiser. At $q=0$ the robust portfolio reproduces mean–variance exactly (agreement to $5\times10^{-6}$); as $q$ grows large it converges on the minimum-variance portfolio, because with enough distrust of the return estimates the only thing left worth optimising is risk. Everything interesting happens on the path between: sweeping $q$ traces a continuous morph from one to the other, with gross leverage falling as the portfolio de-risks.

What robustness buys

What robustness actually buys is best seen as insurance. On a synthetic market with a known truth, raising $q$ from 0 to 8 moves the mean certainty-equivalent from $-1.87$ to $+0.55$ and — more tellingly — the worst-case outcome from $-8.95$ to $+0.09$, while the spread of results collapses from $4.32$ to $0.49$ — a ninefold reduction in dispersion. The naive plug-in is not merely worse on average; it is catastrophically worse in its bad draws, and that dispersion is exactly what robustness is paid to remove.

Two treatments of the same disease

The project's most useful section sets two different treatments of the same disease side by side. Bayesian allocation attacks estimation error by shrinking the inputs; robust allocation attacks it by distrusting them in the objective. On the synthetic study the Bayesian route dominates on its own (mean CE 1.13 against 0.21 for robust-on-sample), and applying robustness on top of an already-Bayesian estimate produces only the modest insurance trade-off you would expect — slightly lower mean, slightly better worst case. The PyMC companion shows why they are two views of one idea: the robust weights derived from the posterior dispersion coincide with those from the analytic uncertainty ellipsoid to four decimal places.

Out of Sample

The out-of-sample horse-race is where the project earns its honesty. On ten sector ETFs the estimation-risk hierarchy holds cleanly — sample MV, then Bayesian, then robust, improving in Sharpe, volatility and turnover together — but equal weighting beats all of them outright, and the notebook says so. At $N/T\approx0.1$ there is barely any estimation error to fight, and $1/N$ pays no estimation cost at all. That is the correct read of a ten-asset problem, not an embarrassment.

The picture reverses exactly where the theory says it should. Pushing to 48 stocks and the Fama–French 48 industries, sample mean–variance degenerates — a Sharpe of 0.43 at 199% volatility on stocks, 0.30 at 88% on industries — while the Bayesian and robust portfolios stay near 1.0 at 12% and overtake $1/N$ (0.83). The R engine reproduces the same table independently. Robustness costs a little Sharpe against pure Bayes at high dimension while running slightly lower volatility, which is the trade it advertises; and at scale it produces a markedly more tradeable book — gross leverage 14.9 against 24.5 for the naive alternative.

Notebooks

Downloads

Robust Allocation Module — Source Code

"""
robust.py -- Robust Bayesian mean-variance allocation.

Backs the notebooks in  "Robust Bayesian Allocation".
Follows Meucci, "Risk and Asset Allocation" (Springer 2005), chapter 9-C/D
(robust and robust-Bayesian allocation).

THE IDEA
--------
Projects 1-3 shrank the *inputs* to a mean-variance optimiser. But even a shrunk /
Bayesian estimate is only a point, surrounded by a cloud of uncertainty (the
posterior). A standard optimiser trusts that point completely and happily loads up
on whatever direction looks best -- precisely the directions where the estimate is
least reliable. ROBUST allocation refuses to be fooled: it optimises the WORST case
over an uncertainty region around the estimate.

For the expected-return vector, take the ellipsoid
    U = { mu : (mu - mu_hat)' Theta^-1 (mu - mu_hat) <= q^2 }
where mu_hat is the (Bayesian) estimate, Theta its estimation-error covariance
(for the NIW posterior, Theta = Sigma1 / T1), and q sets the size of the region.
The worst-case expected return of a portfolio w over U has a closed form:
    min_{mu in U} w' mu = w' mu_hat - q * sqrt(w' Theta w).
So the robust mean-variance problem is
    max_w   w' mu_hat - q * sqrt(w' Theta w) - (gamma/2) w' Sigma w   s.t.  w'1 = 1.
The extra term  -q*sqrt(w' Theta w)  is a penalty on positions whose expected return
is uncertain. It is a SECOND-ORDER CONE program (Meucci solves it with SeDuMi; we use
scipy's SLSQP). Two limits:
    q -> 0        : ordinary (Bayesian) mean-variance -- trust the estimate fully.
    q -> infinity : the penalty dominates the mean; the allocation collapses toward the
                    minimum-variance portfolio -- ignore the means entirely.
Between them sits a family of increasingly cautious, increasingly stable portfolios.
"""

import numpy as np
from scipy.optimize import minimize


# --------------------------------------------------------------------------- #
#  Moments and the NIW posterior (compact, self-contained copy)                 #
# --------------------------------------------------------------------------- #

def sample_moments(X, mle=True):
    X = np.asarray(X, float); T = X.shape[0]
    mu = X.mean(0); Xc = X - mu
    return mu, Xc.T @ Xc / (T if mle else T - 1)


def niw_posterior(X, mu0, T0, Sigma0, nu0):
    X = np.asarray(X, float); T, N = X.shape
    mu_hat, S_hat = sample_moments(X)
    T1 = T0 + T
    mu1 = (T0 * mu0 + T * mu_hat) / T1
    nu1 = nu0 + T
    d = (mu0 - mu_hat).reshape(-1, 1)
    Sigma1 = (nu0 * Sigma0 + T * S_hat + (T * T0 / T1) * (d @ d.T)) / nu1
    return dict(mu1=mu1, T1=T1, Sigma1=Sigma1, nu1=nu1)


def posterior_moments(post):
    """Posterior mean of mu, posterior mean of Sigma, and the estimation-error
    covariance Theta = Sigma1/T1 (the size of the location ellipsoid)."""
    mu1, T1, Sigma1, nu1 = post["mu1"], post["T1"], post["Sigma1"], post["nu1"]
    N = len(mu1)
    Sigma_mean = nu1 * Sigma1 / (nu1 - N - 1)
    Theta = Sigma1 / T1
    return mu1, Sigma_mean, Theta


# --------------------------------------------------------------------------- #
#  Plain and robust mean-variance optimisers                                    #
# --------------------------------------------------------------------------- #

def _budget_constraint():
    return {"type": "eq", "fun": lambda w: np.sum(w) - 1.0}


def mv_weights(mu, Sigma, gamma, long_only=False):
    """Ordinary mean-variance: max w'mu - gamma/2 w'Sigma w  s.t. w'1 = 1."""
    N = len(mu)
    obj = lambda w: -(w @ mu - 0.5 * gamma * (w @ Sigma @ w))
    bounds = [(0, None)] * N if long_only else None
    res = minimize(obj, np.ones(N) / N, method="SLSQP",
                   constraints=[_budget_constraint()], bounds=bounds,
                   options={"maxiter": 500, "ftol": 1e-10})
    return res.x


def robust_mv(mu, Sigma, Theta, q, gamma, long_only=False):
    """Robust mean-variance with an ellipsoidal uncertainty set on mu.

        max_w  w'mu - q*sqrt(w'Theta w) - gamma/2 w'Sigma w   s.t.  w'1 = 1.

    q is the radius of the location ellipsoid (aversion to estimation error).
    """
    N = len(mu)
    def neg_obj(w):
        pen = q * np.sqrt(max(w @ Theta @ w, 1e-18))
        return -(w @ mu - pen - 0.5 * gamma * (w @ Sigma @ w))
    bounds = [(0, None)] * N if long_only else None
    res = minimize(neg_obj, np.ones(N) / N, method="SLSQP",
                   constraints=[_budget_constraint()], bounds=bounds,
                   options={"maxiter": 1000, "ftol": 1e-11})
    return res.x


def min_var_weights(Sigma, long_only=False):
    N = len(Sigma)
    if not long_only:
        Si = np.linalg.inv(Sigma); one = np.ones(N)
        return Si @ one / (one @ Si @ one)
    obj = lambda w: w @ Sigma @ w
    res = minimize(obj, np.ones(N) / N, method="SLSQP",
                   constraints=[_budget_constraint()], bounds=[(0, None)] * N,
                   options={"maxiter": 500, "ftol": 1e-12})
    return res.x


# --------------------------------------------------------------------------- #
#  Robust frontier: sweep the estimation-risk aversion q                        #
# --------------------------------------------------------------------------- #

def robust_path(mu, Sigma, Theta, gamma, qs, long_only=False):
    """Return the weights for a grid of robustness radii q (0 = Bayesian MV,
    large q -> minimum variance)."""
    return np.array([robust_mv(mu, Sigma, Theta, q, gamma, long_only) for q in qs])


# --------------------------------------------------------------------------- #
#  Synthetic market                                                             #
# --------------------------------------------------------------------------- #

def make_true_market(N, rho=0.5, vol_lo=1.0, vol_hi=4.0, mu_scale=0.6, seed=0):
    rng = np.random.default_rng(seed)
    vols = np.linspace(vol_lo, vol_hi, N)
    C = (1 - rho) * np.eye(N) + rho * np.ones((N, N))
    Sigma = np.outer(vols, vols) * C
    mu = mu_scale * Sigma @ np.ones(N) / N
    return mu, Sigma, rng

References

• Meucci, A. (2005). Risk and Asset Allocation. Springer. — chapter 9, robust and Bayesian allocation as complementary responses to estimation risk
• Goldfarb, D. & Iyengar, G. (2003). Robust portfolio selection problems. Mathematics of Operations Research 28(1), 1–38. — the ellipsoidal uncertainty set and its tractable reformulation
• Ben-Tal, A. & Nemirovski, A. (1998). Robust convex optimization. Mathematics of Operations Research 23(4), 769–805. — the worst-case framework underlying the optimiser
• Tütüncü, R. H. & Koenig, M. (2004). Robust asset allocation. Annals of Operations Research 132, 157–187. — robust allocation in practice, and its relation to shrinkage
• Garlappi, L., Uppal, R. & Wang, T. (2007). Portfolio selection with parameter and model uncertainty. Review of Financial Studies 20(1), 41–81. — why ambiguity aversion and Bayesian shrinkage act on the same problem from different directions
• DeMiguel, V., Garlappi, L. & Uppal, R. (2009). Optimal versus naive diversification. Review of Financial Studies 22(5), 1915–1953. — the 1/N benchmark that wins the low-dimensional universe here