Finance — Shrinking the Factor Zoo

Python · scikit-learn · R (glmnet)  ·  Download factor data

The Problem

Empirical asset pricing has a factor zoo problem: hundreds of published "priced factors", most of them repackagings of a few themes. The modern response is to let regularisation pick the sparse subset that matters — Feng, Giglio & Xiu (2020), Kozak, Nagel & Santosh (2020). This applies the elastic-net machinery from Ridge, Lasso & Elastic Net to exactly that, on 20 long-short factors over 755 months (1963–2026, Ken French library), four of them value proxies and three low-risk proxies by construction.

What a "factor" actually is

Worth being concrete about the word, because it carries a lot of weight here. A factor is not a statistic computed on an index — it is itself the return on a portfolio: a monthly series in percent, and the dataset is twenty such series side by side. All but one are self-financing long–short portfolios. Ken French sorts every US stock on some characteristic, then buys the high group and sells the low group in equal dollar amounts — HML is long high book-to-market (value) and short low (growth), SMB long small caps and short large, Mom long the past year's winners and short its losers, and the Hi-minus-Lo anomaly series do the same across top and bottom deciles.

Because the short leg finances the long leg, the position costs roughly nothing to open, and what it earns is the premium attached to the characteristic with the market's general movement netted out. That is also why a factor can have a negative mean — the premium simply runs the other way, and you would hold the trade reversed. The one exception is Mkt-RF, which is long-only: the entire US market minus the risk-free rate. So the object built below is a portfolio of twenty trading strategies, not a portfolio of twenty stocks and not a tilt on an index — and "shrinking the zoo" means deciding how many of those strategies you actually need to run. The reason to expect fewer than twenty is that several are the same bet in different clothing: Value_BEME, Earn_EP, Cash_CFP and Div_DP are four ways of asking whether a stock is cheap.

From factors to a portfolio

The vehicle is the stochastic discount factor. Under a linear factor model the tangency (maximum-Sharpe) weights are b∝Σ−1μb \propto \Sigma^{-1}\mu, and a result of Britten-Jones (1999) turns that into a regression: the tangency weights are the coefficients of regressing a vector of ones on the factor excess returns, no intercept. That lets Ridge, Lasso and Elastic Net drop straight in and produce a penalised, sparse SDF.

Mt=1−b′(ft−Ef),b∝Σ−1μ  =  arg⁡min⁡b∥1−Fb∥2M_t = 1 - b'(f_t - E f), \qquad b \propto \Sigma^{-1}\mu \;=\; \arg\min_b \lVert \mathbf{1} - Fb \rVert^2

Reading the numbers: the Sharpe ratio

Everything below is measured in Sharpe ratios — mean excess return per unit of volatility, SR=rˉ/σr\text{SR}=\bar r/\sigma_r, annualised by 12\sqrt{12}. It is the natural yardstick here because the tangency portfolio is defined as the one that maximises it, and it is invariant to leverage, so a 20-factor portfolio and a single-factor one compare on equal terms. For scale: a broad equity index earns about 0.4–0.5 over long samples, a good active strategy sustains something near 1, and a backtest reporting 2+ is usually a diagnosis rather than an achievement.

How redundant is the zoo?

How redundant is the zoo, actually? Ten principal components are needed to reach 90% of the covariation among the 20 factors. On its own that number means nothing — the benchmark that makes it meaningful is that 20 independent series of this length need 18. So the redundancy is real and substantial. But it is not the "the zoo is really one factor" story sometimes told either: PC1 alone is only 32%, and it takes five components to reach 75%. Enough duplication for a selector to exploit; not enough to call the whole enterprise a facade.

The In-Sample Mirage

Fit the tangency SDF in-sample on 1963–2000 and it posts an annualised Sharpe of 2.41 — against the 0.4–0.5 an index earns. Held out on 2001–2026 the same weights deliver 0.40. That collapse is Σ−1μ\Sigma^{-1}\mu fitting estimation noise across correlated factors, and it is the identical instability treated with a prior in Bayesian Estimation & Estimation Risk and Shrinkage Estimation of Mean and Covariance.

The sharpest way to see the damage is the number beside it. The 20-factor tangency portfolio scores 0.40 out of sample; simply holding the market alone scores 0.55. Sixty years of data, twenty published factors, a fully optimised portfolio — and it cannot beat one of its own inputs. That is the estimation-risk problem stated as plainly as it can be.

SDF, annualised Sharpein-sample 1963–2000out-of-sample 2001–2026factors
kitchen-sink tangency (OLS)2.41 — the mirage0.4020
Lasso, cross-validated—0.4410
best point on the Lasso path—0.4512
market factor alone (Mkt-RF)—0.55 — beats them all1

Selection

The Lasso path tells a more interesting story than the usual one. In-sample Sharpe climbs with every factor added, as expected. Out-of-sample it does not plateau early — it rises steadily (0.13 at one factor, 0.28 at five, 0.39 at eight), peaks at 0.45 around a dozen factors, and then falls back to 0.40 at twenty. An inverted U, not a plateau. The cross-validated pick of 10 factors lands essentially on that peak, at 0.44, without ever seeing the test period — and so slightly beats the kitchen sink rather than merely matching it.

The pruning falls exactly where the duplication is. Of five value proxies only HML survives; of three low-risk proxies only residual variance. Market, size, momentum, both reversals and the profitability/investment pair are kept. That is the selector doing what the factor-zoo literature hopes it will do.

How much of the sparsity is the data?

One caveat the R companion measures rather than assumes, and it matters for reading this literature. How sparse the selected SDF is depends on the cross-validation rule more than on anything else. At lambda.min, glmnet keeps 19 of 20 factors and prunes nothing at all — retaining every value proxy and every low-risk proxy — and across six seeds it ranges 15–19. At the one-standard-error rule it holds steady at 10–12 and scores better out of sample (0.45 against 0.41). A paper reporting that "the Lasso selects a sparse SDF" is reporting its cross-validation rule at least as much as its data.

The verdict

The honest verdict is a modest one. With 20 well-diversified factors over 60 years, penalisation buys parsimony and interpretability, not a materially higher Sharpe — 0.44 against 0.40. Shrinkage's edge grows with the dimension of the problem; Kozak–Nagel–Santosh need hundreds of characteristic portfolios before it dominates. What this example does establish cleanly is the equivalence: penalising the SDF weights and putting a prior on μ\mu and Σ\Sigma are the same medicine for the same disease, approached from opposite directions.

Where this sits

The connections run in three directions. The method is the Lasso of Ridge, Lasso & Elastic Net, applied to finance. The object is the mean-variance portfolio of Shrinkage Estimation of Mean and Covariance, Bayesian Estimation & Estimation Risk and The Black–Litterman Model, where the same Σ−1μ\Sigma^{-1}\mu instability is treated with a prior instead of a penalty. And since the Lasso is the MAP estimate under a Laplace prior on the SDF weights, the fully Bayesian version — a posterior over which factors are priced — is the Variable Selection arc.

Notebooks

Python The factor zoo and its measured redundancy, the Britten-Jones tangency regression, the in-sample mirage against the market benchmark, the full Lasso path with its out-of-sample peak, and the sparse SDF with its survivors named (Python) Open → R Independent R implementation via glmnet — the same penalised SDF, plus a measurement of how far the selected sparsity depends on the cross-validation rule rather than the data (R) Open →

Downloads

factor_zoo.csv 20 long-short factor returns, monthly, 1963–2026 — the Fama-French five, momentum, both reversals and a dozen anomaly deciles fetch_factor_zoo.py Builds the dataset from the Ken French data library, so the whole example is reproducible from source

Data Builder — Source Code

# -*- coding: utf-8 -*-
"""Fetch a 'factor zoo' from the Ken French data library and assemble a monthly
panel of long-short factor returns -> factor_zoo.csv in the ex3 folder."""
import subprocess, zipfile, io, os, re, numpy as np, pandas as pd
BASE="https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/"
# both paths resolve against this file, so the script runs wherever the folder is checked out
HERE=os.path.dirname(os.path.abspath(__file__))
TMP=os.path.join(HERE, "_kf")           # scratch space for the downloaded zips
OUT=os.path.join(HERE, "factor_zoo.csv")
os.makedirs(TMP, exist_ok=True)

def dl(zipname):
    path=os.path.join(TMP, zipname)
    subprocess.run(["curl","-sL","-o",path,BASE+zipname], check=True)
    with zipfile.ZipFile(path) as z:
        nm=[n for n in z.namelist() if n.lower().endswith(".csv")][0]
        return z.read(nm).decode("latin-1")

def parse_monthly(text):
    """First monthly block (value-weighted) as a DataFrame indexed by YYYYMM int."""
    lines=text.splitlines()
    hdr=None
    for i,l in enumerate(lines):
        if l.strip().startswith(",") and any(ch.isalpha() for ch in l):
            hdr=i; break
    cols=[c.strip() for c in lines[hdr].split(",")]
    rows=[]
    for l in lines[hdr+1:]:
        m=re.match(r"^\s*(\d{6})\s*,",l)
        if not m: break                                  # stop at first non-monthly (blank / annual / next block)
        vals=[x.strip() for x in l.split(",")]
        rows.append([int(vals[0])]+[float(x) for x in vals[1:len(cols)]])
    df=pd.DataFrame(rows, columns=["ym"]+cols[1:]).set_index("ym")
    return df.replace([-99.99,-999,-99.99],np.nan)

# --- factor files (single-column factors, already long-short) ---
FACTORS={}
ff5=parse_monthly(dl("F-F_Research_Data_5_Factors_2x3_CSV.zip"))
for c in ["Mkt-RF","SMB","HML","RMW","CMA"]: FACTORS[c]=ff5[c]
RF=ff5["RF"]
FACTORS["Mom"]=parse_monthly(dl("F-F_Momentum_Factor_CSV.zip")).iloc[:,0]
FACTORS["ST_Rev"]=parse_monthly(dl("F-F_ST_Reversal_Factor_CSV.zip")).iloc[:,0]
FACTORS["LT_Rev"]=parse_monthly(dl("F-F_LT_Reversal_Factor_CSV.zip")).iloc[:,0]

# --- decile portfolio files -> long-short (Hi 10 - Lo 10), a deliberately redundant zoo ---
DECILE={"Value_BEME":"Portfolios_Formed_on_BE-ME_CSV.zip",
        "Size_ME":"Portfolios_Formed_on_ME_CSV.zip",
        "Prof_OP":"Portfolios_Formed_on_OP_CSV.zip",
        "Inv_INV":"Portfolios_Formed_on_INV_CSV.zip",
        "Earn_EP":"Portfolios_Formed_on_E-P_CSV.zip",
        "Cash_CFP":"Portfolios_Formed_on_CF-P_CSV.zip",
        "Div_DP":"Portfolios_Formed_on_D-P_CSV.zip",
        "Accr_AC":"Portfolios_Formed_on_AC_CSV.zip",
        "NetIss_NI":"Portfolios_Formed_on_NI_CSV.zip",
        "Var":"Portfolios_Formed_on_VAR_CSV.zip",
        "ResVar":"Portfolios_Formed_on_RESVAR_CSV.zip",
        "Beta":"Portfolios_Formed_on_BETA_CSV.zip"}
for name,zp in DECILE.items():
    try:
        df=parse_monthly(dl(zp)); cols={c.strip():c for c in df.columns}
        hi=df[cols.get("Hi 10","Hi 10")]; lo=df[cols.get("Lo 10","Lo 10")]
        FACTORS[name]=hi-lo
    except Exception as e:
        print("SKIP",name,repr(e)[:80])

panel=pd.DataFrame(FACTORS)
panel["RF"]=RF
panel=panel.dropna()
panel=panel[(panel.index>=196307)]
panel.to_csv(OUT)
print("saved", OUT, panel.shape)
print("date range:", panel.index.min(), "-", panel.index.max())
print("factors:", [c for c in panel.columns if c!="RF"])
print(panel.drop(columns="RF").mean().round(3).to_string())      # monthly mean returns (% ), sanity

References