PCA, SVD & Factor Analysis — the Cross-Section's Factors

Variance-Maximising Directions

Clustering groups the observations; dimensionality reduction compresses the features. PCA (principal component analysis) finds the orthogonal directions of maximum variance — a ranked coordinate system in which the first few axes carry most of the variation. In words: the first axis is the single direction along which the 48 stocks move most; the second is the direction of most remaining movement once the first has been accounted for, and so on, each at right angles to those before it. The clean way to compute all of them at once is the singular value decomposition — a standard factorisation that splits any matrix into three simpler ones — applied to the data after subtracting each column's mean, $X = USV^\top$: the columns of $V$ are the principal directions, $US$ the coordinates, and $S_k^2/(n-1)$ the variance along axis $k$. Built from scratch here and matched to scikit-learn exactly.

The market, rediscovered

The finance payoff is one of the more useful facts in the field: many assets are driven by a few common factors. On 48 stocks with a mean pairwise correlation of 0.38, PC1 explains 40.0% of the covariation, its loadings are all one sign — a long-everything portfolio — and its return correlates 0.998 with the equal-weight index. PCA rediscovers the market factor from the data alone, with no index supplied. Higher components are mixed-sign long/short style portfolios: PC2 goes long CVX, XOM and COP against AMZN, ADBE and NVDA, an energy-versus-technology contrast that is uncorrelated with the market rather than a bet on it — which is what orthogonality buys here: each component captures movement the earlier ones have already stripped out, so they do not double-count.

How Low-Rank Is It, Really?

The low-rank claim needs more care than it usually gets, because the obvious way to check it is wrong. The object to approximate is the correlation matrix itself, so the right construction is a spectral truncation — keep the top $k$ eigenpairs of $C$, which is provably its best rank-$k$ approximation in the Frobenius norm — the measure that squares every individual entry of the difference between two matrices and adds them all up, so no single correlation can be badly wrong without the total noticing.

The tempting alternative is to rebuild the data at rank $k$ and correlate that. It measures something else entirely. A rank-$k$ matrix has only $k$ independent directions, and correlating it rescales every column by its own reconstructed standard deviation — so the correlations are driven toward ±1 no matter how good the fit is. At rank 1 every pair reads exactly 1.000; at rank 3 the mean absolute correlation is 0.699 against the true 0.375. The reconstruction looks more correlated than the data, not closer to it.

Done properly, the structure is real and worth stating at its real size. A rank-3 truncation reproduces the correlation matrix to 19.5% relative error (the size of what the approximation missed, as a fraction of the size of the matrix it was approximating: 0% would be exact, 100% no better than returning nothing), or 12.2% on the off-diagonal entries alone — the diagonal is 1 by construction and flatters any reconstruction, so it is worth separating. Rank-10 brings those to 11.9% and 7.4%. Three numbers per stock carry most of a 48×48 correlation matrix, which is a strong result; "almost perfectly" would be overselling a 12% off-diagonal error.

What Factor Analysis Adds

PCA and factor analysis are often conflated but answer different questions. PCA finds directions of maximum total variance, mixing common co-movement with each stock's idiosyncratic noise. FA is an explicit probabilistic model, $X = Lf + \varepsilon$: each stock is a weighted combination of a few common latent factors — latent meaning inferred rather than observed, quantities no column of the data contains but whose existence would explain why the columns move together — plus a noise term belonging to that stock alone. Because the two appear separately in the model, the fit separates them. On this cross-section the average stock is 53% systematic and 47% diversifiable. That split is exactly the one that matters for risk, and the one PCA blurs.

Where this sits

The low-rank structure PCA exposes here is what Ledoit–Wolf shrinkage targets from the other side: PCA shows the structure directly, shrinkage uses it to stabilise a noisy sample covariance. The "many assets, few dimensions" result is the factor zoo finding again — where 10 principal components reach 90% of the covariation against 18 for independent series. And FA is the frequentist sibling of the Bayesian factor models elsewhere in the collection, which put a posterior on the same split — communality being the share of a variable explained by the shared factors, uniqueness the share left to itself, the two numbers that add to the 53%/47% quoted above.

Notebook

Downloads

PCA Module — Source Code

"""From-scratch principal component analysis via the singular value decomposition.

PCA finds the orthogonal directions of maximum variance in the (centered) data.
The clean way to compute it is the SVD of the centered matrix X = U S V^T:

    * the rows of V^T (columns of V) are the principal directions / loadings,
    * the scores (coordinates in the new basis) are X V = U S,
    * the variance explained by component k is S_k^2 / (n-1).

This is numerically preferable to eigen-decomposing the covariance matrix and is
exactly what scikit-learn's PCA does; the notebook validates the match.

Factor analysis (used via scikit-learn in the notebook) is the probabilistic
cousin: it models the data as a few COMMON latent factors plus per-variable
UNIQUE noise, X = L f + e, separating shared covariation from idiosyncratic
variance -- whereas PCA lumps all variance together into orthogonal directions.
"""
import numpy as np


def pca(X, standardize=False):
    """Principal component analysis by SVD.

    Parameters
    ----------
    X : (n_samples, n_features) data.
    standardize : if True, scale each feature to unit variance first
                  (correlation-based PCA); else covariance-based.

    Returns dict with scores, loadings (components as rows), explained-variance
    ratio, singular values, and the center/scale used.
    """
    X = np.asarray(X, float)
    mean = X.mean(0)
    Xc = X - mean
    scale = np.ones(X.shape[1])
    if standardize:
        scale = Xc.std(0, ddof=1); scale[scale == 0] = 1.0
        Xc = Xc / scale
    U, S, Vt = np.linalg.svd(Xc, full_matrices=False)
    var = S ** 2 / (len(X) - 1)
    return {
        "scores": U * S,            # coordinates of each sample on the components
        "loadings": Vt,             # principal directions (one per row)
        "explained_variance": var,
        "explained_variance_ratio": var / var.sum(),
        "singular_values": S,
        "mean": mean, "scale": scale,
    }


def reconstruct(res, n_components):
    """Rank-`n_components` reconstruction of the (centered, scaled) data."""
    Z = res["scores"][:, :n_components] @ res["loadings"][:n_components]
    return Z * res["scale"] + res["mean"]

References

• Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology 24(6), 417–441. — PCA
• Eckart, C. & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika 1(3), 211–218. — why the spectral truncation is the right rank-k approximation
• Connor, G. & Korajczyk, R. A. (1988). Risk and return in an equilibrium APT. Journal of Financial Economics 21(2), 255–289. — principal components as asset-pricing factors
• Bartlett, M. S. (1937). The statistical conception of mental factors. British Journal of Psychology 28(1), 97–104. — factor analysis and the common/unique split
• Ledoit, O. & Wolf, M. (2004). A well-conditioned estimator for large-dimensional covariance matrices. Journal of Multivariate Analysis 88(2), 365–411. — the structure exploited from the other direction