Factor Model

Definition

A factor model represents a $p$-dimensional observed vector $y$ as a linear combination of $k < p$ latent factors $f$ plus idiosyncratic noise: $y = \mu + Bf + \varepsilon$, where $B$ is a $p \times k$ factor loading matrix, $f \sim \mathcal{N}(0, V)$, and $\varepsilon \sim \mathcal{N}(0, D)$ with $D$ diagonal. The implied covariance structure is $\Sigma = BVB' + D$ — a low-rank-plus-diagonal decomposition. Factor models reduce the number of free covariance parameters from $O(p^2)$ to $O(pk)$ loadings plus $p$ idiosyncratic variances.

Key Ideas

• Dimensionality reduction: With $k \ll p$, the model captures common variation through $k$ factors while treating the remainder as idiosyncratic. Critical when $n < p(p+1)/2$ makes the Wishart estimator degenerate or severely biased.
• Identification (Geweke-Zhou 1996): The loadings matrix is generically unidentified up to rotation: $Bf = (BR)(R^{-1}f)$ for any orthogonal $R$. The standard fix is to constrain $B$ to be lower triangular with positive diagonal ($b_{jj} > 0$ and $b_{js} = 0$ for $s > j$). This requires a meaningful ordering of the $p$ series.
• Factor covariance $V$: Setting $V = I_k$ forces orthogonal factors. Freeing $V$ to be a general positive-definite matrix allows correlated factors and, combined with lower-triangular $B$, produces sparser loading matrices (Hahn-Carvalho-Scott 2012): inter-factor correlation is absorbed by $V$ rather than off-diagonal entries of $B$.
• Sparse loading priors: When $p$ is large, further regularization via spike-and-slab priors on $b_{js}$ is desirable. The inclusion probability $q_s \sim \text{Beta}(1,1)$ per factor column provides automatic multiplicity correction (Scott-Berger 2006). See Spike-and-Slab Prior and Sparse Factor Probit.
• Static vs. dynamic: The model above is static (cross-sectional or i.i.d. observations). In time-series applications, the factor variances $H_t$ and idiosyncratic variances $\Psi_t$ evolve over time — this is the Dynamic Factor Model. See that page for state-space representation, Carter-Kohn FFBS sampling, and stochastic volatility extensions.

How It Works

Given $B$, $V$, $D$, the factors $f_i$ for each observation are Normal with posterior mean and variance: $$f_i \mid y_i \sim \mathcal{N}(m_i, M), \quad M = (V^{-1} + B'D^{-1}B)^{-1}, \quad m_i = M B'D^{-1}(y_i - \mu)$$

With a lower-triangular $B$ and free $V$, the Gibbs sampler cycles over:

1. Factor scores $f_i \mid B, V, D, y_i$ — multivariate Normal (above)
2. Factor covariance $V \mid \{f_i\} \sim \text{IW}$ (conjugate)
3. Each loading row $B_j \mid \{f_i\}, D, y$ — Normal (regression of $y_j$ on $f$)
4. Idiosyncratic variances $D_{jj} \mid \{f_i\}, B, y$ — Inverse-Gamma
5. Sparse loading indicators (if spike-and-slab) — Bernoulli via Bayes factor

Why It Matters

Factor models are the principal tool for high-dimensional covariance estimation in Bayesian statistics. They underpin multivariate stochastic volatility (Aguilar-West 2000: $k=2$ factor SV for 20 exchange rates, 200 free parameters instead of $O(p^2)$), sparse multivariate probit (Hahn-Carvalho-Scott 2012: roll-call votes), and term-structure models. Simulation evidence (Hahn-Carvalho-Scott 2012, Table 1): at $p=100$, $n=50$, the sparse-6-factor model achieves Stein loss 43.4 versus 503.1 for the unstructured Wishart estimator.

Open Questions

• Choosing $k$: standard practice uses cross-validation, likelihood-based criteria (BIC, DIC), or reversible-jump MCMC. Overfitting $k$ is common and difficult to detect post-hoc.
• Asymmetric factor priors (e.g., half-Cauchy on loadings) versus symmetric spike-and-slab are an active trade-off: the former scales better in $p$ but sacrifices exact sparsity.
• The lower-triangular identification imposes a series ordering that can influence results; robustness to this ordering is rarely checked.

Related

• Dynamic Factor Model
• Spike-and-Slab Prior
• Sparse Factor Probit
• Covariance Matrix Estimation
• Aguilar and West (2000)
• Hahn, Carvalho, Scott (2012)