Kalman Filter

Definition

The Kalman filter is a recursive algorithm that computes the minimum mean squared error (MMSE) linear estimator of the latent state vector $\boldsymbol{\alpha}_t$ in a linear Gaussian state-space model, given all observations up to time $t$ . At each step it alternates between a prediction step (propagating the state one period forward via the transition equation) and an update step (revising the prediction using the new observation and the Kalman gain). Under Gaussianity the filtered estimates equal the conditional means, and the filter delivers the exact log-likelihood as a by-product via the innovation decomposition.

Key Ideas

Optimality: Among all linear estimators, the Kalman filter minimises the mean squared error of $\hat{\boldsymbol{\alpha}}_{t|t} = E[\boldsymbol{\alpha}_t \mid y_1, \ldots, y_t]$ ; under joint normality it is also the minimum-variance unbiased estimator (MVUE).
Innovation representation: The one-step-ahead prediction error $\mathbf{v}_t = \mathbf{y}_t - \mathbf{Z}_t \mathbf{a}_{t|t-1}$ is serially uncorrelated with variance $\mathbf{F}_t = \mathbf{Z}_t \mathbf{P}_{t|t-1} \mathbf{Z}_t' + \mathbf{H}_t$ . The log-likelihood factorises as $\ell = -\frac{1}{2}\sum_t (\log |\mathbf{F}_t| + \mathbf{v}_t' \mathbf{F}_t^{-1} \mathbf{v}_t)$ , making maximum likelihood estimator (MLE) evaluation $O(T)$ .
Kalman gain: $\mathbf{K}_t = \mathbf{P}_{t|t-1}\mathbf{Z}_t'\mathbf{F}_t^{-1}$ is the optimal weight given to the innovation, balancing state uncertainty $\mathbf{P}_{t|t-1}$ against measurement noise $\mathbf{H}_t$ .
Kalman smoother: A backward Rauch-Tung-Striebel or de Jong pass computes the smoothed estimates $\mathbf{a}_{t|T} = E[\boldsymbol{\alpha}_t \mid y_1,\ldots,y_T]$ , using all $T$ observations. Required for the Expectation-Maximization (EM) algorithm's E-step and for simulation smoothers used in Bayesian Markov Chain Monte Carlo (MCMC) estimation.
Diffuse initialisation: When the initial state distribution is unknown, the filter starts with $\mathbf{P}_{1|0} \to \infty$ ; the exact diffuse filter (Koopman 1997; Durbin-Koopman 2001) handles this rigorously without ad-hoc large constants (see below).
Kim filter: For Markov-switching (MS) state-space models, running separate filters per regime generates $M^t$ branches. Kim's (1994) probability-weighted collapse approximation reduces this to $O(M^2 T)$ ; see Plucking Model and Markov-Switching VAR.

How It Works

State-Space Form

$\mathbf{y}_t = \mathbf{Z}_t \boldsymbol{\alpha}_t + \boldsymbol{\varepsilon}_t, \qquad \boldsymbol{\varepsilon}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{H}_t) \tag{measurement}$

$\boldsymbol{\alpha}_t = \mathbf{T}_t \boldsymbol{\alpha}_{t-1} + \mathbf{R}_t \boldsymbol{\eta}_t, \qquad \boldsymbol{\eta}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{Q}_t) \tag{transition}$

where $\mathbf{y}_t$ is $p \times 1$ observed, $\boldsymbol{\alpha}_t$ is $m \times 1$ latent, and $(\boldsymbol{\varepsilon}_t, \boldsymbol{\eta}_t)$ are mutually uncorrelated at all leads and lags.

Prediction Step

$\mathbf{a}_{t|t-1} = \mathbf{T}_t \mathbf{a}_{t-1|t-1}, \qquad \mathbf{P}_{t|t-1} = \mathbf{T}_t \mathbf{P}_{t-1|t-1} \mathbf{T}_t' + \mathbf{R}_t \mathbf{Q}_t \mathbf{R}_t'$

Update Step

$\mathbf{v}_t = \mathbf{y}_t - \mathbf{Z}_t \mathbf{a}_{t|t-1}, \qquad \mathbf{F}_t = \mathbf{Z}_t \mathbf{P}_{t|t-1} \mathbf{Z}_t' + \mathbf{H}_t$

$\mathbf{K}_t = \mathbf{P}_{t|t-1} \mathbf{Z}_t' \mathbf{F}_t^{-1}$

$\mathbf{a}_{t|t} = \mathbf{a}_{t|t-1} + \mathbf{K}_t \mathbf{v}_t, \qquad \mathbf{P}_{t|t} = (\mathbf{I} - \mathbf{K}_t \mathbf{Z}_t) \mathbf{P}_{t|t-1}$

Kalman Smoother (Rauch-Tung-Striebel)

Run the filter forward; then pass backward for $t = T-1, \ldots, 1$ :

$\mathbf{L}_t = \mathbf{P}_{t|t} \mathbf{T}_{t+1}' \mathbf{P}_{t+1|t}^{-1}$

$\mathbf{a}_{t|T} = \mathbf{a}_{t|t} + \mathbf{L}_t (\mathbf{a}_{t+1|T} - \mathbf{a}_{t+1|t}), \qquad \mathbf{P}_{t|T} = \mathbf{P}_{t|t} + \mathbf{L}_t (\mathbf{P}_{t+1|T} - \mathbf{P}_{t+1|t}) \mathbf{L}_t'$

Log-Likelihood for MLE

$\ell(\theta) = -\frac{Tp}{2}\log(2\pi) - \frac{1}{2}\sum_{t=1}^T \left(\log|\mathbf{F}_t| + \mathbf{v}_t'\mathbf{F}_t^{-1}\mathbf{v}_t\right)$

Maximised over model parameters $\theta$ via quasi-Newton methods (e.g., Broyden-Fletcher-Goldfarb-Shanno (BFGS)).

Exact Diffuse Initialisation (Koopman 1997)

Nonstationary models (regression effects, stochastic trend/seasonal, ARIMA components) have a partly diffuse initial state: $\boldsymbol{\alpha}_1 = \mathbf{a} + \mathbf{A}\boldsymbol{\eta} + \mathbf{B}\boldsymbol{\delta}$ with $\boldsymbol{\delta}\sim\mathcal N(\mathbf 0,\kappa\mathbf I)$ and $\kappa\to\infty$ , so the initial variance splits as $\mathbf{P}_1 = \mathbf{P}_* + \kappa\mathbf{P}_\infty$ . The ordinary filter cannot run because $\mathbf{P}_\infty\neq\mathbf 0$ . The historical fix — set $\kappa$ to a large constant ( $\sim 10^7$ ) — is inexact and rounding-prone.

Koopman (1997) obtains the exact solution analytically: write each filtering quantity as an explicit function of $\kappa$ ( $\mathbf{F}_t=\mathbf{F}_{*,t}+\kappa\mathbf{F}_{\infty,t}$ , $\mathbf{M}_t=\mathbf{M}_{*,t}+\kappa\mathbf{M}_{\infty,t}$ ), expand $\mathbf{F}_t^{-1}$ in powers of $1/\kappa$ , and let $\kappa\to\infty$ to get exact recursions for $\mathbf{a}_{t+1}$ , the finite part $\mathbf{P}_{*,t+1}$ , and the diffuse part $\mathbf{P}_{\infty,t+1}$ — from a trivial initialization $(\mathbf{P}_{*,1},\mathbf{P}_{\infty,1})$ . As diffuse information is consumed the rank of $\mathbf{P}_{\infty,t}$ falls ( $r(\mathbf{P}_{\infty,t+1})\le r(\mathbf{P}_{\infty,t})-r(\mathbf{F}_{\infty,t})$ ) and the recursion collapses automatically back to the ordinary Kalman filter — no manual switch. An exact diffuse smoother, the diffuse likelihood and score, and clean missing-observation handling (delete the relevant rows of $\mathbf{Z}_t,\mathbf{G}_t$ ) all follow. This supersedes the Ansley-Kohn (1985, 1990) transformation (exact but hard to implement) and de Jong's (1991) augmented diffuse Kalman filter (exact but costlier, with awkward smoothing corrections), and it underlies later state-space software (SsfPack) and the Durbin-Koopman (2001) textbook treatment.

Why It Matters

The innovation decomposition converts the joint density of $(y_1,\ldots,y_T)$ into a product of conditionals, making exact likelihood evaluation $O(T)$ instead of $O(T^3)$ . This is the computational foundation for likelihood-based estimation of all linear state-space models.
Time-varying parameter vector autoregressions (VARs), unobserved-components models, dynamic factor models, and affine term-structure models all rely on the Kalman filter for parameter estimation and state extraction.
The smoother's output drives the E-step of the EM algorithm and feeds the simulation smoother (de Jong-Shephard 1995; Durbin-Koopman 2002) used in Bayesian MCMC state-space estimation.
The Kim filter extends the Kalman recursion to Markov-switching models, enabling approximate maximum likelihood estimation of state-space models with discrete regime changes at $O(M^2 T)$ cost.

Open Questions

Nonlinear and non-Gaussian extensions: The extended Kalman filter (EKF) linearises nonlinear measurement/transition equations; the unscented Kalman filter (UKF) uses sigma points; particle filters (sequential Monte Carlo (SMC)) are asymptotically exact but expensive. None dominates uniformly.
High-dimensional systems: When $m$ or $p$ is large (e.g., large dynamic factor models), the $m \times m$ matrix inversions in $\mathbf{P}_{t|t-1}$ become costly; low-rank and sparse representations are active research areas.
Exact vs. approximate regime-switching: Kim's (1994) approximation error is hard to assess analytically; the exact Gibbs-sampling approach of Kim-Nelson (1999) is more accurate but more expensive.