Gal-Ghahramani (2016) Dropout as a Bayesian Approximation

Summary

Gal and Ghahramani develop a theoretical framework casting dropout training in deep neural networks as approximate Bayesian inference in deep Gaussian processes. A direct consequence is a practical tool — Monte Carlo dropout — for representing model uncertainty in existing dropout networks without changing the architecture, sacrificing test accuracy, or increasing computational cost. They study the properties of dropout's uncertainty across architectures and nonlinearities on regression and classification (MNIST), report improved predictive log-likelihood and RMSE over prior methods, and demonstrate the uncertainty in deep reinforcement learning.

Key Claims

• Dropout as variational inference. Dropout training with $L_2$ regularization minimizes the KL divergence between an approximate weight posterior (units zeroed with probability $p$) and the posterior of a deep Gaussian process — so dropout networks are already performing approximate Bayesian inference.
• MC-dropout prediction. Keeping dropout active at test time and averaging $T$ stochastic forward passes approximates the predictive mean; the empirical scatter of the passes gives uncertainty.
• Predictive variance. $\mathrm{Var}(y^*)\approx\tau^{-1}I+\tfrac1T\sum_t\hat y_t^\top\hat y_t-\bar y^\top\bar y$ = sample variance of the passes plus inverse model precision $\tau^{-1}$, with $\tau=pl^2/(2N\lambda)$ linking precision to dropout rate $p$, length-scale $l$, data size $N$, and weight decay $\lambda$.
• Predictive log-likelihood is estimated by Monte Carlo integration (log-sum-exp over the $T$ passes), scoring fit of mean and uncertainty jointly.
• Free uncertainty. The method extracts uncertainty "thrown away" by standard dropout, requiring no retraining and no loss of accuracy; it improved predictive log-likelihood/RMSE and drove exploration in deep RL.

Concepts Introduced or Extended

• Monte Carlo Dropout

Entities Mentioned

• Yarin Gal
• Zoubin Ghahramani

Quotes

"We develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs – extracting information from existing models that has been thrown away so far."

My Take

A high-leverage reinterpretation: by proving that ordinary dropout is variational inference in a deep GP, it hands practitioners Bayesian uncertainty essentially for free — run the network a few dozen times with dropout on and read off the scatter. That practicality is why MC-dropout became a ubiquitous epistemic-uncertainty baseline. The caveats are real and worth stating alongside the recipe: the Bernoulli variational family is a coarse posterior approximation, the uncertainty can be poorly calibrated, and — unlike a genuine posterior — it need not concentrate correctly as data grow. It belongs to the same "how confident is the model?" conversation as conformal prediction and calibration, but answers it with a model-based Bayesian approximation rather than a coverage guarantee.