angelopoulos-bates-2023

---

title: Angelopoulos-Bates (2023) Conformal Prediction: A Gentle Introduction tags: [conformal-prediction, uncertainty-quantification, distribution-free, prediction-interval, coverage, exchangeability, machine-learning, literature-survey] sources: [] updated: 2026-08-12 kind: paper author: Anastasios N. Angelopoulos, Stephen Bates date: 2023-01-01 url:

Summary

A hands-on tutorial on conformal prediction and distribution-free uncertainty quantification. It shows how to wrap any pre-trained black-box model with a calibration step that produces prediction sets guaranteed to contain the truth with a user-specified probability, using only the assumption of exchangeable data. The paper walks through the split-conformal recipe, worked examples (adaptive prediction sets, conformalized quantile regression, conformalizing Bayes), how to evaluate the sets, and extensions to structured outputs, distribution shift, time series, and outliers — with Python/Jupyter code throughout.

Key Claims

• Split-conformal algorithm. With a calibration set of size $n$: choose a score $s(x,y)$ (larger = worse fit), set $\hat q=\lceil (n+1)(1-\alpha)\rceil/n$ empirical quantile of the calibration scores, and output $C(X_{test})=\{y: s(X_{test},y)\le\hat q\}$.
• Coverage guarantee (Theorem 1; Vovk–Gammerman–Saunders): for exchangeable (or i.i.d.) data, $1-\alpha \le P(Y_{test}\in C(X_{test})) \le 1-\alpha+\tfrac{1}{n+1}$ — non-asymptotic, distribution-free, model-free.
• Marginal coverage. The guarantee is averaged over calibration and test randomness; conditional coverage is strictly stronger and unattainable without assumptions.
• Validity is free, adaptivity is earned. Any score function yields valid sets, but set size and input-adaptivity depend on choosing a good score; the tutorial devotes attention to evaluating adaptivity and the effect of calibration-set size.
• Named procedures. Adaptive Prediction Sets (APS) for classification, Conformalized Quantile Regression (CQR) for regression intervals, and recipes for conformalizing scalar uncertainty estimates and Bayesian posteriors.
• Extensions. Group-balanced and class-conditional coverage, weighted conformal for distribution shift, conformal methods for time series, outlier/anomaly detection, abstaining models, and conformal risk control.

Concepts Introduced or Extended

• Conformal Prediction

Entities Mentioned

• Anastasios N. Angelopoulos
• Stephen Bates

Quotes

"Conformal prediction can be seen as a method for taking any heuristic notion of uncertainty from any model and converting it to a rigorous one."

"One can use conformal prediction with any pre-trained model … to produce sets that are guaranteed to contain the ground truth with a user-specified probability … valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions."

My Take

This is the reference on-ramp that made conformal prediction go mainstream — its value is pedagogical clarity rather than new theorems (the coverage result is Vovk et al.'s). The framing "validity is free, but a good score function is what buys you tight, adaptive sets" is the single most useful takeaway, and the honest treatment of the marginal-vs-conditional distinction keeps practitioners from over-claiming. The obvious caveat for a time-series wiki: the clean guarantee rests on exchangeability, which temporal dependence and distribution shift violate — the paper's time-series/weighted extensions restore coverage only under added structure, and that is exactly the frontier worth watching.