Conformal Prediction

Definition

Conformal prediction is a framework for turning the point predictions of any model into prediction sets/intervals with a finite-sample, distribution-free coverage guarantee: for a user-chosen error rate $\alpha$, the set $C(X_{test})$ is built so that it contains the true label with probability at least $1-\alpha$, with no assumptions on the data distribution or the model, requiring only that the data be exchangeable (Angelopoulos-Bates 2023).

Key Ideas

• Split (inductive) conformal — the standard recipe. Given a pre-trained model and a held-out calibration set of size $n$: (1) pick a score function $s(x,y)$ where larger = worse agreement (e.g. $s=1-\hat f(x)_y$, one minus the softmax of the true class); (2) compute $\hat q=$ the $\lceil (n+1)(1-\alpha)\rceil/n$ empirical quantile of the calibration scores; (3) form $C(X_{test})=\{y: s(X_{test},y)\le \hat q\}$.
• Coverage guarantee (Vovk–Gammerman–Saunders). If $(X_i,Y_i)$ and $(X_{test},Y_{test})$ are exchangeable (i.i.d. suffices), then $1-\alpha \le P(Y_{test}\in C(X_{test})) \le 1-\alpha+\tfrac{1}{n+1}$ — exact up to the $1/(n+1)$ discretization, non-asymptotic, and free of distributional/model assumptions.
• Marginal, not conditional. The probability is marginal — averaged over the randomness of calibration and test points. Conditional coverage ($P(Y\in C(X)\mid X=x)\ge 1-\alpha$ for every $x$) is strictly stronger and provably unattainable distribution-free; good score functions approximate it.
• The score function is where the modelling lives. Validity holds for any score, but the size and usefulness (adaptivity) of the sets depend entirely on a well-chosen score; a bad score yields valid-but-wide sets.
• Adaptivity. Desirable sets are larger on hard inputs and smaller on easy ones; the paper stresses evaluating adaptivity (e.g. size-stratified coverage), not just marginal coverage.

How It Works

Conformal prediction is a wrapper: it takes a heuristic notion of uncertainty from a black-box model and converts it into a rigorous one, at the cost of one held-out calibration split and a single quantile computation. The guarantee comes from exchangeability: the test score is equally likely to fall at any rank among the $n+1$ scores, so thresholding at the $(1-\alpha)$ empirical quantile controls the miscoverage rate. Named procedures build specific score functions: Adaptive Prediction Sets (APS) and Conformalized Quantile Regression (CQR) for regression intervals, plus recipes for conformalizing standard-deviation estimates and Bayesian posteriors.

Why It Matters

• Model-agnostic, assumption-light UQ for high-stakes deployment (medical, vision, NLP, RL): wrap any pre-trained model and get calibrated-coverage sets with a few lines of code.
• Separates coverage from calibration. Distinct from Probability Calibration (which asks that stated probabilities match observed frequencies): conformal targets set coverage at a chosen level, a different guarantee — the two are complementary tools for uncertainty quantification.
• Extensible. Group-balanced/class-conditional coverage, weighted conformal for distribution shift, methods for time series, outlier/anomaly detection, models that abstain, and conformal risk control extend the core beyond exchangeable i.i.d. data.

Open Questions

• Exchangeability breaks under distribution shift and temporal dependence; time-series and covariate-shift extensions restore coverage only under extra assumptions or with weighting.
• Conditional coverage is impossible distribution-free; how close can practical score functions get, and how to certify it?
• Efficiency vs validity: designing score functions that minimize set size while preserving coverage, especially for structured/high-dimensional outputs.

Related

• Probability Calibration
• Angelopoulos-Bates (2023) Conformal Prediction: A Gentle Introduction
• Anastasios N. Angelopoulos
• Stephen Bates
• Monte Carlo Dropout