Conformal Prediction — Guaranteed Coverage, and What It Does Not Guarantee

Python · scikit-learn · Open the notebook

A Guarantee That Needs No Model to Be Right

Conformal prediction wraps any trained model and, using only a held-out calibration set, produces intervals (regression) or label sets (classification) whose coverage is guaranteed to be at least $1-\alpha$ — in finite samples, with no distributional assumptions, and with no assumption that the model is any good. It is the natural complement to the Bayesian credible intervals elsewhere in the collection, which are only as trustworthy as the model that produced them.

The mechanism is three lines. On the calibration set compute a conformity score for each point — the absolute residual $|y-\hat y|$ for regression. Then take a high percentile of those scores as a threshold $\hat q$ — for 90% coverage, roughly the 90th percentile of the calibration errors, so that about nine in ten calibration points scored below it. For a new point, the prediction set is every outcome scoring at or below $\hat q$ : in words, anything that would not have looked unusual on the calibration data. The exact rule is the $\lceil (n+1)(1-\alpha)\rceil/n$ empirical quantile, and the small correction from $n$ to $n+1$ is what turns an approximation into a finite-sample guarantee: it accounts for the new point itself being one of the exchangeable draws. On California housing with a gradient booster, empirical coverage tracks the target at every level — 0.897 against 0.90, 0.952 against 0.95.

\hat q = \text{Quantile}\Bigl(\{s_i\}_{i=1}^{n};\ \tfrac{\lceil (n+1)(1-\alpha)\rceil}{n}\Bigr), \qquad C(x) = \bigl[\hat y(x) - \hat q,\ \hat y(x) + \hat q\bigr]

What “Marginal” Costs

The guarantee is marginal: coverage is at least $1-\alpha$ averaged over the whole population, and it says nothing about any subgroup. That is the single most important caveat about the method, so it is measured rather than mentioned — splitting the test set by the model's own predicted value and checking coverage decile by decile.

A constant-width 90% interval covers 98% of the cheapest decile of houses and 72% of the eighth, a spread of 0.26. The marginal claim is completely true and almost useless for pricing an individual property: for expensive houses it is really a 72% interval. Marginal validity is exactly as narrow as it sounds.

decile of predicted value	mean prediction	split conformal	CQR	CQR width
0 (cheapest)	0.72	0.981	0.937	1.06
2	1.31	0.969	0.864	1.30
4	1.75	0.944	0.901	1.43
6	2.23	0.862	0.881	1.87
8	3.18	0.719	0.855	2.50
9 (dearest)	4.20	0.811	0.935	2.56
marginal coverage	—	0.897	0.895	—
spread across deciles	—	0.262	0.082	—

Conformalized quantile regression is the repair. Fit models for the 5% and 95% quantiles, then conformalize those quantiles on the calibration set to restore the finite-sample guarantee. What it buys is worth stating precisely, because the obvious metric points the wrong way: CQR's mean interval is 9% wider than split conformal's, so on average width alone it looks like a step backwards. What it actually does is spend that width where the uncertainty is — 1.06 in the cheapest decile rising to 2.56 in the dearest — cutting the coverage spread from 0.26 to 0.08. That trade, not the varying widths themselves, is the reason to prefer it.

Neither method achieves exact conditional coverage, and no distribution-free method can — that is a theorem (Foygel Barber et al., 2021), not a shortcoming of the implementation. Approximate conditional validity is the most that is available, which makes checking it part of the job rather than an optional extra.

Prediction Sets for Classification

For classification the output is a set of labels. Adaptive Prediction Sets score a point by the cumulative sorted class probability down to and including the true label, so sets grow for ambiguous inputs and shrink for easy ones. The construction has to mirror that score exactly: labels are kept while the running total stays at or below $\hat q$ , and the label that pushes the total past $\hat q$ belongs outside the set. Including it anyway adds one label to every set — costing 34% in set size and pushing coverage to 0.955 against a 0.90 target. Over-covering is not free, because a prediction set is only useful to the extent that it is small.

noisy digits, target coverage 0.90	coverage	mean set size
set matched to the score definition	0.916	2.83
“add the label, then test”	0.955	3.79 +34%

Done correctly, coverage lands at 0.916 rather than 0.900 — expected, since this deterministic form of APS over-covers by construction without a randomised tie-breaking term. On deliberately noisy digits where the base classifier is only 68% accurate, set size carries the uncertainty: 2.65 labels when the model is right against 3.22 when it is wrong. A conformal classifier does not become more accurate — it becomes honest about when to hedge.

Conformal vs Bayesian

The two philosophies of uncertainty in the collection are complementary. Bayesian methods give a full predictive distribution and decompose the uncertainty, but their coverage is only correct if the model is — in the Bayesian Nonparametrics benchmark a Gaussian process's 90% intervals covered about 82%. Conformal gives a guarantee on any model, distribution-free, but only marginally and as a set rather than a distribution. The strongest option is to combine them: conformalize a Bayesian model's output and get both a principled distribution and a guarantee.

Where this sits

This is the fourth of the evaluation examples and they share a method: establish what a metric reports when there is nothing to report, then read the result against that. Calibration simulates an ECE noise floor, Model Selection grades criteria on fresh draws, and Interpretability hands the measures a column of pure noise. Conformal is the strongest statement of the four because the guarantee holds without trusting the model at all — which is also precisely why it can only be marginal.

Notebook

Python Split conformal built from scratch on California housing with the guarantee verified at six coverage levels, conditional coverage measured decile by decile to show what marginal validity does not promise, CQR justified on the trade it actually makes, and adaptive prediction sets for classification with the score and set construction matched (Python) Open →

Downloads

cali_housing.csv California housing, 20,640 census block groups — eight predictors and median house value in $100k, capped at 5.0

References

Vovk, V., Gammerman, A. & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer. — conformal prediction
Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J. & Wasserman, L. (2018). Distribution-free predictive inference for regression. JASA 113(523), 1094–1111. — split conformal
Romano, Y., Patterson, E. & Candès, E. (2019). Conformalized quantile regression. NeurIPS 32. — CQR
Romano, Y., Sesia, M. & Candès, E. (2020). Classification with valid and adaptive coverage. NeurIPS 33. — adaptive prediction sets
Foygel Barber, R., Candès, E., Ramdas, A. & Tibshirani, R. J. (2021). The limits of distribution-free conditional predictive inference. Information and Inference 10(2), 455–482. — why exact conditional coverage is unattainable
Angelopoulos, A. N. & Bates, S. (2023). Conformal prediction: a gentle introduction. Foundations and Trends in Machine Learning 16(4), 494–591.