Tabular Foundation Models

A Model That Does Not Train On Your Data

Every other model in this section is fitted to the data in front of it — gradient descent over a network's weights, a boosting loop over residuals, a closed-form ridge solution. A tabular foundation model does none of that. Its weights were fixed long before it saw this dataset, and the training rows arrive as input at prediction time, the way a prompt arrives at a language model. On this problem .fit takes 0.3 s and does nothing but store the table; the work happens in .predict, where the rows are pushed through the transformer as context.

The training that did happen is the interesting part. A prior-data fitted network is trained on millions of synthetic datasets: sample a causal graph, sample functional forms and noise, generate a table, hide some rows, and learn to predict them. What the network acquires is not any one dataset but the mapping from "here is a labelled table" to "here is the predictive distribution for a new row".

That makes a forward pass an approximation to the posterior predictive distribution — the same object the Bayesian notebooks in this collection compute with MCMC. The PFN does not sample it; it was trained to emit it. The inference is amortised: the expensive part happened once, offline, and is reused for every dataset afterwards. The price is that the prior lives in the weights, where it cannot be written down, varied in a sensitivity check, or argued with — the exact opposite of the explicit priors used everywhere else here.

Does It Compete?

The first question is whether a model that never takes a gradient step on this series can stand in a roster of models that were all fitted to it. Dropped into the capstone's race unchanged — same eleven engineered features, same 80/20 temporal split, same 2,607 training and 652 test days — it lands fourth at 0.3311, and a Diebold–Mariano test against the leading ridge returns t = 0.43, p = 0.67. Indistinguishable from the best model in the race, with no tuning of any kind, in 16 s on a CPU.

How Much History Does It Need?

The claim in circulation is that these models win hardest when data is scarce, since there is not enough of it to tune a competitor. That is testable rather than quotable, so the notebook tests it: hold the test set fixed and vary only how much history each model receives, always the most recent days before the split, re-tuning every competitor at each size so nothing is handicapped by construction.

The claim is half right, and the half that fails is the instructive one. Against the other flexible learners it holds emphatically. With 150 days of history a gradient-boosted ensemble has nowhere near enough data to fit depth-three trees on eleven collinear features and it falls apart at 0.4350; TabPFN, carrying a prior learned from millions of synthetic tables, degrades gracefully to 0.3809 — better by 0.0541 at $p < 0.001$, and better than the random forest by 0.0231.

Against a well-specified linear model it fails. Realized volatility is close to linear in these features — which is why HAR-RV has survived as a benchmark for fifteen years — and at 150 days a ridge exploits that structure better than any general-purpose prior can, winning by 0.0210 at $p < 0.001$. TabPFN needs roughly 600 observations to draw level, and once level it stays level rather than pulling ahead.

The lesson generalises past this one model. A foundation model's prior substitutes for data, not for knowing something about the problem. Where the truth is simple and you know it, saying so is still worth more than any amount of pretraining. What the prior buys is insurance against the case where you do not know — and against having to tune a booster you cannot afford to tune. Read alongside the capstone's own finding, that the gain came from the features and the simplest model collected it, a model with no tuning at all does not overturn the conclusion: it makes flexibility cheap without making it profitable.

Its Uncertainty Is Not Calibrated

A PFN is trained to emit a distribution rather than a point, so it can be asked for quantiles directly — no bootstrap, no ensemble, no variational approximation. That is a real advantage over most of the roster. Whether the distribution is correct is a separate question, and one this collection asks of every model that claims uncertainty. Its nominal 90% interval covers 81.6% of the test days.

The standard repair is split-conformal: size a band so that 90% of held-out calibration errors would have fitted inside it. That lifts coverage to 85.0% — better, and still short. The reason is visible in the residuals. Calibration-period absolute errors average 0.219 with a 90th percentile of 0.471; the test period's average 0.255 with a 90th percentile of 0.540. The band is cut to 0.471 when the period needs 0.540.

What makes that worth stating is the direction. The errors are larger in the test period even though it is the calmer market — the 2011–2013 stretch, against a calibration window containing the 2008 crisis. So this is not the obvious story of a quiet sample failing to anticipate a turbulent one. The error distribution moved for its own reasons, and conformal's guarantee is exact under exchangeability, which a volatility series does not offer. The guarantee did not fail through a bug. It failed because its premise did not hold, which is the only way a finite-sample guarantee ever fails.

What It Cannot Do

One number governs how precisely any of this can be stated. TabPFN randomises its internal feature and target transforms, so it is stochastic even though nothing is trained. Across three seeds the spread in RMSE is 0.0010 at the full sample — the same size as the gap separating ridge from lasso in the capstone table — and 0.0070 at 150 rows, several times larger. Every figure above is averaged over three seeds for that reason, and a single-seed ranking of this model against its neighbours would be reporting noise.

Three further limits matter more than the RMSE column. The prior is fixed and opaque, and section 3 is what that costs when the truth is simple. Rows are exchangeable to it: the transformer attends over the training table with no notion of order, so nothing stops it attending to a day in 2003 while forecasting 2013, and it cannot prefer recent evidence — the split here is temporal and the features carry history explicitly, which is what keeps the comparison fair. And it has a size ceiling: the pretraining regime is thousands of rows and roughly a hundred features, beyond which the quadratic attention cost over the context bites.

A practical note the notebook states in its first cell. It pins tabpfn==2.2.1 deliberately. Later releases gate the model weights behind a licence acceptance and an interactive browser login, which a notebook cannot complete; 2.2.1 is the last release whose weights download without authentication. The package also ships a usage-reporting client, disabled here by environment variable.

Notebook

References

• Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T. & Hutter, F. (2025). Accurate predictions on small data with a tabular foundation model. Nature 637, 319–326. — TabPFN v2, the model used here
• Müller, S., Hollmann, N., Arango, S. P., Grabocka, J. & Hutter, F. (2022). Transformers can do Bayesian inference. ICLR. — the prior-data fitted network idea, and the posterior-predictive reading
• Vaswani, A. et al. (2017). Attention is all you need. NIPS 30. — the architecture underneath
• Vovk, V., Gammerman, A. & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer. — conformal prediction and the exchangeability it needs
• Corsi, F. (2009). A simple approximate long-memory model of realized volatility. Journal of Financial Econometrics 7(2), 174–196. — the linear benchmark the prior could not beat on short samples
• Diebold, F. X. & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics 13(3), 253–263. — the test behind every verdict above