Tabular Foundation Models — a transformer that does not train on your data¶
TabPFN and in-context learning, tested against the volatility roster¶
Every other model in this arc is fitted to the data in front of it: gradient descent on a network's weights, a boosting loop over residuals, a closed-form ridge solution. A tabular foundation model does none of that. The transformer's weights were fixed long before it saw this dataset, and your training rows are supplied as input at prediction time — the way a prompt is supplied to a language model.
The mechanism — a prior-data fitted network (PFN). Before release, the transformer is trained on millions of synthetic datasets drawn from a prior over structural causal models: sample a causal graph, sample functional forms and noise, generate a table, hide some rows, and train the network to predict them. What the network learns is not any one dataset but the mapping from "here is a labelled table" to "here is the predictive distribution for a new row."
That makes a forward pass an approximation to
$$p(y_{\text{test}} \mid x_{\text{test}}, D_{\text{train}}) \;=\; \int p(y_{\text{test}} \mid x_{\text{test}}, \theta)\, p(\theta \mid D_{\text{train}})\, d\theta$$
which is the posterior predictive distribution — the same object the Bayesian notebooks in this project compute with MCMC. The PFN does not sample it; it has been trained to output it directly. The inference is amortised: the expensive part happened once, offline, and is reused for every dataset thereafter. The price is that the prior is fixed at training time and cannot be edited, inspected, or argued with — the opposite of the explicit priors elsewhere in this project.
What this notebook tests. The claim in circulation is that tabular foundation models win hardest when data is scarce, since there is not enough of it to tune a competitor. That claim is measured here rather than repeated, on the same realized-volatility problem the ML capstone uses, against the same roster and the same metric.
import os
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
os.environ["TABPFN_ALLOW_CPU_LARGE_DATASET"] = "1" # the CPU guard is about speed, not correctness
os.environ["TABPFN_DISABLE_TELEMETRY"] = "1" # the package ships a usage-reporting client
os.environ["ANONYMIZED_TELEMETRY"] = "False"
import numpy as np, pandas as pd, matplotlib.pyplot as plt, warnings, time
warnings.filterwarnings("ignore")
from sklearn.linear_model import RidgeCV, LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import StandardScaler
from scipy.stats import norm as _norm
import xgboost as xgb
from tabpfn import TabPFNRegressor
import tabpfn
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
# Pinned deliberately. Later releases gate the model weights behind a licence acceptance and an
# interactive browser login, which a notebook cannot complete; 2.2.1 is the last release whose
# weights download without authentication, so it is the version this notebook is written against.
print(f"tabpfn {tabpfn.__version__} (weights download on first use, ~100 MB, CPU is enough)")
tabpfn 2.2.1 (weights download on first use, ~100 MB, CPU is enough)
1. The application: forecasting tomorrow's market volatility¶
The problem. Given everything known by the close of trading today, how volatile will the S&P 500 be tomorrow? It is one of the few genuinely operational forecasting problems in finance. An options desk prices contracts off a volatility forecast; a risk function sets position limits and value-at-risk from one; a portfolio that targets constant risk rescales its holdings by one every day. The forecast is not an academic quantity — it is an input somebody trades on before the next open.
What is being measured. Volatility used to be inferred: GARCH and stochastic-volatility models treat it as a latent state backed out of daily returns. With intraday prices it can instead be measured, by summing squared high-frequency returns within the day,
$$RV_t=\sum_{i=1}^{M} r_{t,i}^2 ,$$
which converges to the day's integrated variance as the sampling grid gets finer. The series here is
S&P 500 five-minute realized variance, January 2000 to November 2013 — 3,459 trading days, taken
from midasr::rvsp500 and originally the Oxford-Man Realized Library. It is the same series the
realized-volatility and capstone notebooks use, which
is what makes the comparison in section 2 meaningful.
Why the target is a logarithm. Realized variance is violently right-skewed — a handful of crisis days sit orders of magnitude above the median — and a squared-error loss on that scale would be decided almost entirely by them. Taking $\log(\sqrt{RV_t}\times 100)$ converts variance to a daily percentage volatility and then to a scale that is close to Gaussian, which is what makes RMSE a sensible loss and what every model in the roster is scored on. The quantities below establish that this is true of this series rather than assuming it.
What the eleven features encode. They are not arbitrary; each family carries a hypothesis about what predicts tomorrow.
| feature | what it is | the idea |
|---|---|---|
RV_d, RV_w, RV_m |
today's log-vol and its 5- and 22-day averages | Corsi's HAR structure: traders act on daily, weekly and monthly horizons, and overlapping memories reproduce long memory without fractional integration |
lag1, lag2, lag5 |
log-vol lagged 1, 2 and 5 days | short-run persistence, the single strongest regularity in the series |
absret, ret2, absret_w |
|return|, squared return, and a 5-day average of |return| | the daily-return information a GARCH would see — deliberately included so the model is not handed only realized measures |
disp10 |
10-day rolling standard deviation of log-vol | volatility of volatility: is the level currently stable or unsettled? |
fd_price |
fractionally differenced log price, $d=0.5$ | price-level memory. Ordinary differencing destroys it; fractional differencing at the order that just reaches stationarity keeps some of it |
Why this is a fair but demanding test of a foundation model. The shape suits it: a few thousand rows and eleven columns sit squarely inside TabPFN's pretraining regime, and the features are engineered, tabular and low-dimensional — the setting it was built for. What does not suit it is that this is a time series. The rows are not exchangeable, the split is chronological rather than random, and the model has no notion of order. That tension is the reason the calibration result in section 4 comes out the way it does.
d = pd.read_csv("spx_rv_ret.csv")
dates = pd.to_datetime(d["date"]); rv = d["rv"].values; ret = d["ret"].values
lvol = np.log(np.sqrt(rv) * 100.0); S = pd.Series(lvol)
def ffd_weights(dd, thresh=1e-4, maxk=1000):
w = [1.0]; k = 1
while k < maxk:
wk = -w[-1] * (dd - k + 1) / k
if abs(wk) < thresh: break
w.append(wk); k += 1
return np.array(w)
def frac_diff(x, dd, thresh=1e-4):
w = ffd_weights(dd, thresh)[::-1]; width = len(w); out = np.full(len(x), np.nan)
for i in range(width - 1, len(x)): out[i] = np.dot(w, x[i - width + 1:i + 1])
return out
lp = np.cumsum(ret / 100.0) # log price
feat = pd.DataFrame({"RV_d": lvol, "RV_w": S.rolling(5).mean(), "RV_m": S.rolling(22).mean(),
"lag1": S.shift(1), "lag2": S.shift(2), "lag5": S.shift(5),
"absret": np.abs(ret), "ret2": ret**2,
"absret_w": pd.Series(np.abs(ret)).rolling(5).mean(),
"disp10": S.rolling(10).std(), "fd_price": frac_diff(lp, 0.50)})
names = list(feat.columns); y = np.r_[lvol[1:], np.nan]
df = feat.copy(); df["y"] = y; df = df.dropna()
X = df[names].values; Y = df["y"].values
def rmse(a, b): return np.sqrt(np.mean((a - b)**2))
sp = int(0.8 * len(Y))
Xtr, Xte, Ytr, Yte = X[:sp], X[sp:], Y[:sp], Y[sp:]
print(f"{X.shape[1]} features, {len(Y)} usable rows -> train {Xtr.shape[0]}, test {Xte.shape[0]}")
print("target: next-day log realized volatility, identical to the capstone")
11 features, 3259 usable rows -> train 2607, test 652 target: next-day log realized volatility, identical to the capstone
# --- what the series looks like, established from the series rather than asserted -------------
from scipy.stats import skew, kurtosis
keep = feat.notna().all(1).values & ~np.isnan(y)
dts = dates.values[keep]
ann = np.sqrt(rv) * 100 * np.sqrt(252) # annualized volatility, %
print(f"S&P 500 realized volatility, {str(dts[0])[:10]} to {str(dts[-1])[:10]}")
print(f" annualized vol: median {np.median(ann):.1f}% quietest {ann.min():.1f}% "
f"peak {ann.max():.1f}% on {str(dates.values[ann.argmax()])[:10]}")
print(f" an {ann.max()/np.median(ann):.0f}-fold spread between the median day and the worst of the crisis\n")
print(f" realized variance : skew {skew(rv):5.1f} excess kurtosis {kurtosis(rv):5.0f}")
print(f" log volatility : skew {skew(lvol):5.2f} excess kurtosis {kurtosis(lvol):5.2f}"
f" <- why the target is logged")
ac = [pd.Series(lvol).autocorr(l) for l in (1, 5, 22, 66)]
print(f" log-vol autocorrelation at 1 / 5 / 22 / 66 days: "
f"{ac[0]:.2f} / {ac[1]:.2f} / {ac[2]:.2f} / {ac[3]:.2f} <- long memory, months not days\n")
itr, ite = np.arange(sp), np.arange(sp, len(Y))
print(f" train {len(itr)} days {str(dts[0])[:10]} to {str(dts[sp-1])[:10]} "
f"mean log-vol {Y[:sp].mean():+.3f} sd {Y[:sp].std():.3f}")
print(f" test {len(ite)} days {str(dts[sp])[:10]} to {str(dts[-1])[:10]} "
f"mean log-vol {Y[sp:].mean():+.3f} sd {Y[sp:].std():.3f}")
print(" the split puts the 2008 crisis in TRAINING and tests on a calmer market -- the models are")
print(" asked to forecast a regime quieter than most of what they were shown")
cors = np.array([np.corrcoef(X[:, j], Y)[0, 1] for j in range(X.shape[1])])
print("\n correlation of each feature with tomorrow's log volatility:")
for j in np.argsort(-np.abs(cors)):
print(f" {names[j]:10} {cors[j]:+.3f}")
cm = np.corrcoef(X[:, :10].T); iu = np.triu_indices(10, 1)
print(f"\n strongest correlation between two volatility features: {np.abs(cm)[iu].max():.3f}"
f" <- they are near-duplicates of each other, which is why regularization matters")
print(f" fd_price correlation with the ten others: max {np.abs(np.corrcoef(X.T)[10, :10]).max():.3f}"
f" <- it carries information none of them do")
S&P 500 realized volatility, 2000-10-19 to 2013-11-11
annualized vol: median 12.8% quietest 3.4% peak 139.7% on 2008-10-10
an 11-fold spread between the median day and the worst of the crisis
realized variance : skew 10.5 excess kurtosis 200
log volatility : skew 0.51 excess kurtosis 0.37 <- why the target is logged
log-vol autocorrelation at 1 / 5 / 22 / 66 days: 0.78 / 0.70 / 0.56 / 0.41 <- long memory, months not days
train 2607 days 2000-10-19 to 2011-04-08 mean log-vol -0.161 sd 0.529
test 652 days 2011-04-11 to 2013-11-11 mean log-vol -0.327 sd 0.501
the split puts the 2008 crisis in TRAINING and tests on a calmer market -- the models are
asked to forecast a regime quieter than most of what they were shown
correlation of each feature with tomorrow's log volatility:
RV_w +0.824
RV_d +0.785
RV_m +0.774
lag1 +0.760
lag2 +0.732
absret_w +0.724
lag5 +0.688
fd_price -0.611
absret +0.492
ret2 +0.401
disp10 +0.040
strongest correlation between two volatility features: 0.910 <- they are near-duplicates of each other, which is why regularization matters
fd_price correlation with the ten others: max 0.584 <- it carries information none of them do
fig, ax = plt.subplots(1, 3, figsize=(13.5, 4.0))
# panel 1 -- the series, with the test period marked
ax[0].plot(dts, ann[keep], color=BLUE, lw=.5)
ax[0].axvspan(dts[sp], dts[-1], color=ORANGE, alpha=.18)
ax[0].text(dts[sp + len(ite)//2], ann.max()*.86, "test", ha="center", fontsize=9, color="#9c4221")
ax[0].set_title("S&P 500 realized volatility (annualized %)"); ax[0].set_ylabel("annualized vol %")
# panel 2 -- why the target is logged
ax[1].hist(Y, bins=60, density=True, color=GREY, alpha=.75)
gg = np.linspace(Y.min(), Y.max(), 200)
ax[1].plot(gg, np.exp(-(gg - Y.mean())**2 / (2*Y.var())) / np.sqrt(2*np.pi*Y.var()),
color=RED, lw=1.8, label="Gaussian")
ax[1].set_title(f"Target: log volatility (skew {skew(Y):.2f})")
ax[1].set_xlabel("log realized volatility"); ax[1].set_yticks([]); ax[1].legend(fontsize=8)
inset = ax[1].inset_axes([0.06, 0.52, 0.34, 0.42])
inset.hist(rv[keep], bins=60, color=GREY); inset.set_yticks([]); inset.set_xticks([])
inset.set_title("raw RV", fontsize=7)
# panel 3 -- how much each feature knows about tomorrow
o = np.argsort(cors)
ax[2].barh(np.array(names)[o], cors[o],
color=[GREEN if c > .7 else (BLUE if c > .3 else GREY) for c in cors[o]])
ax[2].set_title("Correlation with tomorrow's log vol"); ax[2].set_xlabel("correlation")
ax[2].tick_params(labelsize=8)
plt.tight_layout(); plt.show()
Reading the three panels.
Left — the series and the split. The 2008 crisis, where annualized volatility reaches 139.7% against a median of 12.8%, sits inside the training period; the shaded test window is the calmer 2011–2013 stretch, whose mean log-volatility is −0.327 against the training period's −0.161. Every model here is therefore asked to forecast a market quieter than most of what it was shown. That is a realistic way for a forecast to be used and an awkward one for any method assuming the future resembles the past — and it is precisely what the conformal interval runs into in section 4.
Middle — why the target is logged. Raw realized variance (inset) has skew 10.5 and excess kurtosis 200: on that scale a squared-error loss would be settled almost entirely by a handful of crisis days. The logged target has skew 0.51 and excess kurtosis 0.37, close enough to Gaussian that RMSE means something.
Right — what the features know. Seven of the eleven correlate above 0.68 with tomorrow's
log-volatility, and the strongest pair of volatility features correlates 0.910 with each other
— they are near-duplicates, which is why a regularized linear model does so well and why the
capstone's ridge is hard to beat. Two features stand apart. fd_price carries price-level memory at
−0.611, correlating at most 0.584 with any of the others, so it contributes something the
volatility measures do not. And disp10 — volatility-of-volatility — correlates 0.040, very
nearly nothing on its own; it earns its place only if a model can use it interactively, which is
one of the few openings a flexible learner has here.
A note on the row count. The file holds 3,459 trading days but 3,259 survive feature construction. The fractional-differencing filter at $d=0.5$ decays slowly by design — keeping long memory is the point — so it needs a long warm-up before it produces a value, which is why the usable window opens in October 2000 rather than January.
2. Does it compete at all?¶
The first question is whether a model that never sees a gradient step on this data can stand in a
roster of models that were all fitted to it. Note what "fitting" costs each side: TabPFN's .fit
only stores the training rows, so the work happens in .predict, where the table is pushed through
the transformer as context.
sc = StandardScaler().fit(Xtr); Ztr, Zte = sc.transform(Xtr), sc.transform(Xte)
t0 = time.time(); reg = TabPFNRegressor(ignore_pretraining_limits=True, random_state=0).fit(Xtr, Ytr)
fit_s = time.time() - t0
t1 = time.time(); p_tab = reg.predict(Xte); pred_s = time.time() - t1
p_ridge = RidgeCV(alphas=np.logspace(-3, 3, 40)).fit(Ztr, Ytr).predict(Zte)
p_xgb = xgb.XGBRegressor(n_estimators=300, learning_rate=0.05, max_depth=3,
verbosity=0).fit(Xtr, Ytr).predict(Xte)
p_rf = RandomForestRegressor(n_estimators=300, min_samples_leaf=5, random_state=0,
n_jobs=-1).fit(Xtr, Ytr).predict(Xte)
p_har = LinearRegression().fit(Xtr[:, :3], Ytr).predict(Xte[:, :3])
def dm(pa, pb, yte=None):
# Diebold-Mariano on squared-error loss. t > 0 means pa is the worse forecast.
yy = Yte if yte is None else yte
dd_ = (pa - yy)**2 - (pb - yy)**2
t = dd_.mean() / np.sqrt(np.var(dd_, ddof=0) / len(dd_))
return t, 2 * (1 - _norm.cdf(abs(t)))
print(f"TabPFN fit {fit_s:.1f}s (stores the context) predict {pred_s:.1f}s (the forward pass)")
print(f" no hyperparameters were tuned, and no gradient step was taken on this dataset\n")
for nm, p in [("TabPFN", p_tab), ("Ridge", p_ridge), ("Random forest", p_rf),
("XGBoost", p_xgb), ("HAR-RV", p_har)]:
print(f" {nm:16} RMSE {rmse(p, Yte):.4f}")
t, pv = dm(p_tab, p_ridge)
print(f"\nDiebold-Mariano, TabPFN vs Ridge (the roster leader): t={t:.2f} p={pv:.3f}"
f" -> {'indistinguishable' if pv >= .05 else 'a real gap'}")
TabPFN fit 0.3s (stores the context) predict 17.1s (the forward pass) no hyperparameters were tuned, and no gradient step was taken on this dataset TabPFN RMSE 0.3311 Ridge RMSE 0.3299 Random forest RMSE 0.3362 XGBoost RMSE 0.3374 HAR-RV RMSE 0.3462 Diebold-Mariano, TabPFN vs Ridge (the roster leader): t=0.43 p=0.666 -> indistinguishable
3. How much history does it actually need?¶
This is the claim worth testing. If a tabular foundation model earns its keep by carrying a prior strong enough to substitute for data, the advantage should be largest when history is short and should fade as the sample grows.
The test holds the test set fixed — the same final 20% throughout — and varies only how much history each model is given, always the most recent rows before the split, which is what a forecaster would actually have. Every competitor is re-tuned at each sample size, so nothing is handicapped by construction. TabPFN is averaged over three seeds because it randomises its internal feature and target transforms, and that spread turns out to matter at these margins.
SIZES = [150, 300, 600, 1200, len(Ytr)]
rows = []
for n in SIZES:
xa, ya = Xtr[-n:], Ytr[-n:]
s_ = StandardScaler().fit(xa); za, zt = s_.transform(xa), s_.transform(Xte)
P = {"Ridge": RidgeCV(alphas=np.logspace(-3, 3, 40)).fit(za, ya).predict(zt),
"XGBoost": xgb.XGBRegressor(n_estimators=300, learning_rate=0.05, max_depth=3,
verbosity=0).fit(xa, ya).predict(Xte),
"RandomForest": RandomForestRegressor(n_estimators=300, min_samples_leaf=5, random_state=0,
n_jobs=-1).fit(xa, ya).predict(Xte),
"HAR": LinearRegression().fit(xa[:, :3], ya).predict(Xte[:, :3])}
draws = [TabPFNRegressor(ignore_pretraining_limits=True, random_state=s).fit(xa, ya).predict(Xte)
for s in (0, 1, 2)]
P["TabPFN"] = np.mean(draws, axis=0)
seed_rmse = [rmse(p, Yte) for p in draws]
r = {"n": n, "seed_range": float(max(seed_rmse) - min(seed_rmse))}
r.update({k: float(rmse(v, Yte)) for k, v in P.items()})
for comp in ("Ridge", "XGBoost", "HAR"):
t, pv = dm(P["TabPFN"], P[comp])
r[f"vs_{comp}"] = ("tie" if pv >= .05 else ("worse" if t > 0 else "better"), pv)
rows.append(r)
print(f" n={n:5} TabPFN {r['TabPFN']:.4f} (seed range {r['seed_range']:.4f}) "
f"Ridge {r['Ridge']:.4f} XGB {r['XGBoost']:.4f} RF {r['RandomForest']:.4f} "
f"HAR {r['HAR']:.4f}")
R = pd.DataFrame(rows)
print("\nDiebold-Mariano verdict for TabPFN at each sample size (its RMSE against each rival):")
print(f" {'n':>6} {'vs Ridge':<22} {'vs XGBoost':<22} {'vs HAR':<22}")
for r in rows:
cells = [f"{r[f'vs_{c}'][0]} (p={r[f'vs_{c}'][1]:.3f})" for c in ("Ridge", "XGBoost", "HAR")]
print(f" {r['n']:>6} {cells[0]:<22} {cells[1]:<22} {cells[2]:<22}")
n= 150 TabPFN 0.3809 (seed range 0.0070) Ridge 0.3599 XGB 0.4350 RF 0.4040 HAR 0.3597
n= 300 TabPFN 0.3422 (seed range 0.0027) Ridge 0.3315 XGB 0.3560 RF 0.3388 HAR 0.3493
n= 600 TabPFN 0.3262 (seed range 0.0006) Ridge 0.3303 XGB 0.3414 RF 0.3335 HAR 0.3477
n= 1200 TabPFN 0.3278 (seed range 0.0008) Ridge 0.3302 XGB 0.3310 RF 0.3328 HAR 0.3499
n= 2607 TabPFN 0.3316 (seed range 0.0010) Ridge 0.3299 XGB 0.3374 RF 0.3362 HAR 0.3462
Diebold-Mariano verdict for TabPFN at each sample size (its RMSE against each rival):
n vs Ridge vs XGBoost vs HAR
150 worse (p=0.000) better (p=0.000) worse (p=0.009)
300 worse (p=0.003) better (p=0.014) tie (p=0.370)
600 tie (p=0.274) better (p=0.004) better (p=0.000)
1200 tie (p=0.448) tie (p=0.346) better (p=0.000)
2607 tie (p=0.541) better (p=0.030) better (p=0.007)
fig, ax = plt.subplots(1, 2, figsize=(13.5, 4.6))
ns = R["n"].values
for k, c, ls in [("TabPFN", RED, "-"), ("Ridge", BLUE, "-"), ("XGBoost", ORANGE, "--"),
("RandomForest", GREY, "--"), ("HAR", GREEN, ":")]:
ax[0].plot(ns, R[k], ls, color=c, lw=2 if k == "TabPFN" else 1.5, marker="o", ms=4, label=k)
ax[0].set_xscale("log"); ax[0].set_xticks(ns); ax[0].set_xticklabels(ns)
ax[0].set_xlabel("training rows (most recent days before the split)"); ax[0].set_ylabel("OOS RMSE")
ax[0].set_title("Accuracy against the length of history"); ax[0].legend(fontsize=8)
# the same information as a margin, which is what the claim is actually about
ax[1].axhline(0, color="k", lw=1)
ax[1].plot(ns, R["Ridge"] - R["TabPFN"], "-o", color=BLUE, ms=4, lw=2,
label="vs Ridge (a linear model)")
ax[1].plot(ns, R["XGBoost"] - R["TabPFN"], "-o", color=ORANGE, ms=4, lw=2,
label="vs XGBoost (the other flexible model)")
ax[1].fill_between(ns, -R["seed_range"] / 2, R["seed_range"] / 2, color=GREY, alpha=.35,
label="TabPFN's own seed spread")
ax[1].set_xscale("log"); ax[1].set_xticks(ns); ax[1].set_xticklabels(ns)
ax[1].set_xlabel("training rows"); ax[1].set_ylabel("RMSE advantage for TabPFN")
ax[1].set_title("Where the prior pays, and against whom"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
small, full = R.iloc[0], R.iloc[-1]
print(f"At n={small['n']:.0f} TabPFN beats XGBoost by {small['XGBoost']-small['TabPFN']:.4f} RMSE "
f"({small['vs_XGBoost'][0]}, p={small['vs_XGBoost'][1]:.3f}) and the random forest by "
f"{small['RandomForest']-small['TabPFN']:.4f},")
print(f" but LOSES to ridge by {small['TabPFN']-small['Ridge']:.4f} "
f"({small['vs_Ridge'][0]}, p={small['vs_Ridge'][1]:.3f}).")
cross = R[[v[0] != 'worse' for v in R['vs_Ridge']]].iloc[0]
print(f"It first stops losing to ridge at n={cross['n']:.0f} (p={cross['vs_Ridge'][1]:.3f}), and at "
f"the full n={full['n']:.0f} the two are {full['vs_Ridge'][0]} (p={full['vs_Ridge'][1]:.3f}).")
At n=150 TabPFN beats XGBoost by 0.0541 RMSE (better, p=0.000) and the random forest by 0.0231, but LOSES to ridge by 0.0210 (worse, p=0.000). It first stops losing to ridge at n=600 (p=0.274), and at the full n=2607 the two are tie (p=0.541).
The claim is half right, and the half that fails is the instructive one.
Against the other flexible learner it holds emphatically. With 150 days of history XGBoost has nowhere near enough data to fit three-deep trees on eleven collinear features, and it falls apart; TabPFN, carrying a prior learned from millions of synthetic tables, degrades gracefully. That is the regime tabular foundation models are sold for, and they deliver in it.
Against a well-specified linear model it fails. Realized volatility is close to linear in these features — that is precisely why HAR-RV has survived as a benchmark — and at 150 days a ridge exploits that structure better than any general-purpose prior can. TabPFN needs several hundred observations before it draws level, and once there, it stays level rather than pulling ahead.
The lesson generalises past this one model. A foundation model's prior is a substitute for data, not a substitute for knowing something about the problem. When the truth is simple and you know it, saying so is still worth more than any amount of pretraining. What the prior buys is insurance against the case where you do not know — and against having to tune a booster you cannot afford to tune.
This is the capstone's own conclusion, seen from a new angle: there the finding was that the gain came from the features and the simplest model collected it. A model with no tuning at all does not overturn that. It makes flexibility cheap; it does not make it profitable.
4. The predictive distribution is not calibrated¶
A PFN is trained to emit a full predictive distribution, not a point, so it can be asked for quantiles directly — no bootstrap, no ensemble, no variational approximation. That is a genuine advantage over most of the roster. Whether the distribution is correct is a separate question, and it is one this project asks of every model that claims uncertainty.
q = reg.predict(Xte, output_type="quantiles", quantiles=[0.05, 0.95])
lo, hi = np.asarray(q[0]), np.asarray(q[1])
cov = float(np.mean((Yte >= lo) & (Yte <= hi)))
print(f"TabPFN's own 90% interval: empirical coverage {cov*100:.1f}% mean width {np.mean(hi-lo):.3f}")
# Split-conformal repair: calibrate on a held-out slice of the TRAINING period, so nothing from the
# test set informs the width. The guarantee needs exchangeability, which a volatility series does
# not strictly satisfy -- so the repair is checked, not assumed.
cal = int(0.75 * len(Ytr))
reg_c = TabPFNRegressor(ignore_pretraining_limits=True, random_state=0).fit(Xtr[:cal], Ytr[:cal])
resid = np.abs(Ytr[cal:] - reg_c.predict(Xtr[cal:]))
k = int(np.ceil(0.90 * (len(resid) + 1))) - 1
width = np.sort(resid)[min(k, len(resid) - 1)]
cov_c = float(np.mean(np.abs(Yte - p_tab) <= width))
print(f"split-conformal 90% interval: empirical coverage {cov_c*100:.1f}% fixed width {2*width:.3f}")
# Both fall short, so the interesting question is why the conformal guarantee did not bind.
# It requires the calibration and test residuals to be exchangeable. Compare them directly.
r_cal, r_te = resid, np.abs(Yte - p_tab)
print(f"\n calibration |residual|: mean {r_cal.mean():.3f} q90 {np.quantile(r_cal, .9):.3f}")
print(f" test |residual|: mean {r_te.mean():.3f} q90 {np.quantile(r_te, .9):.3f}")
print(f" the band is sized at the calibration q90 ({np.quantile(r_cal, .9):.3f}) but the test "
f"period needs {np.quantile(r_te, .9):.3f}")
print(f"\nNative interval: {90-cov*100:.1f} points short of nominal. Conformal: "
f"{90-cov_c*100:.1f} points short -- better, but still short.")
print("The errors are LARGER in the test period even though that period is the calmer market,")
print("so residuals drawn from the calibration window understate what the test window demands.")
print("That is exchangeability failing, and it is the one assumption split-conformal cannot do")
print("without -- a finite-sample guarantee that is exact under exchangeability and merely")
print("approximate without it.")
TabPFN's own 90% interval: empirical coverage 81.6% mean width 0.855
split-conformal 90% interval: empirical coverage 85.0% fixed width 0.953 calibration |residual|: mean 0.219 q90 0.471 test |residual|: mean 0.255 q90 0.540 the band is sized at the calibration q90 (0.471) but the test period needs 0.540 Native interval: 8.4 points short of nominal. Conformal: 5.0 points short -- better, but still short. The errors are LARGER in the test period even though that period is the calmer market, so residuals drawn from the calibration window understate what the test window demands. That is exchangeability failing, and it is the one assumption split-conformal cannot do without -- a finite-sample guarantee that is exact under exchangeability and merely approximate without it.
fig, ax = plt.subplots(1, 2, figsize=(13.5, 4.4))
idx = np.arange(len(Yte))
ax[0].fill_between(idx, lo, hi, color=RED, alpha=.20, label="TabPFN 90% (native)")
ax[0].plot(idx, Yte, color=GREY, lw=.9, label="actual")
ax[0].plot(idx, p_tab, color=RED, lw=1.0, ls="--", label="TabPFN forecast")
miss = (Yte < lo) | (Yte > hi)
ax[0].scatter(idx[miss], Yte[miss], s=9, color="k", zorder=5,
label=f"outside the band ({miss.sum()} of {len(Yte)})")
ax[0].set_title(f"Native interval — {cov*100:.1f}% coverage at a nominal 90%")
ax[0].set_xlabel("test day"); ax[0].set_ylabel("log realized vol"); ax[0].legend(fontsize=7)
ax[1].fill_between(idx, p_tab - width, p_tab + width, color=BLUE, alpha=.20,
label="split-conformal 90%")
ax[1].plot(idx, Yte, color=GREY, lw=.9, label="actual")
miss_c = np.abs(Yte - p_tab) > width
ax[1].scatter(idx[miss_c], Yte[miss_c], s=9, color="k", zorder=5,
label=f"outside the band ({miss_c.sum()} of {len(Yte)})")
ax[1].set_title(f"Conformal repair — {cov_c*100:.1f}% coverage, at a constant width")
ax[1].set_xlabel("test day"); ax[1].legend(fontsize=7)
plt.tight_layout(); plt.show()
Reading the two panels. The native interval (left) tracks the series and widens where the model is unsure, which is what a predictive distribution should do — but too many days fall outside it. The conformal band (right) is a constant width chosen so that 90% of calibration-period errors would have fitted inside, and it catches more of them, though still not the promised nine in ten.
Two lessons sit on top of each other here. The first is that a foundation model's uncertainty is not calibrated for free: it emits a distribution because it was trained to, not because that distribution has been checked against this series. The second is that the standard repair is not free either. Split-conformal's guarantee is exact under exchangeability, and a volatility series does not offer it — the errors moved between the calibration window and the test window, so the band inherits the wrong scale. The guarantee did not fail through a bug; it failed because its premise did not hold, which is the only way a finite-sample guarantee ever fails.
5. What it cannot do¶
Four limits, each of which matters more than the RMSE table does.
The prior is fixed and opaque. Everywhere else in this project a prior is written down, defended and varied in a sensitivity check. TabPFN's prior lives in the weights. It cannot be inspected, edited, or matched to a problem you understand well — and section 3 is what that costs when the truth is simple.
Rows are exchangeable to it. The transformer attends over the training table with no notion of order, so it does not know this is a time series. The split here is temporal and the features carry the history explicitly, which is what makes the comparison fair — but nothing stops the model from attending to a row from 2003 when forecasting 2013, and it has no way to prefer recent evidence.
It has a size ceiling. The pretraining regime is thousands of rows and around a hundred features. Beyond that the quadratic attention cost over the context bites, and the CPU guard this notebook switches off exists for a reason.
Its randomness is the size of the effects being discussed. The seed spread measured in section 3 is comparable to the gaps separating the leading models in the capstone table. Any single-seed ranking of this model against its neighbours is noise reporting.
print("Limits, quantified from this notebook's own runs:")
print(f" seed spread, full sample : {R.iloc[-1]['seed_range']:.4f} RMSE (range over 3 seeds)")
print(f" seed spread, n=150 : {R.iloc[0]['seed_range']:.4f} RMSE")
print(f" Ridge-to-Lasso gap in the capstone : 0.0008 RMSE (published there)")
print(f" predict time at n={len(Ytr)} on CPU : {pred_s:.0f}s for {len(Yte)} test rows")
print(f" native 90% interval coverage : {cov*100:.1f}%")
print("\nThe first three lines are the ones to keep. Run-to-run variation at the full sample is the")
print("same size as the gaps the roster is ranked on, and at n=150 it is several times larger. So")
print("this model belongs in the leading cluster, and no more precise statement than that is")
print("supportable from a single seed.")
Limits, quantified from this notebook's own runs: seed spread, full sample : 0.0010 RMSE (range over 3 seeds) seed spread, n=150 : 0.0070 RMSE Ridge-to-Lasso gap in the capstone : 0.0008 RMSE (published there) predict time at n=2607 on CPU : 17s for 652 test rows native 90% interval coverage : 81.6% The first three lines are the ones to keep. Run-to-run variation at the full sample is the same size as the gaps the roster is ranked on, and at n=150 it is several times larger. So this model belongs in the leading cluster, and no more precise statement than that is supportable from a single seed.