Neural Networks IV — Transformers & Self-Attention

Attention from scratch → the transformer block, its long-range advantage, and a finance application

The LSTM carried memory through a sequence one step at a time. The transformer (Vaswani et al., 2017, "Attention Is All You Need") throws recurrence out entirely: every position looks directly at every other position in a single, parallel operation called self-attention. There is no distance penalty — position 1 and position 1,000 are one step apart — and no sequential bottleneck, so the whole sequence is processed at once. That combination (unlimited range + parallelism) is why the transformer scaled to the models behind modern AI (GPT, BERT).

The mechanism is scaled dot-product attention. Each position emits a query $q$, a key $k$, and a value $v$. A query is compared to every key by dot product; the scores are softmax-normalised into weights; the output is the weighted sum of values: $$\text{Attention}(Q,K,V)=\text{softmax}\!\Big(\frac{QK^\top}{\sqrt{d}}\Big)V.$$ The softmax weights are the attention — a learned, content-based decision about what each position should read from the rest.

We build this from scratch and validate it against PyTorch, assemble the full transformer block (multi-head attention + positional encoding + feed-forward + residual/layernorm), demonstrate its decisive long-range advantage over the LSTM on a controlled recall task, and apply a small transformer to the realized-volatility series from the previous notebook. Python-only.

1. Scaled dot-product attention — from scratch

The whole operation is three matrix multiplies and a softmax. sdpa_scratch below implements it and returns both the output and the attention-weight matrix — row $i$ shows how position $i$ distributes its attention over all positions. We confirm it matches PyTorch's optimised F.scaled_dot_product_attention to $10^{-7}$, then visualise the weights: every query (row) is a probability distribution over the keys (columns), the model's learned "what should I look at" for each position.

import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"
import numpy as np, matplotlib.pyplot as plt, time, warnings
warnings.filterwarnings("ignore")
import torch, torch.nn as nn, torch.nn.functional as F; torch.set_num_threads(2)
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"; PURP="#6b46c1"
def sdpa_scratch(Q,K,V):
    d=Q.shape[-1]; s=(Q@K.T)/np.sqrt(d)                 # scaled scores
    s=s-s.max(1,keepdims=True); w=np.exp(s); w/=w.sum(1,keepdims=True)   # softmax over keys
    return w@V, w                                        # weighted sum of values, + weights
rng=np.random.default_rng(0)
Q=rng.standard_normal((6,8)).astype(np.float32); K=rng.standard_normal((6,8)).astype(np.float32); V=rng.standard_normal((6,4)).astype(np.float32)
out,W=sdpa_scratch(Q,K,V)
with torch.no_grad(): ref=F.scaled_dot_product_attention(torch.tensor(Q),torch.tensor(K),torch.tensor(V)).numpy()
print("self-attention from scratch vs torch F.scaled_dot_product_attention: max|diff| =", float(np.abs(out-ref).max()))
fig,ax=plt.subplots(figsize=(5,4)); im=ax.imshow(W,cmap="viridis",vmin=0)
ax.set_xlabel("key position (attended TO)"); ax.set_ylabel("query position (attending FROM)"); ax.set_title("Attention weights: each row sums to 1")
plt.colorbar(im,ax=ax,fraction=0.046); plt.tight_layout(); plt.show()
print("Each query row is a softmax over all keys -- a learned, content-based mixture. Stacking these and learning Q,K,V")
print("is the entire attention mechanism; nothing is recurrent, and every position reaches every other in one step.")

self-attention from scratch vs torch F.scaled_dot_product_attention: max|diff| = 9.473282849836728e-08

Each query row is a softmax over all keys -- a learned, content-based mixture. Stacking these and learning Q,K,V
is the entire attention mechanism; nothing is recurrent, and every position reaches every other in one step.

2. The transformer block — multi-head attention, positions, and the rest

Three additions turn bare attention into the transformer used everywhere:

• Multi-head attention. Run several attention operations in parallel, each with its own $Q,K,V$ projections, and concatenate. Different "heads" specialise — one may track local neighbours, another long-range matches — like a CNN's multiple filters.
• Positional encoding. Attention is permutation-invariant — shuffle the inputs and the outputs shuffle identically; it has no innate sense of order. So we add a position signal to each input. The original sinusoidal encoding (below) gives every position a unique, smoothly-varying fingerprint the network can read.
• Residual connections + layer norm + a feed-forward sub-layer. Each block is x → x + attention(x) → x + FFN(x) with layer normalisation, which keeps very deep stacks trainable.

PyTorch packages all of this as nn.TransformerEncoderLayer; we use it below. The heatmap shows the sinusoidal positional encoding — each row is one position's fingerprint.

def positional_encoding(T,d):
    pe=np.zeros((T,d)); pos=np.arange(T)[:,None]; div=10000**(np.arange(0,d,2)/d)
    pe[:,0::2]=np.sin(pos/div); pe[:,1::2]=np.cos(pos/div); return pe
PE=positional_encoding(40,32)
fig,ax=plt.subplots(1,2,figsize=(12,3.8))
im=ax[0].imshow(PE,cmap="RdBu_r",aspect="auto"); ax[0].set_xlabel("encoding dimension"); ax[0].set_ylabel("position in sequence"); ax[0].set_title("Sinusoidal positional encoding (added to inputs)")
plt.colorbar(im,ax=ax[0],fraction=0.046)
ax[1].plot(PE[:,0],label="dim 0 (low freq)"); ax[1].plot(PE[:,4],label="dim 4"); ax[1].plot(PE[:,16],label="dim 16 (high freq)")
ax[1].set_xlabel("position"); ax[1].set_ylabel("value"); ax[1].set_title("Each dimension is a sinusoid of a different frequency"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
demo=nn.TransformerEncoderLayer(d_model=32,nhead=4,dim_feedforward=64,batch_first=True)
print("One transformer encoder block (PyTorch):"); print(demo)
print(f"\nParameters in one block: {sum(p.numel() for p in demo.parameters()):,}  (multi-head attention + a 2-layer FFN)")

One transformer encoder block (PyTorch):
TransformerEncoderLayer(
  (self_attn): MultiheadAttention(
    (out_proj): NonDynamicallyQuantizableLinear(in_features=32, out_features=32, bias=True)
  )
  (linear1): Linear(in_features=32, out_features=64, bias=True)
  (dropout): Dropout(p=0.1, inplace=False)
  (linear2): Linear(in_features=64, out_features=32, bias=True)
  (norm1): LayerNorm((32,), eps=1e-05, elementwise_affine=True, bias=True)
  (norm2): LayerNorm((32,), eps=1e-05, elementwise_affine=True, bias=True)
  (dropout1): Dropout(p=0.1, inplace=False)
  (dropout2): Dropout(p=0.1, inplace=False)
)

Parameters in one block: 8,544  (multi-head attention + a 2-layer FFN)

3. Why attention wins — the long-range recall test

Here is attention's decisive advantage over recurrence, in a controlled experiment. Each input is a sequence of random values; one position is marked, and the task is to output the marked position's value. Solving it requires reaching back to an arbitrary position — trivial for attention (it looks everywhere at once), hard for an LSTM once the marked position is far in the past and its memory has been overwritten.

We train a small transformer and an LSTM of comparable size at sequence lengths 20, 60 and 120, and plot the out-of-sample error against length. The LSTM degrades sharply as the sequence grows; the transformer stays accurate regardless of distance — the property that lets transformers model long documents, genomes, and long price histories.

def make_recall(nseq,T,seed):
    r=np.random.default_rng(seed); val=r.uniform(0,1,(nseq,T)); pos=r.integers(0,T,nseq)
    mark=np.zeros((nseq,T)); mark[np.arange(nseq),pos]=1
    X=np.stack([val,mark],-1); y=val[np.arange(nseq),pos]
    return torch.tensor(X,dtype=torch.float32),torch.tensor(y,dtype=torch.float32).view(-1,1)
class RecallTransformer(nn.Module):
    def __init__(self,d=32,T=60):
        super().__init__(); self.emb=nn.Linear(2,d); self.pos=nn.Parameter(torch.randn(1,T,d)*.02)
        self.enc=nn.TransformerEncoder(nn.TransformerEncoderLayer(d,4,64,batch_first=True,dropout=0),2); self.head=nn.Linear(d,1)
    def forward(self,x): h=self.emb(x)+self.pos[:,:x.shape[1]]; return self.head(self.enc(h).mean(1))
class RecallLSTM(nn.Module):
    def __init__(self,d=32): super().__init__(); self.l=nn.LSTM(2,d,batch_first=True); self.h=nn.Linear(d,1)
    def forward(self,x): o,_=self.l(x); return self.h(o[:,-1])
def fit_recall(model,T):
    Xtr,ytr=make_recall(3000,T,1); Xte,yte=make_recall(1000,T,2)
    opt=torch.optim.Adam(model.parameters(),lr=3e-3); lf=nn.MSELoss()
    for ep in range(30):
        pm=torch.randperm(3000)
        for b in range(0,3000,64): bi=pm[b:b+64]; opt.zero_grad(); lf(model(Xtr[bi]),ytr[bi]).backward(); opt.step()
    with torch.no_grad(): return float(((model(Xte)-yte)**2).mean()**.5)
Ts=[20,60,120]; tf=[]; ls=[]; tfs_sd=[]; ls_sd=[]; t=time.time()
for T in Ts:                                   # 3 seeds each: one run of this is noisy enough to mislead
    a=[];b=[]
    for sd_ in range(3):
        torch.manual_seed(sd_); a.append(fit_recall(RecallTransformer(32,T),T))
        torch.manual_seed(sd_); b.append(fit_recall(RecallLSTM(32),T))
    tf.append(np.mean(a)); ls.append(np.mean(b)); tfs_sd.append(np.std(a)); ls_sd.append(np.std(b))
    print(f"T={T:3d}: transformer RMSE {tf[-1]:.4f} (sd {tfs_sd[-1]:.4f}) | LSTM RMSE {ls[-1]:.4f} (sd {ls_sd[-1]:.4f})"
          f"  -> {ls[-1]/tf[-1]:.1f}x")
fig,ax=plt.subplots(figsize=(7,4.2))
ax.plot(Ts,tf,"o-",color=BLUE,lw=2,label="transformer"); ax.plot(Ts,ls,"o-",color=RED,lw=2,label="LSTM")
ax.set_xlabel("sequence length"); ax.set_ylabel("recall RMSE (lower=better)"); ax.set_title("Long-range recall: attention holds, recurrence forgets"); ax.legend()
plt.tight_layout(); plt.show()
print(f"({time.time()-t:.0f}s) The transformer's error is essentially flat in sequence length while the LSTM's grows by")
print(f"an order of magnitude: the gap runs {ls[0]/tf[0]:.0f}x at T=20, {ls[1]/tf[1]:.0f}x at T=60 and {ls[2]/tf[2]:.0f}x at T=120. Attention reaches any")
print("position in one step; the LSTM must carry the marked value through 120 sequential updates and loses it. Averaging")
print("three seeds matters here -- single runs of this task are noisy enough to reverse the ordering at short lengths.")

T= 20: transformer RMSE 0.0045 (sd 0.0009) | LSTM RMSE 0.0138 (sd 0.0034)  -> 3.1x

T= 60: transformer RMSE 0.0123 (sd 0.0033) | LSTM RMSE 0.1470 (sd 0.1297)  -> 12.0x

T=120: transformer RMSE 0.0064 (sd 0.0015) | LSTM RMSE 0.2882 (sd 0.0005)  -> 45.0x

(264s) The transformer's error is essentially flat in sequence length while the LSTM's grows by
an order of magnitude: the gap runs 3x at T=20, 12x at T=60 and 45x at T=120. Attention reaches any
position in one step; the LSTM must carry the marked value through 120 sequential updates and loses it. Averaging
three seeds matters here -- single runs of this task are noisy enough to reverse the ordering at short lengths.

4. A finance application — transformer on realized volatility

We add a small transformer encoder to the volatility horse race from the previous notebook: same S&P realized-volatility data, same 22-day input window, same next-day log-RV target and test period, so its out-of-sample RMSE drops straight onto the scoreboard beside HAR-RV, GARCH, AR(1) and the random walk. The classical benchmarks are refitted here rather than copied across, so the comparison cannot silently go stale.

How this relates to the volatility work elsewhere in the collection. Realized Volatility runs the same family — HAR, GARCH-t, Realized GARCH, stochastic volatility — on this identical series, and its numbers are deliberately not comparable with these. Three things differ. It evaluates on the variance scale using QLIKE (Patton's robust loss for variance forecasts), where this notebook uses RMSE on log-volatility. It splits 60/40, so its test period includes the 2008 crisis, while the 80/20 split here tests on the calmer 2011–2013 stretch. And it fits GARCH by its own maximum-likelihood code rather than the arch package. It also carries two models absent here — Realized GARCH and stochastic volatility — of which Realized GARCH edges HAR on QLIKE. So “the network beats GARCH” here means beating the weakest member of that family, on an easier test period, under a different loss.

import pandas as pd
d=pd.read_csv("spx_rv_ret.csv"); rv=d["rv"].values; ret=d["ret"].values; n=len(rv); lvol=np.log(np.sqrt(rv)*100.0)
# Window ends at t-1, target is t: a genuine ONE-step-ahead forecast on the same information set the
# classical benchmarks get. (Targeting lvol[t+1] here would skip day t -- correlation 0.78 with the target
# -- making this a two-step-ahead problem and costing ~0.02 RMSE against models that are not handicapped.)
L=22; Xs=np.array([lvol[t-L:t] for t in range(L,n)]); ys=np.array([lvol[t] for t in range(L,n)])
sp=int(0.8*len(ys)); mu,sd=Xs[:sp].mean(),Xs[:sp].std()
Xt=torch.tensor(((Xs-mu)/sd).reshape(-1,L,1),dtype=torch.float32); yt=torch.tensor((ys-mu)/sd,dtype=torch.float32).view(-1,1)
class VolTransformer(nn.Module):
    def __init__(self,d=32,T=22):
        super().__init__(); self.emb=nn.Linear(1,d); self.pos=nn.Parameter(torch.randn(1,T,d)*.02)
        self.enc=nn.TransformerEncoder(nn.TransformerEncoderLayer(d,4,64,batch_first=True,dropout=0.1),2); self.head=nn.Linear(d,1)
    def forward(self,x): h=self.emb(x)+self.pos[:,:x.shape[1]]; return self.head(self.enc(h).mean(1))
torch.manual_seed(0); vt=VolTransformer(); opt=torch.optim.Adam(vt.parameters(),lr=3e-3); lf=nn.MSELoss()
for ep in range(40):
    pm=torch.randperm(sp)
    for b in range(0,sp,64): bi=pm[b:b+64]; opt.zero_grad(); lf(vt(Xt[bi]),yt[bi]).backward(); opt.step()
with torch.no_grad(): tf_pred=vt(Xt[sp:]).numpy().ravel()*sd+mu
def rmse(a,b): return float(np.sqrt(np.mean((a-b)**2)))
# Benchmarks refitted here rather than copied across notebooks, so the scoreboard cannot go stale.
from numpy.linalg import lstsq
from arch import arch_model
_wk=pd.Series(lvol).rolling(5).mean().values; _mo=pd.Series(lvol).rolling(22).mean().values
_Xh=np.column_stack([lvol,_wk,_mo]); _yh=np.r_[lvol[1:],np.nan]
_ok=~np.isnan(_Xh).any(1)&~np.isnan(_yh); _Xh,_yhv=_Xh[_ok],_yh[_ok]; _s=int(0.8*len(_yhv))
_rw=rmse(_Xh[_s:,0],_yhv[_s:])
_ba=lstsq(np.column_stack([np.ones(_s),_Xh[:_s,0]]),_yhv[:_s],rcond=None)[0]
_ar=rmse(_ba[0]+_ba[1]*_Xh[_s:,0],_yhv[_s:])
_bh=lstsq(np.column_stack([np.ones(_s),_Xh[:_s]]),_yhv[:_s],rcond=None)[0]
har_pred=np.column_stack([np.ones(len(_Xh)-_s),_Xh[_s:]])@_bh; _har=rmse(har_pred,_yhv[_s:])
_split=int(0.8*n); _gr=arch_model(ret,mean="Constant",vol="GARCH",p=1,q=1,dist="t").fit(last_obs=_split,disp="off")
_fc=_gr.forecast(horizon=1,start=_split,reindex=False); _gv=np.log(np.sqrt(_fc.variance.values[:,0]))
_tg=lvol[_split+1:_split+1+len(_gv)]; _m=min(len(_gv),len(_tg)); _garch=rmse(_gv[:_m],_tg[:_m])
_act=ys[sp:]
assert len(har_pred)==len(tf_pred)==len(_act) and np.allclose(_act,_yhv[_s:]), "test sets must align"
board={"HAR-RV":_har,"transformer":rmse(tf_pred,_act),"AR(1)":_ar,"random walk":_rw,"GARCH(1,1)-t":_garch}
tab=pd.Series(board).sort_values()
print("OOS RMSE on log realized volatility (lower = better):"); print(tab.round(4).to_string())
fig,ax=plt.subplots(1,3,figsize=(16.5,4.2))
cols=[PURP if k=="transformer" else (GREEN if k=="HAR-RV" else GREY) for k in tab.index]
ax[0].barh(tab.index,tab.values,color=cols); ax[0].invert_yaxis(); ax[0].set_xlabel("OOS RMSE"); ax[0].set_title("Volatility horse race (transformer added)")
for i,v in enumerate(tab.values): ax[0].text(v+0.002,i,f"{v:.3f}",va="center",fontsize=8)
td=pd.to_datetime(d["date"]).values[L:][sp:]      # ys[i] is lvol at index L+i
assert len(td)==len(tf_pred)
ax[1].plot(td,np.exp(ys[sp:]),color="black",lw=1,label="actual RV"); ax[1].plot(td,np.exp(tf_pred),color=PURP,lw=1,label="transformer")
ax[1].set_ylabel("realized volatility (%, daily)"); ax[1].set_title("Transformer vol forecast vs actual (2011-2013)"); ax[1].legend()
# proportions vs predictions, the diagnostic the rest of the collection uses
_lm=[min(tf_pred.min(),_act.min()),max(tf_pred.max(),_act.max())]
for _p,_c,_n in [(tf_pred,PURP,"transformer"),(har_pred,GREEN,"HAR-RV")]:
    _q=np.quantile(_p,np.linspace(0,1,11)); _b=np.clip(np.digitize(_p,_q[1:-1]),0,9)
    ax[2].plot([_p[_b==k].mean() for k in range(10)],[_act[_b==k].mean() for k in range(10)],"o-",color=_c,lw=2,label=_n)
ax[2].plot(_lm,_lm,"k--",lw=1,label="perfect"); ax[2].set_xlabel("predicted log-vol"); ax[2].set_ylabel("actual log-vol")
ax[2].set_title("Proportions vs predictions"); ax[2].legend(fontsize=8)
plt.tight_layout(); plt.show()
_q=np.quantile(tf_pred,np.linspace(0,1,11)); _b=np.clip(np.digitize(tf_pred,_q[1:-1]),0,9)
_g=[_act[_b==k].mean()-tf_pred[_b==k].mean() for k in range(10)]
_qh=np.quantile(har_pred,np.linspace(0,1,11)); _bh=np.clip(np.digitize(har_pred,_qh[1:-1]),0,9)
_gh=[_act[_bh==k].mean()-har_pred[_bh==k].mean() for k in range(10)]
_sl=np.polyfit(tf_pred,_act,1)[0]; _slh=np.polyfit(har_pred,_act,1)[0]
print(f"\nThe third panel says something the RMSE column cannot. Transformer decile gaps run {min(_g):+.3f} to {max(_g):+.3f}")
print(f"against HAR's {min(_gh):+.3f} to {max(_gh):+.3f}, and the slope of actual on predicted is {_sl:.3f} against HAR's {_slh:.3f}")
print("(the LSTM in the previous notebook manages 0.954).")
print(f"\n  {'decile':>7} {'predicted':>10} {'actual':>9} {'gap':>8}")
for k in range(10):
    print(f"  {k+1:>7} {tf_pred[_b==k].mean():>10.3f} {_act[_b==k].mean():>9.3f} {_g[k]:>+8.3f}")
print("\nThis is a Mincer-Zarnowitz regression, and the benchmark is 1.0: an optimal forecast satisfies")
print("cov(actual, forecast) = var(forecast), which makes the slope exactly 1 no matter how much the forecast")
print("shrinks. (A good forecast SHOULD have smaller spread than the outcome -- it is a conditional mean -- so")
print("comparing standard deviations proves nothing. The slope is the test.)")
print("\nA slope below 1 therefore means the forecast varies MORE than its own information content justifies: it")
print("over-reacts, running too low where it predicts low and too high where it predicts high. The decile gaps")
print("above show which end carries it. All three models here sit below 1, so all three over-react a little, but")
print(f"the transformer at {_sl:.3f} is furthest from the benchmark -- HAR {_slh:.3f}, the LSTM 0.954.")
print("\nSo the transformer does not merely lose to HAR by a small margin of random error; it loses in a systematic")
print("direction, exaggerating swings the data does not support. For a risk application that is the more useful")
print("diagnostic of the two: RMSE says the forecast is slightly worse, the slope says HOW it is worse, and only")
print("the second tells you whether the error shows up as false alarms or missed ones.")
def _nw(x,lag=5):
    x=x-x.mean(); v=np.mean(x*x)
    for k in range(1,lag+1): v+=2*(1-k/(lag+1))*np.mean(x[k:]*x[:-k])
    return np.sqrt(v/len(x))
_dl=(tf_pred-_act)**2-(har_pred-_act)**2; _dm=_dl.mean()/_nw(_dl)
print(f"\nTransformer OOS RMSE {board['transformer']:.4f} against HAR-RV's {_har:.4f}, on the same {len(_act)} test days:")
print(f"Diebold-Mariano = {_dm:+.2f} against a 1.96 critical value. That sits right on the boundary, and it moves to")
print("either side of it depending on the training seed, so the honest reading is that the transformer is at best level")
print("with a four-coefficient regression and quite possibly a little behind -- weaker than the LSTM's clean tie in the")
print("previous notebook (DM -0.35). Either way it does not win, and no amount of attention changes that here.")
print(f"\nOne footnote on the GARCH row ({board['GARCH(1,1)-t']:.4f}): that is the RAW figure, and the previous notebook shows")
print("about a third of it is a constant level offset -- realized volatility omits the overnight move, GARCH does not.")
print("Corrected with an offset estimated on training data alone it scores ~0.378 and moves ahead of the random walk.")
print("It is quoted raw here only to keep the scoreboard on one consistent footing.")
print("\nThat is not a failure of attention; it is attention being asked for something this problem does not contain.")
print("Section 3 showed its advantage is long-range recall. A 22-day window of one autocorrelated series has no")
print("long-range structure to recover -- HAR's daily/weekly/monthly terms already span it -- so all-to-all mixing buys")
print("nothing and costs parameters. Attention's edge is latent until the data is long, broad and nonlinear enough.")

OOS RMSE on log realized volatility (lower = better):
HAR-RV          0.3473
transformer     0.3531
AR(1)           0.3916
random walk     0.4206
GARCH(1,1)-t    0.4568

The third panel says something the RMSE column cannot. Transformer decile gaps run -0.185 to +0.066
against HAR's -0.100 to +0.060, and the slope of actual on predicted is 0.859 against HAR's 0.929
(the LSTM in the previous notebook manages 0.954).

   decile  predicted    actual      gap
        1     -0.801    -0.773   +0.029
        2     -0.697    -0.701   -0.004
        3     -0.611    -0.546   +0.066
        4     -0.523    -0.474   +0.049
        5     -0.423    -0.431   -0.008
        6     -0.332    -0.375   -0.044
        7     -0.225    -0.214   +0.010
        8     -0.105    -0.290   -0.185
        9      0.161     0.047   -0.114
       10      0.615     0.481   -0.134

This is a Mincer-Zarnowitz regression, and the benchmark is 1.0: an optimal forecast satisfies
cov(actual, forecast) = var(forecast), which makes the slope exactly 1 no matter how much the forecast
shrinks. (A good forecast SHOULD have smaller spread than the outcome -- it is a conditional mean -- so
comparing standard deviations proves nothing. The slope is the test.)

A slope below 1 therefore means the forecast varies MORE than its own information content justifies: it
over-reacts, running too low where it predicts low and too high where it predicts high. The decile gaps
above show which end carries it. All three models here sit below 1, so all three over-react a little, but
the transformer at 0.859 is furthest from the benchmark -- HAR 0.929, the LSTM 0.954.

So the transformer does not merely lose to HAR by a small margin of random error; it loses in a systematic
direction, exaggerating swings the data does not support. For a risk application that is the more useful
diagnostic of the two: RMSE says the forecast is slightly worse, the slope says HOW it is worse, and only
the second tells you whether the error shows up as false alarms or missed ones.

Transformer OOS RMSE 0.3531 against HAR-RV's 0.3473, on the same 688 test days:
Diebold-Mariano = +1.98 against a 1.96 critical value. That sits right on the boundary, and it moves to
either side of it depending on the training seed, so the honest reading is that the transformer is at best level
with a four-coefficient regression and quite possibly a little behind -- weaker than the LSTM's clean tie in the
previous notebook (DM -0.35). Either way it does not win, and no amount of attention changes that here.

One footnote on the GARCH row (0.4568): that is the RAW figure, and the previous notebook shows
about a third of it is a constant level offset -- realized volatility omits the overnight move, GARCH does not.
Corrected with an offset estimated on training data alone it scores ~0.378 and moves ahead of the random walk.
It is quoted raw here only to keep the scoreboard on one consistent footing.

That is not a failure of attention; it is attention being asked for something this problem does not contain.
Section 3 showed its advantage is long-range recall. A 22-day window of one autocorrelated series has no
long-range structure to recover -- HAR's daily/weekly/monthly terms already span it -- so all-to-all mixing buys
nothing and costs parameters. Attention's edge is latent until the data is long, broad and nonlinear enough.

# Two figures the write-up quotes that the horse race above does not print.
# (1) GARCH's error is mostly a LEVEL error: it forecasts conditional vol on a different
#     scale, so removing a constant from its forecasts is a fair thing to report alongside
#     the raw number -- an RMSE with the mean error taken out is just the error's sd.
_ge = _gv[:_m] - _tg[:_m]
_garch_debiased = float(np.std(_ge))
print(f"GARCH(1,1)-t: raw RMSE {_garch:.4f}, mean error {_ge.mean():+.4f}, "
      f"RMSE after removing that constant {_garch_debiased:.4f}")
print("Most of GARCH's gap to the others is a level shift rather than bad dynamics -- worth")
print("saying, because the raw number alone reads as a much heavier defeat than it is.\n")

# (2) A forecast SHOULD have smaller spread than the outcome -- it is a conditional mean --
#     so comparing standard deviations proves nothing about shrinkage. State them anyway,
#     because the Mincer-Zarnowitz slope below is the test that actually does the work.
for _nm, _pv in (("HAR-RV", har_pred), ("transformer", tf_pred)):
    print(f"{_nm:12s} predictions sd {np.std(_pv):.3f}   outcome sd {np.std(_act):.3f}   "
          f"ratio {np.std(_pv)/np.std(_act):.2f}")
print("Both forecasts are less variable than what they predict, exactly as a conditional mean")
print("should be. That is why the slope of actual on predicted, not the spread, is the diagnostic.")

GARCH(1,1)-t: raw RMSE 0.4568, mean error +0.2589, RMSE after removing that constant 0.3764
Most of GARCH's gap to the others is a level shift rather than bad dynamics -- worth
saying, because the raw number alone reads as a much heavier defeat than it is.

HAR-RV       predictions sd 0.384   outcome sd 0.497   ratio 0.77
transformer  predictions sd 0.415   outcome sd 0.497   ratio 0.83
Both forecasts are less variable than what they predict, exactly as a conditional mean
should be. That is why the slope of actual on predicted, not the spread, is the diagnostic.

5. Summary

A transformer replaces recurrence with self-attention: every position attends directly to every other, in parallel, with no distance penalty. We built scaled dot-product attention from scratch (matched to PyTorch at $10^{-7}$), assembled the block (multi-head attention + positional encoding + residual/FFN), and saw the payoff and the limit:

• The payoff — on the long-range recall task the transformer stayed accurate as the sequence grew to 120 while the LSTM's error exploded. Reaching any position in one step is a categorically better inductive bias for long-range dependence than carrying state through hundreds of sequential updates. This is why transformers, not RNNs, power modern language models.
• The limit (honest, as ever) — on the short univariate volatility series the transformer did not beat HAR-RV, and a Diebold-Mariano test puts the gap right on the 1.96 boundary: at best level with a four-coefficient regression, quite possibly a little behind, and weaker than the LSTM's clean tie. Attention's advantage is latent until the data is long, broad and nonlinear enough to reward it; on a 22-day window of one autocorrelated series there is no long-range structure left for it to find that HAR's daily/weekly/monthly terms have not already spanned.

The recurring theme of the subsection holds once more: match the architecture to the structure of the data. Attention's structure — content-based, all-to-all mixing — is transformative for language and long sequences, unnecessary for a short, strongly-autocorrelated financial series.

Next — the capstone. Every net so far returns a point prediction. The final notebook adds calibrated uncertainty: MC-dropout (dropout at test time ≈ approximate Bayesian inference, Gal & Ghahramani 2016) and deep ensembles, turning any of these architectures — MLP, CNN, LSTM, transformer — into a model that reports how sure it is. That closes the subsection and ties neural networks back to the portfolio's Bayesian core (BART, Gaussian processes), where uncertainty was the whole point.

Next: Bayesian deep learning — uncertainty in neural networks.