Gradient Boosting — correcting errors in sequence

AdaBoost (Freund & Schapire, 1997) and Gradient Boosting (Friedman, 2001)

The random forest cut variance by averaging many independent deep trees. Boosting attacks the other half of the error — bias — from the opposite direction: it adds many small, dependent trees in sequence, each fit to the mistakes the running ensemble still makes. A single shallow tree underfits (high bias, low variance); stack hundreds of them, each nudging the fit toward the residual errors, and the bias melts away.

Two seminal algorithms, both built here from scratch (gboost.py) and checked against scikit-learn:

• AdaBoost (Freund & Schapire) — the ancestor. After each weak learner it re-weights the training points, up-weighting those still misclassified, then combines the weak learners by a weighted vote.
• Gradient Boosting (Friedman) — the generalisation. View the ensemble $F(x)$ as a function optimised by gradient descent in function space: each round fits a tree to the negative gradient of the loss (the "pseudo-residuals"), then steps $F \leftarrow F + \nu\,\text{tree}$, with learning rate $\nu$. For squared loss the pseudo-residual is the ordinary residual $y-F$; for the binary log-loss it is $y-\sigma(F)$.

The from-scratch core is the boosting loop (the trees themselves were built from scratch in the CART notebook, so we reuse shallow regression trees as base learners). Same Taiwan credit-default and California-housing data as the previous notebooks; all metrics out of sample on a 30% test set. ROC-AUC is defined in the CART notebook (0.5 = chance, 1 = perfect).

import numpy as np, pandas as pd, matplotlib.pyplot as plt, time
from sklearn.model_selection import train_test_split
from sklearn.ensemble import (GradientBoostingClassifier, GradientBoostingRegressor,
                              AdaBoostClassifier, RandomForestClassifier)
from sklearn.metrics import roc_auc_score, mean_squared_error
import gboost
BLUE="#2b6cb0"; RED="#c53030"; GREEN="#2f855a"; ORANGE="#dd6b20"; GREY="#a0aec0"
d=pd.read_csv("credit_default.csv"); feat=[c for c in d.columns if c!="default"]
X=d[feat].to_numpy(float); y=d["default"].to_numpy(int)
Xtr,Xte,ytr,yte=train_test_split(X,y,test_size=0.3,random_state=0,stratify=y)
def sig(z): return 1/(1+np.exp(-np.clip(z,-30,30)))
print(f"credit default: train {len(ytr):,}, test {len(yte):,}; default rate {y.mean():.1%}")

credit default: train 21,000, test 9,000; default rate 22.1%

1. AdaBoost from scratch — re-weighting the hard cases

AdaBoost fits a decision stump (a depth-1 tree), sees which points it gets wrong, up-weights those, and refits — repeating so that later stumps specialise on the stubborn cases. The final prediction is a weighted vote, weights $\alpha_m=\tfrac12\log\frac{1-\text{err}_m}{\text{err}_m}$. Watch the test AUC climb as stumps accumulate, then plateau.

ada=gboost.AdaBoost(n_estimators=300,random_state=0).fit(Xtr,ytr)
ska=AdaBoostClassifier(n_estimators=300,random_state=0).fit(Xtr,ytr)
stg=[roc_auc_score(yte,sig(2*F)) for F in ada.staged_decision_function(Xte)]
print(f"AdaBoost (300 stumps): from-scratch test AUC {stg[-1]:.4f}  |  scikit-learn {roc_auc_score(yte,ska.predict_proba(Xte)[:,1]):.4f}")
fig,ax=plt.subplots(figsize=(7.5,4))
ax.plot(range(1,301),stg,color=BLUE); ax.axhline(stg[-1],color=GREY,ls=":")
ax.set_xlabel("number of stumps"); ax.set_ylabel("test AUC"); ax.set_title("AdaBoost: weak stumps compound into a strong classifier")
plt.tight_layout(); plt.show()
print("Each depth-1 stump is barely better than chance, but re-weighting the errors and voting them together lifts AUC to ~0.76.")

AdaBoost (300 stumps): from-scratch test AUC 0.7655  |  scikit-learn 0.7655

Each depth-1 stump is barely better than chance, but re-weighting the errors and voting them together lifts AUC to ~0.76.

2. Gradient boosting from scratch — descending the loss

Friedman's insight: boosting is gradient descent in function space. At each round we fit a regression tree to the negative gradient of the loss and take a shrunken step. We build it for both the log-loss (classification) and squared loss (regression), and confirm the training loss falls monotonically as trees are added.

gb=gboost.GradientBoosting(n_estimators=300,learning_rate=0.1,max_depth=3,loss="log",random_state=0).fit(Xtr,ytr)
te_auc=[roc_auc_score(yte,sig(F)) for F in gb.staged_decision_function(Xte)]
fig,ax=plt.subplots(1,2,figsize=(13,4))
ax[0].plot(gb.train_loss_,color=RED); ax[0].set_xlabel("trees added"); ax[0].set_ylabel("training log-loss"); ax[0].set_title("Training loss descends with each tree")
ax[1].plot(range(1,301),te_auc,color=BLUE); ax[1].set_xlabel("trees added"); ax[1].set_ylabel("test AUC"); ax[1].set_title("Test AUC climbs, then flattens")
plt.tight_layout(); plt.show()
print(f"from-scratch gradient boosting: test AUC {te_auc[-1]:.4f}  (competitive with the random forest ~0.775 and BART ~0.762)")
print("The additive, gradient-driven loop reduces BIAS -- the shallow trees individually underfit, together they do not.")

from-scratch gradient boosting: test AUC 0.7719  (competitive with the random forest ~0.775 and BART ~0.762)
The additive, gradient-driven loop reduces BIAS -- the shallow trees individually underfit, together they do not.

3. From-scratch vs scikit-learn

The boosting loop is precise, so results should track GradientBoostingClassifier/Regressor closely. Both implement Friedman's per-leaf line search for the log-loss — a Newton step in each terminal region, $\gamma_L=\sum_L (y_i-p_i)\,/\,\sum_L p_i(1-p_i)$, rather than the raw tree output — which is why they track each other to four decimals. The cell below also switches that step off, so what it buys is measured rather than asserted.

rows=[]
for M in [50,150,300]:
    kw=dict(n_estimators=M,learning_rate=0.1,max_depth=3,loss="log",random_state=0)
    g0=gboost.GradientBoosting(**kw,newton=False).fit(Xtr,ytr)   # raw tree output as the step
    g1=gboost.GradientBoosting(**kw,newton=True ).fit(Xtr,ytr)   # Friedman's per-leaf Newton step
    s =GradientBoostingClassifier(n_estimators=M,learning_rate=0.1,max_depth=3,random_state=0).fit(Xtr,ytr)
    rows.append([M,roc_auc_score(yte,g0.predict_proba(Xte)[:,1]),
                   roc_auc_score(yte,g1.predict_proba(Xte)[:,1]),
                   roc_auc_score(yte,s.predict_proba(Xte)[:,1])])
tab=pd.DataFrame(rows,columns=["n_trees","no per-leaf step","with per-leaf step","scikit-learn"])
print(tab.round(4).to_string(index=False))
d50=tab.loc[0,"with per-leaf step"]-tab.loc[0,"no per-leaf step"]
print(f"\nThe per-leaf line search is worth {d50:.4f} AUC at 50 trees. Without it every update")
print("understates the optimal constant in each region, so the ensemble still converges -- it just")
print("climbs more slowly: by 300 trees the plain-gradient version has reached only what the")
print(f"corrected one reached at 50. With it, the from-scratch loop matches scikit-learn to four")
print("decimals, which is the check that the implementation is right rather than merely close.")
h=pd.read_csv("cali_housing.csv"); hf=[c for c in h.columns if c!="MedHouseVal"]
Xh=h[hf].to_numpy(float); yh=h["MedHouseVal"].to_numpy(float)
Xhtr,Xhte,yhtr,yhte=train_test_split(Xh,yh,test_size=0.3,random_state=0)
gr=gboost.GradientBoosting(n_estimators=300,learning_rate=0.1,max_depth=3,loss="ls",random_state=0).fit(Xhtr,yhtr)
sr=GradientBoostingRegressor(n_estimators=300,learning_rate=0.1,max_depth=3,random_state=0).fit(Xhtr,yhtr)
print(f"\nregression RMSE: from-scratch {mean_squared_error(yhte,gr.predict(Xhte))**.5:.4f} | sklearn {mean_squared_error(yhte,sr.predict(Xhte))**.5:.4f}")

 n_trees  no per-leaf step  with per-leaf step  scikit-learn
      50            0.7558              0.7703        0.7702
     150            0.7652              0.7721        0.7721
     300            0.7703              0.7719        0.7719

The per-leaf line search is worth 0.0144 AUC at 50 trees. Without it every update
understates the optimal constant in each region, so the ensemble still converges -- it just
climbs more slowly: by 300 trees the plain-gradient version has reached only what the
corrected one reached at 50. With it, the from-scratch loop matches scikit-learn to four
decimals, which is the check that the implementation is right rather than merely close.

regression RMSE: from-scratch 0.4933 | sklearn 0.4933

4. The two knobs — learning rate and number of trees

Boosting's central trade-off: a small learning rate $\nu$ makes each step timid, so you need more trees, but the result generalises better ("slow learning"). A large $\nu$ with many trees eventually overfits — test error turns back up. We sweep learning rates, tracking test AUC as trees accumulate (scikit-learn's staged_predict_proba).

fig,ax=plt.subplots(figsize=(8,4.6))
for lr,col in zip([0.02,0.1,0.5,1.0],[GREEN,BLUE,ORANGE,RED]):
    s=GradientBoostingClassifier(n_estimators=400,learning_rate=lr,max_depth=3,random_state=0).fit(Xtr,ytr)
    auc=[roc_auc_score(yte,p[:,1]) for p in s.staged_predict_proba(Xte)]
    ax.plot(range(1,401),auc,color=col,label=f"lr={lr}")
ax.set_xlabel("number of trees"); ax.set_ylabel("test AUC"); ax.set_title("Small learning rate + many trees generalises best; large lr overfits")
ax.legend(frameon=False); plt.tight_layout(); plt.show()
print("Low lr (0.02) rises slowly but to the best, most stable AUC; lr=1.0 shoots up then DECLINES as it overfits -- the reason")
print("production practice is small lr + early stopping. Bias falls with trees; past the sweet spot, variance creeps back.")

Low lr (0.02) rises slowly but to the best, most stable AUC; lr=1.0 shoots up then DECLINES as it overfits -- the reason
production practice is small lr + early stopping. Bias falls with trees; past the sweet spot, variance creeps back.

5. Boosting vs the random forest — bias vs variance, side by side

The clearest way to see the difference: add trees to each and watch the test AUC. The random forest climbs and plateaus — more trees only reduce variance, never hurt. Boosting climbs faster (bias falling), peaks, then turns down as it starts to overfit. Averaging-independent vs adding-dependent, drawn in one picture.

rf=RandomForestClassifier(n_estimators=400,max_features="sqrt",min_samples_leaf=5,random_state=0,n_jobs=-1,warm_start=True)
rf_auc=[]
for nt in range(10,401,10):
    rf.set_params(n_estimators=nt).fit(Xtr,ytr); rf_auc.append(roc_auc_score(yte,rf.predict_proba(Xte)[:,1]))
gbc=GradientBoostingClassifier(n_estimators=400,learning_rate=0.1,max_depth=3,random_state=0).fit(Xtr,ytr)
gb_auc=[roc_auc_score(yte,p[:,1]) for p in gbc.staged_predict_proba(Xte)]
fig,ax=plt.subplots(figsize=(8,4.6))
ax.plot(range(10,401,10),rf_auc,color=GREEN,lw=2,label="random forest (plateaus)")
ax.plot(range(1,401),gb_auc,color=BLUE,lw=2,label="gradient boosting (peaks then overfits)")
pk=int(np.argmax(gb_auc))+1; ax.axvline(pk,color=GREY,ls=":",label=f"boosting peak ~{pk} trees")
ax.set_xlabel("number of trees"); ax.set_ylabel("test AUC"); ax.set_title("Forest averages (never overfits with trees); boosting adds (can overfit)")
ax.legend(frameon=False); plt.tight_layout(); plt.show()
print("The forest is robust to too many trees; boosting needs the tree count tuned (early stopping). In exchange, well-tuned")
print("boosting usually reaches a slightly higher ceiling -- which is why the production libraries (next notebook) are boosters.")

The forest is robust to too many trees; boosting needs the tree count tuned (early stopping). In exchange, well-tuned
boosting usually reaches a slightly higher ceiling -- which is why the production libraries (next notebook) are boosters.

Are boosted probabilities honest?

Every comparison so far has been AUC, which only cares about ranking. A model can rank perfectly and still report probabilities that are systematically wrong — and boosting is the classic offender, because stage-wise fitting of the log-odds keeps pushing confident cases further out. If a score is to be used as a probability of default rather than a sort key, that has to be checked separately.

Below: bin the test set by predicted probability and plot the observed default rate in each bin. On the diagonal the prediction is the outcome rate. Gradient boosting is shown against a random forest and a logistic regression on the same split — and the result does not go the way the folklore predicts.

from sklearn.calibration import calibration_curve
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
_mods={
 "gradient boosting":GradientBoostingClassifier(n_estimators=300,learning_rate=0.05,max_depth=3,random_state=0).fit(Xtr,ytr),
 "random forest":RandomForestClassifier(n_estimators=400,min_samples_leaf=5,random_state=0,n_jobs=-1).fit(Xtr,ytr),
 "logistic":make_pipeline(StandardScaler(),LogisticRegression(max_iter=2000)).fit(Xtr,ytr)}
_cols={"gradient boosting":BLUE,"random forest":GREEN,"logistic":GREY}
fig,ax=plt.subplots(1,2,figsize=(13,4.4))
print(f"{'model':>20} {'AUC':>7} {'ECE':>7} {'mean pred':>10} {'base rate':>10} {'max |gap|':>10}")
for _n,_m in _mods.items():
    _p=_m.predict_proba(Xte)[:,1]
    _q=np.quantile(_p,np.linspace(0,1,11)); _b=np.clip(np.digitize(_p,_q[1:-1]),0,9)
    _pp=[_p[_b==k].mean() for k in range(10)]; _oo=[yte[_b==k].mean() for k in range(10)]
    _ece=float(np.sum([np.mean(_b==k)*abs(_oo[k]-_pp[k]) for k in range(10)]))
    ax[0].plot(_pp,_oo,"o-",color=_cols[_n],lw=2,label=f"{_n} (ECE {_ece:.3f})")
    ax[1].hist(_p,bins=40,histtype="step",lw=2,color=_cols[_n],label=_n)
    print(f"{_n:>20} {roc_auc_score(yte,_p):>7.3f} {_ece:>7.3f} {_p.mean():>10.3f} {yte.mean():>10.3f} "
          f"{max(abs(np.array(_oo)-np.array(_pp))):>10.3f}")
_mx=max(_p.max() for _p in [m.predict_proba(Xte)[:,1] for m in _mods.values()])
ax[0].plot([0,_mx],[0,_mx],"k--",lw=1,label="perfect")
ax[0].set_xlabel("predicted P(default)"); ax[0].set_ylabel("observed default rate")
ax[0].set_title("Reliability: do the probabilities mean what they say?"); ax[0].legend(fontsize=8)
ax[1].set_xlabel("predicted P(default)"); ax[1].set_ylabel("clients"); ax[1].set_yscale("log")
ax[1].set_title("Distribution of predicted probabilities"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("\nThe textbook expectation is that boosting is the badly calibrated one -- stage-wise fitting of the log-odds")
print("keeps pushing confident cases outward, so its probabilities pile up near 0 and 1. That is not what happens")
print("here, and the ordering is worth reading carefully because it inverts twice.")
print("\nGradient boosting is the BEST calibrated of the three, and the logistic regression -- the model that actually")
print("estimates a probability by maximum likelihood -- is comfortably the worst, off by up to 0.11 in a decile.")
print("The reason is that the classic boosting pathology comes from OVERFITTING, and this fit is regularised into")
print("behaving: depth 3, learning rate 0.05, and 21,000 rows. The logistic fails in the opposite direction -- it is")
print("UNDERFIT, unable to bend to the nonlinearity in PAY_1, so its probabilities are systematically wrong at both")
print("ends however well it ranks in the middle.")
print("\nWhich makes the general point sharper than the folklore: miscalibration is not a property of a model CLASS,")
print("it is a property of a fit. AUC and calibration measure different things -- the ranking order above is not the")
print("calibration order -- so judge a model on the job it is doing. A sort key for a review queue needs the first")
print("column; a probability feeding an expected-loss calculation needs the diagonal, and the logistic would fail")
print("that job here despite being the only model in the table designed for it.")

               model     AUC     ECE  mean pred  base rate  max |gap|
   gradient boosting   0.773   0.012      0.219      0.221      0.021
       random forest   0.775   0.014      0.225      0.221      0.033
            logistic   0.715   0.059      0.219      0.221      0.110

The textbook expectation is that boosting is the badly calibrated one -- stage-wise fitting of the log-odds
keeps pushing confident cases outward, so its probabilities pile up near 0 and 1. That is not what happens
here, and the ordering is worth reading carefully because it inverts twice.

Gradient boosting is the BEST calibrated of the three, and the logistic regression -- the model that actually
estimates a probability by maximum likelihood -- is comfortably the worst, off by up to 0.11 in a decile.
The reason is that the classic boosting pathology comes from OVERFITTING, and this fit is regularised into
behaving: depth 3, learning rate 0.05, and 21,000 rows. The logistic fails in the opposite direction -- it is
UNDERFIT, unable to bend to the nonlinearity in PAY_1, so its probabilities are systematically wrong at both
ends however well it ranks in the middle.

Which makes the general point sharper than the folklore: miscalibration is not a property of a model CLASS,
it is a property of a fit. AUC and calibration measure different things -- the ranking order above is not the
calibration order -- so judge a model on the job it is doing. A sort key for a review queue needs the first
column; a probability feeding an expected-loss calculation needs the diagonal, and the logistic would fail
that job here despite being the only model in the table designed for it.

6. Graphical comparison — boosting's fitted shape

As in the forest and BART notebooks, one-feature views over continuous predictors: $P(\text{default})$ vs credit limit, and California value vs median income. Boosting bends to the nonlinear trend where the linear/logistic baseline is straight, and is far finer than a single tree. But note the difference from the forest: boosting fits residuals rather than averaging bootstrap trees, so it is less damped — sharper and wigglier in the sparse tails (extreme credit limits, very high incomes), a visible echo of its greater tendency to overfit (§5). Averaging smooths; sequential correction does not.

from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
fig,ax=plt.subplots(1,2,figsize=(14,4.8))
lim=Xtr[:,feat.index("LIMIT_BAL")]/1000.0
qb=np.quantile(lim,np.linspace(0,1,16)); bidx=np.clip(np.digitize(lim,qb)-1,0,14)
ctr=np.array([lim[bidx==k].mean() for k in range(15)]); emp=np.array([ytr[bidx==k].mean() for k in range(15)])
g=np.linspace(lim.min(),np.percentile(lim,99),300)
lo_=LogisticRegression(max_iter=1000).fit(lim[:,None],ytr).predict_proba(g[:,None])[:,1]
t1=DecisionTreeClassifier(max_depth=4,random_state=0).fit(lim[:,None],ytr).predict_proba(g[:,None])[:,1]
gbx=GradientBoostingClassifier(n_estimators=200,learning_rate=0.05,max_depth=3,min_samples_leaf=100,random_state=0).fit(lim[:,None],ytr).predict_proba(g[:,None])[:,1]
ax[0].scatter(ctr,emp,s=45,color="k",zorder=3,label="empirical"); ax[0].plot(g,lo_,color=ORANGE,lw=2,label="logistic")
ax[0].plot(g,t1,color=GREY,lw=2,label="single tree"); ax[0].plot(g,gbx,color=BLUE,lw=2,label="gradient boosting")
ax[0].set_xlabel("credit limit (NT$ thousands)"); ax[0].set_ylabel("P(default)"); ax[0].set_title("Classification: P(default) vs credit limit"); ax[0].legend(frameon=False,fontsize=8)
xi=Xhtr[:,hf.index("MedInc")]; gh=np.linspace(xi.min(),np.percentile(xi,99),300)
ols=LinearRegression().fit(xi[:,None],yhtr).predict(gh[:,None])
trg=DecisionTreeRegressor(max_depth=4,random_state=0).fit(xi[:,None],yhtr).predict(gh[:,None])
grx=GradientBoostingRegressor(n_estimators=200,learning_rate=0.05,max_depth=3,min_samples_leaf=50,random_state=0).fit(xi[:,None],yhtr).predict(gh[:,None])
ax[1].scatter(xi,yhtr,s=4,alpha=0.07,color="grey"); ax[1].plot(gh,ols,color=ORANGE,lw=2,label="linear")
ax[1].plot(gh,trg,color=GREY,lw=2,label="single tree"); ax[1].plot(gh,grx,color=BLUE,lw=2,label="gradient boosting")
ax[1].set_xlabel("median income"); ax[1].set_ylabel("median house value ($100k)"); ax[1].set_title("Regression: value vs income"); ax[1].legend(frameon=False,fontsize=8)
plt.tight_layout(); plt.show()
print("Boosting bends to the trend and is far finer than the single tree -- but it fits residuals rather than averaging, so it")
print("is LESS damped than the forest: sharper wiggles in the sparse tails, its overfitting tendency showing through (tamed by")
print("shallow trees, a small learning rate, and early stopping). Same flexibility as the forest, reached the opposite way.")

Boosting bends to the trend and is far finer than the single tree -- but it fits residuals rather than averaging, so it
is LESS damped than the forest: sharper wiggles in the sparse tails, its overfitting tendency showing through (tamed by
shallow trees, a small learning rate, and early stopping). Same flexibility as the forest, reached the opposite way.

7. Out-of-sample scoreboard — all techniques so far

A like-for-like comparison on the held-out test sets: the parametric baseline (logistic / linear regression), a single tree, the random forest, and gradient boosting — on credit-default classification (ROC-AUC, higher is better) and California-housing regression (RMSE, lower is better). BART is noted separately, since it was fit on a subsample for MCMC speed.

from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
cl={}                                                              # classification AUC (credit default, full test)
cl["logistic / linear"]=roc_auc_score(yte, make_pipeline(StandardScaler(),LogisticRegression(max_iter=2000)).fit(Xtr,ytr).predict_proba(Xte)[:,1])
cl["single tree"]=roc_auc_score(yte, DecisionTreeClassifier(min_samples_leaf=50,random_state=0).fit(Xtr,ytr).predict_proba(Xte)[:,1])
cl["random forest"]=roc_auc_score(yte, RandomForestClassifier(n_estimators=400,min_samples_leaf=5,random_state=0,n_jobs=-1).fit(Xtr,ytr).predict_proba(Xte)[:,1])
cl["grad. boosting"]=roc_auc_score(yte, GradientBoostingClassifier(n_estimators=300,learning_rate=0.05,max_depth=3,random_state=0).fit(Xtr,ytr).predict_proba(Xte)[:,1])
rg={}                                                              # regression RMSE (California, full test)
rg["logistic / linear"]=mean_squared_error(yhte, LinearRegression().fit(Xhtr,yhtr).predict(Xhte))**.5
rg["single tree"]=mean_squared_error(yhte, DecisionTreeRegressor(max_depth=8,random_state=0).fit(Xhtr,yhtr).predict(Xhte))**.5
rg["random forest"]=mean_squared_error(yhte, RandomForestRegressor(n_estimators=300,min_samples_leaf=3,random_state=0,n_jobs=-1).fit(Xhtr,yhtr).predict(Xhte))**.5
rg["grad. boosting"]=mean_squared_error(yhte, GradientBoostingRegressor(n_estimators=300,learning_rate=0.05,max_depth=3,random_state=0).fit(Xhtr,yhtr).predict(Xhte))**.5
print(pd.DataFrame({"classification AUC (higher=better)":cl,"regression RMSE (lower=better)":rg}).round(3).to_string())
fig,ax=plt.subplots(1,2,figsize=(13,4.4)); cn=list(cl); cols=[ORANGE,GREY,GREEN,BLUE]
ax[0].bar(cn,[cl[k] for k in cn],color=cols); ax[0].set_ylim(0.5,0.82); ax[0].set_ylabel("test AUC"); ax[0].set_title("Credit default — classification (higher better)")
for i,k in enumerate(cn): ax[0].text(i,cl[k]+0.004,f"{cl[k]:.3f}",ha="center",fontsize=8)
rn=list(rg)
ax[1].bar(rn,[rg[k] for k in rn],color=cols); ax[1].set_ylabel("test RMSE ($100k)"); ax[1].set_title("California housing — regression (lower better)")
for i,k in enumerate(rn): ax[1].text(i,rg[k]+0.004,f"{rg[k]:.3f}",ha="center",fontsize=8)
plt.setp(ax[0].get_xticklabels(),rotation=15); plt.setp(ax[1].get_xticklabels(),rotation=15); plt.tight_layout(); plt.show()
print("The ensembles (random forest, gradient boosting) clearly top the single tree and the parametric baseline on both tasks;")
print("forest and boosting are neck-and-neck (boosting usually edges ahead when carefully tuned -- its production forms are next).")
print("BART, on a 3,000-row subsample for MCMC speed, reached AUC ~0.76 / RMSE ~0.66 -- competitive, and with credible intervals.")

                   classification AUC (higher=better)  regression RMSE (lower=better)
logistic / linear                               0.715                           0.737
single tree                                     0.737                           0.666
random forest                                   0.775                           0.523
grad. boosting                                  0.773                           0.522

The ensembles (random forest, gradient boosting) clearly top the single tree and the parametric baseline on both tasks;
forest and boosting are neck-and-neck (boosting usually edges ahead when carefully tuned -- its production forms are next).
BART, on a 3,000-row subsample for MCMC speed, reached AUC ~0.76 / RMSE ~0.66 -- competitive, and with credible intervals.

8. Summary

Boosting is the bias-cutting counterpart to the forest's variance-cutting. We built both seminal forms from scratch — AdaBoost (re-weight the hard cases, weighted vote) and gradient boosting (fit trees to the loss gradient, step with a learning rate) — and matched scikit-learn on credit default and California housing. Out of sample the booster is competitive with the random forest and BART (AUC ≈ 0.77), and the two knobs behave exactly as theory says: a small learning rate with many trees generalises best, while too many trees eventually overfit — the mirror image of the forest, which merely plateaus. Where the forest averages independent deep trees, boosting adds dependent shallow ones; both end at a smooth, flexible fit, from opposite directions.

That extra tuning burden buys a slightly higher ceiling, which is why the production ensembles are boosters. The next notebook takes gradient boosting to its industrial form — XGBoost, LightGBM, and CatBoost — adding second-order (Newton) boosting, histogram splitting, clever regularisation, and native categorical handling, benchmarked head-to-head. In R, this same Friedman gradient boosting is the gbm package (companion notebook). Its Bayesian cousin is BART, which regularises the trees with priors instead of shrinkage.