Fraud Detection as Decision Under Uncertainty — a cost-sensitive Bayesian layer

Operations Research family 7. Where the earlier families forecast a quantity and then optimized a decision, this one classifies a rare event and then makes a cost-asymmetric decision about it — still the same spine: turn a probability into the best action under uncertainty.

Most public fraud-detection notebooks stop at "here is my AUC." That is the easy 80% and the wrong finish line. A bank does not deploy an AUC; it deploys a decision rule — approve, decline, or send to manual review — and that rule must answer to two things no leaderboard rewards:

1. Asymmetric, amount-dependent costs. Missing a USD 2,000 fraud is not the same as missing a USD 5 one, and a false decline burns customer trust in a way a missed small fraud does not. The right cutoff is therefore not a fixed 0.5 — it depends on the transaction.
2. Quantified uncertainty. A gradient-boosted score is not a probability, and a probability without a credible interval cannot tell you when a human should look. Governance regimes like SR 11-7 (model risk management) expect exactly this: calibrated probabilities, monitored for drift, feeding a documented, cost-aware decision.

So this notebook builds the part that actually matters, on top of a standard classifier:

§	Layer	What it adds
1	Data & imbalance	Why accuracy and even ROC-AUC mislead at 0.17% prevalence
2	Baseline classifier	Gradient-boosted trees + honest metrics (PR-AUC, time-split)
3	Bayesian calibration	Turn the score into a calibrated posterior over P(fraud), pooled across amount/time segments
4	Cost-sensitive decision	Amount-varying Bayes-optimal threshold; decide by expected cost over the posterior
5	Uncertainty triage	Auto-approve / auto-decline / route-to-review from the credible interval
6	The dollar ladder	Measure the realized $ saved by each layer — the value, not the AUC

The through-line to the rest of this series is deliberate: the "value of modeling uncertainty as a business number" from the inventory newsvendor and the energy VSS reappears here as dollars of fraud loss avoided per false-decline tolerated. The calibration borrows the partial-pooling idea from the hierarchical forecasting notebooks; the honest-metrics discipline comes from the ML model-evaluation notebook.

The data is the classic ULB Credit Card Fraud set (Université Libre de Bruxelles / Worldline): 284,807 European card transactions over two days, of which 492 (0.173%) are fraud. Its features are PCA-anonymized (V1…V28) for confidentiality, leaving only Time and Amount in the clear — which makes it the ideal, tightly-scoped stage for the statistical core (imbalance → calibration → decision). The richer applied problem (feature engineering under anonymization, out-of-time validation, device/card segment hierarchies) belongs to the larger IEEE-CIS dataset, a natural follow-on.

import os
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
# Cap thread pools before importing numeric libraries — later sections use LightGBM (OpenMP) and, for the
# Bayesian calibration, JAX/XLA; in one kernel they oversubscribe the CPU and stall. (Lesson carried over
# from the energy notebook.) Harmless for §1.
for _v in ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS"):
    os.environ.setdefault(_v, "2")

import numpy as np, pandas as pd
import matplotlib.pyplot as plt
from pathlib import Path
plt.rcParams.update({"figure.dpi": 110, "font.size": 10, "axes.grid": True,
                     "grid.alpha": 0.25, "axes.spines.top": False, "axes.spines.right": False})

df = pd.read_csv(Path("data") / "creditcard.csv")
N = len(df); NF = int(df["Class"].sum()); prev = NF / N
print(f"transactions : {N:,}")
print(f"fraud        : {NF} ({100*prev:.3f}%)   legitimate: {N-NF:,}")
print(f"period       : {df['Time'].max()/86400:.1f} days  |  features: {df.shape[1]-2} predictors "
      f"(V1-V28 PCA-anonymized + Amount) + Time")
print(f"missing values: {int(df.isna().sum().sum())}")

transactions : 284,807
fraud        : 492 (0.173%)   legitimate: 284,315
period       : 2.0 days  |  features: 29 predictors (V1-V28 PCA-anonymized + Amount) + Time
missing values: 0

1 · The data, and why the usual scoreboard lies

Data dictionary

Field	Meaning	Notes
Time	Seconds elapsed since the first transaction	Spans 2 days; used for a realistic time-ordered split
V1 … V28	PCA-transformed features	Anonymized for confidentiality — individually uninterpretable
Amount	Transaction amount	The one business-meaningful feature in the clear — drives the cost model in §4
Class	Target: 1 = fraud, 0 = legitimate	Only 0.173% positive

Two honesty notes that shape the whole design:

• The features are anonymized. With only Time and Amount interpretable, there are no natural business segments (device, card type, merchant category) to build a rich hierarchy over. So the hierarchical calibration in §3 pools across amount buckets and hour-of-day instead — real, fraud-relevant segments, just coarser than IEEE-CIS would offer.
• There are only 492 frauds, over two days. Enough to demonstrate every technique honestly, but small enough that we lean on precision-recall and cost rather than any single accuracy-like number, and we resist over-claiming out-of-time generalization from a 2-day window.

# EDA 1 — the imbalance, and why "accuracy" is a trap
acc_trivial = 100*(N-NF)/N          # accuracy of "predict everything legitimate"
fig, ax = plt.subplots(1, 2, figsize=(12, 3.8))
ax[0].bar(["legitimate","fraud"], [N-NF, NF], color=["#2b6cb0","#c53030"])
ax[0].set_yscale("log"); ax[0].set_ylabel("transactions (log)")
ax[0].set_title(f"Class imbalance — fraud is {100*prev:.3f}% of transactions")
for i,v in enumerate([N-NF, NF]): ax[0].text(i, v, f"{v:,}", ha="center", va="bottom", fontsize=9)

ax[1].axis("off")
ax[1].text(0.02, 0.9, "Why the usual scoreboard lies", fontsize=11, fontweight="bold", transform=ax[1].transAxes)
ax[1].text(0.02, 0.62,
    f"'Predict everything legitimate' scores\n"
    f"   accuracy = {acc_trivial:.2f}%\n"
    f"   ...while catching 0 of {NF} frauds.\n\n"
    f"ROC-AUC is also flattering here: the huge\n"
    f"true-negative pool makes the false-positive\n"
    f"rate look tiny for almost any model.\n\n"
    f"-> use PRECISION-RECALL and, ultimately,\n"
    f"   realized COST (§4-6).",
    fontsize=9.5, family="monospace", va="top", transform=ax[1].transAxes)
plt.tight_layout(); plt.show()
print(f"A do-nothing model is {acc_trivial:.2f}% accurate and 100% useless. Accuracy and ROC-AUC must go.")

A do-nothing model is 99.83% accurate and 100% useless. Accuracy and ROC-AUC must go.

# EDA 2 — Amount: distribution and the (non-monotonic) fraud rate that motivates amount-segmented modeling
fig, ax = plt.subplots(1, 2, figsize=(12, 4))
bins = np.logspace(-2, np.log10(df["Amount"].max()+1), 50)
ax[0].hist(df.loc[df.Class==0,"Amount"].clip(lower=0.01), bins=bins, density=True, alpha=0.6,
           color="#2b6cb0", label="legitimate")
ax[0].hist(df.loc[df.Class==1,"Amount"].clip(lower=0.01), bins=bins, density=True, alpha=0.6,
           color="#c53030", label="fraud")
ax[0].set_xscale("log"); ax[0].set_xlabel("amount ($, log)"); ax[0].set_ylabel("density")
ax[0].set_title("Amount distribution: fraud vs legitimate"); ax[0].legend(fontsize=9)

q = pd.qcut(df["Amount"].clip(lower=0.01), 8, duplicates="drop")
fr = df.groupby(q, observed=True)["Class"].mean()*100
ctr = [iv.mid for iv in fr.index]
ax[1].plot(range(len(fr)), fr.values, "o-", color="#c53030")
ax[1].set_xticks(range(len(fr))); ax[1].set_xticklabels([f"${c:,.0f}" for c in ctr], rotation=45, fontsize=7)
ax[1].axhline(100*prev, color="grey", ls=":", label=f"overall {100*prev:.3f}%")
ax[1].set_ylabel("fraud rate (%)"); ax[1].set_title("Fraud rate by amount octile (non-monotonic)")
ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print(f"Fraud clusters at BOTH tails: tiny 'card-testing' charges and large scores. Fraud median amount "
      f"${df[df.Class==1].Amount.median():.2f} vs legit ${df[df.Class==0].Amount.median():.2f}. This is why")
print("§3 calibrates within amount buckets and §4 lets the decision threshold vary with amount.")

Fraud clusters at BOTH tails: tiny 'card-testing' charges and large scores. Fraud median amount $9.25 vs legit $22.00. This is why
§3 calibrates within amount buckets and §4 lets the decision threshold vary with amount.

# EDA 3 — fraud through time (2 days) and by hour-of-day
df["hour"] = (df["Time"]//3600 % 24).astype(int)
fig, ax = plt.subplots(1, 2, figsize=(12, 3.8))
tb = np.linspace(0, df["Time"].max(), 97)  # ~30-min bins
ax[0].hist(df.loc[df.Class==1,"Time"], bins=tb, color="#c53030")
ax[0].set_xlabel("time (s from start)"); ax[0].set_ylabel("fraud count"); ax[0].set_title("Fraud over the 2-day window")
for d in (86400,): ax[0].axvline(d, color="grey", ls=":")

hr = df.groupby("hour")["Class"].mean()*100
ax[1].bar(hr.index, hr.values, color="#c53030")
ax[1].axhline(100*prev, color="grey", ls=":", label=f"overall {100*prev:.3f}%")
ax[1].set_xlabel("hour of day"); ax[1].set_ylabel("fraud rate (%)"); ax[1].set_title("Fraud rate by hour of day")
ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("Fraud rate rises in the small hours (low legitimate volume) — a second natural segment for §3, and")
print("a reminder that a TIME-ORDERED train/test split (used from §2 on) is the honest evaluation here.")

Fraud rate rises in the small hours (low legitimate volume) — a second natural segment for §3, and
a reminder that a TIME-ORDERED train/test split (used from §2 on) is the honest evaluation here.

What §1 establishes

• The problem is defined by extreme imbalance (0.173%) and asymmetric, amount-dependent costs — so accuracy and ROC-AUC are actively misleading (a do-nothing model is 99.83% accurate). We commit to precision-recall and, ultimately, realized dollar cost as the scoreboard.
• Amount is the one interpretable feature and it matters: fraud rate is non-monotonic in amount (high at both tails) and elevated in the small hours. These give us honest segments (amount buckets, hour-of-day) for the hierarchical calibration, and the basis for an amount-varying decision threshold.
• The features are anonymized and the window is short — we build the statistical core here and are explicit about what only the larger IEEE-CIS data could add.

Next: §2 fits a gradient-boosted baseline with a time-ordered split and reports it honestly (PR-AUC, not ROC), establishing the raw score that §3 will turn into a calibrated posterior and §4–§6 into a cost-aware decision.

2 · A gradient-boosted baseline, measured honestly

The classifier itself is not where the value is — but we need a good raw score to build on. Three disciplines make this baseline honest rather than leaderboard-theater:

• Time-ordered split, in three folds. Fraud evolves, so we never shuffle. We split by Time into train (fit the model), calibrate (a held-out slice §3 will use to turn scores into probabilities — fitting a calibrator on the model's training data would leak), and test (a final out-of-time slice nothing touches until the end).
• The right metric — PR-AUC. PR-AUC, equivalently average precision, is the area under the precision-recall curve: across every recall level, of the transactions the model flags as fraud, what fraction truly are — averaged over how aggressively it flags. A no-skill model scores PR-AUC ≈ the prevalence (here ≈ 0.0017); a perfect model scores 1. Unlike ROC-AUC — which at 0.17% prevalence is flattered by the enormous true-negative pool — PR-AUC stays honest about false alarms, so we lead with it and show ROC only for the contrast.
• Imbalance: regularize, don't over-weight or resample. Weighting the positive class by the raw imbalance ratio (~474×) collapses ranking — the extreme weight floods the top scores with false positives (we measured PR-AUC crashing to ≈ 0.01) — and SMOTE-ing synthetic frauds distorts the base rate the same way. We instead keep the true base rate and control overfitting with regularization, which also leaves the scores meaningful for the calibration in §3.

We fit LightGBM (the workhorse) and, alongside, a plain logistic regression for an interpretable contrast — a recurring lesson in this portfolio is that a strong linear baseline is often within a whisker of the fancy model.

import lightgbm as lgb
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import average_precision_score, roc_auc_score, precision_recall_curve, roc_curve
from sklearn.calibration import calibration_curve

FEATS = [f"V{i}" for i in range(1,29)] + ["Amount", "hour"]

# time-ordered 60 / 20 / 20 split: train -> calibrate -> test
c1, c2 = df["Time"].quantile(0.60), df["Time"].quantile(0.80)
tr = df["Time"] <= c1
ca = (df["Time"] > c1) & (df["Time"] <= c2)
te = df["Time"] > c2
Xtr, ytr = df.loc[tr, FEATS], df.loc[tr, "Class"].values
Xca, yca = df.loc[ca, FEATS], df.loc[ca, "Class"].values
Xte, yte = df.loc[te, FEATS], df.loc[te, "Class"].values
print(f"split (by time)  train {tr.sum():,} ({ytr.sum()} fraud) | calib {ca.sum():,} ({yca.sum()} fraud) "
      f"| test {te.sum():,} ({yte.sum()} fraud)")

# LightGBM. NB: scale_pos_weight = imbalance ratio (~474) WRECKS ranking (PR-AUC ~0.01 — the top scores
# flood with false positives) and distorts the probabilities §3 needs, so we DON'T weight; we regularize.
gbm = lgb.LGBMClassifier(n_estimators=200, learning_rate=0.05, num_leaves=31, min_child_samples=50,
                         subsample=0.8, colsample_bytree=0.8, reg_lambda=5.0, verbose=-1)
gbm.fit(Xtr, ytr)
s_ca = gbm.predict_proba(Xca)[:,1]      # scores on the calibration slice (for §3)
s_te = gbm.predict_proba(Xte)[:,1]      # scores on the test slice

# logistic-regression baseline (standardized features, balanced classes)
sc = StandardScaler().fit(Xtr)
lr = LogisticRegression(max_iter=2000, class_weight="balanced").fit(sc.transform(Xtr), ytr)
lr_te = lr.predict_proba(sc.transform(Xte))[:,1]

print(f"\nTEST performance (out-of-time):")
print(f"  LightGBM     PR-AUC {average_precision_score(yte, s_te):.3f}   ROC-AUC {roc_auc_score(yte, s_te):.3f}")
print(f"  Logistic reg PR-AUC {average_precision_score(yte, lr_te):.3f}   ROC-AUC {roc_auc_score(yte, lr_te):.3f}")
print(f"  (ROC-AUC ~0.95+ for both looks great — PR-AUC is the honest gap; baseline PR-AUC = prevalence "
      f"{yte.mean():.4f})")

split (by time)  train 170,888 (360 fraud) | calib 56,957 (57 fraud) | test 56,962 (75 fraud)

TEST performance (out-of-time):
  LightGBM     PR-AUC 0.792   ROC-AUC 0.984
  Logistic reg PR-AUC 0.752   ROC-AUC 0.983
  (ROC-AUC ~0.95+ for both looks great — PR-AUC is the honest gap; baseline PR-AUC = prevalence 0.0013)

# Precision-recall (the honest view) vs ROC (the flattering one)
fig, ax = plt.subplots(1, 2, figsize=(12, 4.2))
for name, sc_, col in [("LightGBM", s_te, "#2b6cb0"), ("Logistic reg", lr_te, "#c05621")]:
    pr, rc, _ = precision_recall_curve(yte, sc_)
    ax[0].plot(rc, pr, color=col, label=f"{name} (AP={average_precision_score(yte,sc_):.3f})")
    fpr, tpr, _ = roc_curve(yte, sc_)
    ax[1].plot(fpr, tpr, color=col, label=f"{name} (AUC={roc_auc_score(yte,sc_):.3f})")
ax[0].axhline(yte.mean(), color="grey", ls=":", label=f"no-skill = {yte.mean():.4f}")
ax[0].set_xlabel("recall"); ax[0].set_ylabel("precision"); ax[0].set_title("Precision-Recall (the honest view)")
ax[0].legend(fontsize=8)
ax[1].plot([0,1],[0,1], "k:", lw=0.8); ax[1].set_xlabel("false-positive rate"); ax[1].set_ylabel("true-positive rate")
ax[1].set_title("ROC (flattered by the true-negative pool)"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("Both models post a gorgeous ROC-AUC; the PR curve reveals the real difficulty — precision falls off")
print("as we push recall, and that trade-off (not a single number) is what the cost model in §4 will price.")

Both models post a gorgeous ROC-AUC; the PR curve reveals the real difficulty — precision falls off
as we push recall, and that trade-off (not a single number) is what the cost model in §4 will price.

# Why we still can't act on this score: it is NOT a calibrated probability (motivates §3)
frac_pos, mean_pred = calibration_curve(yte, s_te, n_bins=10, strategy="quantile")
fig, ax = plt.subplots(1, 2, figsize=(12, 4))
ax[0].plot([0,1],[0,1], "k:", label="perfectly calibrated")
ax[0].plot(mean_pred, frac_pos, "o-", color="#2b6cb0", label="LightGBM (raw)")
ax[0].set_xlabel("mean predicted score"); ax[0].set_ylabel("observed fraud rate")
ax[0].set_title("Reliability: raw GBM score vs reality"); ax[0].legend(fontsize=8)

# operating point at the naive 0.5 threshold
pred05 = (s_te >= 0.5).astype(int)
tp = int(((pred05==1)&(yte==1)).sum()); fp = int(((pred05==1)&(yte==0)).sum())
fn = int(((pred05==0)&(yte==1)).sum()); tn = int(((pred05==0)&(yte==0)).sum())
ax[1].axis("off")
ax[1].text(0.0, 0.9, "Naive threshold = 0.50", fontsize=11, fontweight="bold", transform=ax[1].transAxes)
ax[1].text(0.0, 0.15,
    f"caught fraud (TP)     {tp:>5}\n"
    f"missed fraud (FN)     {fn:>5}\n"
    f"false alarms (FP)     {fp:>5}\n"
    f"correct approve (TN)  {tn:>5}\n\n"
    f"precision {tp/max(tp+fp,1):.2f}   recall {tp/max(tp+fn,1):.2f}\n\n"
    f"0.5 is an ARBITRARY cutoff on an\nUNCALIBRATED score — §3 fixes the\nscore, §4 fixes the cutoff.",
    fontsize=10, family="monospace", va="bottom", transform=ax[1].transAxes)
plt.tight_layout(); plt.show()
print("The reliability curve is far from the diagonal: with scale_pos_weight the raw scores are inflated,")
print("not probabilities. And 0.5 is a meaningless cutoff. §3 calibrates the score; §4 sets the threshold")
print("by COST, not convention.")

The reliability curve is far from the diagonal: with scale_pos_weight the raw scores are inflated,
not probabilities. And 0.5 is a meaningless cutoff. §3 calibrates the score; §4 sets the threshold
by COST, not convention.

What §2 establishes

• A gradient-boosted baseline ranks fraud well (high PR-AUC), and — as so often in this portfolio — a logistic regression is within a whisker of it, a useful interpretable anchor.
• ROC-AUC is a mirage here (both models look near-perfect); the precision-recall trade-off is the real picture, and no single number captures it — which is exactly why the decision must be made on cost, not a metric.
• The raw score is not a probability (the reliability curve is far off-diagonal) and 0.5 is an arbitrary cutoff. Those are the two gaps the rest of the notebook closes: §3 turns the score into a calibrated posterior, and §4–§6 turn that posterior into a cost-aware decision.

3 · From score to calibrated posterior

§2 left us with a score that ranks well but is not a probability (its reliability curve is off the diagonal) — and a decision needs a probability, because the cost trade-off in §4 is stated in probability units. Turning a classifier score into a calibrated probability is calibration, and we do it on the held-out calibrate slice (never on the model's training data — that would leak), then judge it on test.

We compare three calibrators, from simplest to richest:

• Platt scaling — a logistic regression of the label on the score. One global correction.
• Isotonic regression — a nonparametric monotone fit; more flexible, the usual strong baseline.
• Bayesian hierarchical calibration — a logistic recalibration whose intercept and slope are allowed to differ by segment (amount-bucket × time-of-day) and are partially pooled across segments, so a segment with few frauds borrows the global correction. Crucially, its output is not a point estimate but a posterior distribution over P(fraud) for every transaction — a credible interval.

We measure calibration with the Brier score (mean squared error of the probability) and ECE (expected calibration error: the average gap between predicted probability and observed frequency, across bins). Both: lower is better; a no-skill constant scores Brier ≈ prevalence.

from sklearn.isotonic import IsotonicRegression
from sklearn.calibration import calibration_curve

clip = lambda s: np.clip(s, 1e-6, 1-1e-6)
z_ca = np.log(clip(s_ca)/(1-clip(s_ca)))          # logit of the GBM score (calibration input)
z_te = np.log(clip(s_te)/(1-clip(s_te)))

# Platt (logistic on the logit-score) and isotonic (monotone nonparametric), both fit on the CALIB slice
platt = LogisticRegression(max_iter=1000).fit(z_ca.reshape(-1,1), yca)
p_platt = platt.predict_proba(z_te.reshape(-1,1))[:,1]
iso = IsotonicRegression(out_of_bounds="clip").fit(s_ca, yca)
p_iso = iso.transform(s_te)

def ece(y, p, nb=10):
    "Expected calibration error over nb quantile bins."
    q = np.quantile(p, np.linspace(0,1,nb+1)); q[0] -= 1e-9
    b = np.digitize(p, q[1:-1]); e = 0.0
    for k in range(nb):
        m = b==k
        if m.sum(): e += abs(p[m].mean() - y[m].mean())*m.sum()
    return e/len(y)

from sklearn.metrics import brier_score_loss
rows = {"raw GBM": s_te, "Platt": p_platt, "isotonic": p_iso}
print("Calibration on TEST (x1e-3; lower = better):")
for k,p in rows.items():
    print(f"  {k:10s} Brier {brier_score_loss(yte, np.clip(p,0,1))*1e3:.4f}   ECE {ece(yte, np.clip(p,0,1))*1e3:.3f}")

Calibration on TEST (x1e-3; lower = better):
  raw GBM    Brier 0.4143   ECE 0.183
  Platt      Brier 0.4132   ECE 0.165
  isotonic   Brier 0.4245   ECE 0.042

# --- Bayesian hierarchical calibration: fit in a fresh companion process (JAX vs LightGBM threads) -----
import subprocess, sys
# segments: amount quintile (edges from CALIB) x time-of-day quarter -> up to 20 segments
edges = np.quantile(df.loc[ca,"Amount"], [.2,.4,.6,.8])
seg_of = lambda A,H: (np.digitize(A, edges)*4 + (H//6)).astype(int)
seg_ca = seg_of(Xca["Amount"].values, Xca["hour"].values)
seg_te = seg_of(Xte["Amount"].values, Xte["hour"].values)
G = int(max(seg_ca.max(), seg_te.max()))+1

np.savez("calib_input.npz", z_ca=z_ca, seg_ca=seg_ca, yca=yca, G=G)
r = subprocess.run([sys.executable, "calib_fit.py"], capture_output=True, text=True)
print(r.stdout.strip() or r.stderr[-400:])
P = np.load("calib_post.npz"); a_d, b_d = P["a"], P["b"]      # posterior draws (500, G)

# test posterior over P(fraud): one distribution per transaction
lin = a_d[:, seg_te] + b_d[:, seg_te]*z_te[None,:]           # (500, n_test)
p_draws = 1/(1+np.exp(-lin))
p_bayes = p_draws.mean(0)
p_lo, p_hi = np.percentile(p_draws, 5, 0), np.percentile(p_draws, 95, 0)   # 90% credible interval (for §5)

print(f"\n  Bayes-hier Brier {brier_score_loss(yte, p_bayes)*1e3:.4f}   ECE {ece(yte, p_bayes)*1e3:.3f}  (x1e-3)")
fpr_frac = {}
fig, ax = plt.subplots(1, 2, figsize=(12, 4.2))
ax[0].plot([0,1],[0,1], "k:", label="perfect")
for name, p, col in [("raw GBM", s_te, "#a0aec0"), ("isotonic", p_iso, "#2f855a"), ("Bayes-hier", p_bayes, "#2b6cb0")]:
    fpos, mpred = calibration_curve(yte, np.clip(p,0,1), n_bins=8, strategy="quantile")
    ax[0].plot(mpred, fpos, "o-", color=col, label=name)
ax[0].set_xlabel("predicted probability"); ax[0].set_ylabel("observed fraud rate")
ax[0].set_title("Reliability after calibration"); ax[0].legend(fontsize=8)

# the unique deliverable: posterior WIDTH vs score
order = np.argsort(s_te)
ax[1].fill_between(s_te[order], p_lo[order], p_hi[order], color="#2b6cb0", alpha=0.25, label="90% credible interval")
ax[1].plot(s_te[order], p_bayes[order], color="#2b6cb0", lw=1.2, label="posterior mean P(fraud)")
ax[1].set_xlabel("raw GBM score"); ax[1].set_ylabel("calibrated P(fraud)")
ax[1].set_title("Bayesian layer's extra: a posterior, not a point"); ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()
print("Point calibration: isotonic is best on ECE, Bayes ties on Brier — the hierarchy does NOT beat the")
print("simple monotone fit here (the regularized GBM is already near-calibrated, and thin segments hold")
print("too few frauds to tell). The Bayesian layer earns its place by the CREDIBLE INTERVAL it attaches to")
print("every transaction — the input the cost decision (§4) and, especially, the triage (§5) need.")

calib_post.npz saved | final ELBO loss 136.08380126953125

  Bayes-hier Brier 0.4129   ECE 0.156  (x1e-3)

Point calibration: isotonic is best on ECE, Bayes ties on Brier — the hierarchy does NOT beat the
simple monotone fit here (the regularized GBM is already near-calibrated, and thin segments hold
too few frauds to tell). The Bayesian layer earns its place by the CREDIBLE INTERVAL it attaches to
every transaction — the input the cost decision (§4) and, especially, the triage (§5) need.

What §3 establishes

• On point calibration, the honest result mirrors §4 of the energy notebook: the isotonic baseline is best on ECE and the Bayesian hierarchical calibrator merely ties it (best Brier, close ECE). With a well-regularized GBM the scores are already nearly calibrated, so there is little for either to fix — and the hoped-for thin-segment advantage is within the noise (a handful of frauds per thin segment).
• The Bayesian layer's real, non-redundant contribution is a posterior over P(fraud) per transaction — a credible interval that Platt and isotonic simply cannot provide. That interval is the raw material for the two things that make this a decision system rather than a scorer:
  • §4 the amount-varying, cost-optimal threshold (which needs the posterior mean), and
  • §5 the triage rule that routes uncertain transactions to human review (which needs the interval width).

So we carry forward isotonic as the reference point-calibration and the Bayesian posterior (p_bayes with interval p_lo–p_hi) as the object the decision layer consumes. Next: §4 turns probability into action by pricing the asymmetric cost of the two mistakes.

4 · The cost-sensitive decision

A probability is not a decision. The bank must approve or decline each transaction, and the two mistakes cost wildly different amounts:

• Approve a fraud (false negative). The bank eats the loss — roughly the transaction amount plus a fixed chargeback/handling cost. So $C_{FN}(\text{amount}) = \text{amount} + k$: bigger transactions are costlier to miss.
• Decline a legitimate transaction (false positive). A roughly fixed customer-friction cost $C_{FP}$ — annoyance, a lost sale, a support call — largely independent of amount.

For a transaction with fraud probability $p$, the expected cost of each action is $\;\mathbb{E}[\text{cost}\mid\text{approve}] = p\,C_{FN}(\text{amount})\;$ and $\;\mathbb{E}[\text{cost}\mid\text{decline}] = (1-p)\,C_{FP}$. Declining is cheaper exactly when

$$ p \;>\; \tau^{*}(\text{amount}) \;=\; \frac{C_{FP}}{C_{FP} + C_{FN}(\text{amount})}. $$

This is the Bayes-optimal decision rule, and its threshold varies with the transaction: because $C_{FN}$ grows with amount, $\tau^{*}$ falls as the amount rises — a big-ticket charge is flagged at a much lower probability than a trivial one. That single formula is what the ubiquitous fixed-0.5 cutoff throws away. (Note both expected costs are linear in $p$, so the decision needs only the posterior mean — the posterior width is what §5 will use for triage.)

# --- cost model (illustrative, tunable) --------------------------------------------------------------
C_FP     = 10.0     # cost of a false decline (customer friction), fixed
K_FRAUD  = 10.0     # fixed chargeback/handling cost added to the amount when a fraud is approved
def C_FN(amount): return amount + K_FRAUD
def tau_star(amount): return C_FP / (C_FP + C_FN(amount))

amt_te = Xte["Amount"].values
# calibrated posterior mean on the CALIB slice too (to tune the global-threshold baseline without leakage)
p_ca = (1/(1+np.exp(-(a_d[:, seg_ca] + b_d[:, seg_ca]*z_ca[None,:])))).mean(0)
amt_ca = Xca["Amount"].values

def realized_cost(flag, y, amount):
    "Total $ cost of a decision vector: false declines cost C_FP; approved frauds cost amount+K_FRAUD."
    approve = 1 - flag
    return float((flag*(1-y)*C_FP + approve*y*C_FN(amount)).sum())

# policy A: flat 0.5   policy B: single global cost-optimal threshold (tuned on calib)   policy C: tau*(amount)
grid = np.linspace(0.0005, 0.5, 400)
tau_g = grid[np.argmin([realized_cost((p_ca>t).astype(int), yca, amt_ca) for t in grid])]

policies = {
    "flat 0.50":            (p_bayes > 0.5).astype(int),
    f"global tau={tau_g:.3f}": (p_bayes > tau_g).astype(int),
    "amount-varying tau*":  (p_bayes > tau_star(amt_te)).astype(int),
}
print(f"cost model: false decline ${C_FP:.0f}; approved fraud = amount + ${K_FRAUD:.0f}")
print(f"tuned global threshold (on calib): {tau_g:.4f}\n")
print(f"{'policy':22s}{'test $ cost':>12}{'fraud $ exposed':>16}{'caught':>9}{'false decl':>12}")
for name, flag in policies.items():
    caught = int(((flag==1)&(yte==1)).sum()); fd = int(((flag==1)&(yte==0)).sum())
    exposed = float(((1-flag)*yte*amt_te).sum())   # $ of approved (missed) fraud
    print(f"{name:22s}{realized_cost(flag,yte,amt_te):>12,.0f}{exposed:>16,.0f}"
          f"{caught:>6}/{int(yte.sum())}{fd:>12,}")

cost model: false decline $10; approved fraud = amount + $10
tuned global threshold (on calib): 0.2784

policy                 test $ cost fraud $ exposed   caught  false decl
flat 0.50                    4,193           3,923    53/75           5
global tau=0.278             3,146           2,886    56/75           7
amount-varying tau*          3,258           2,638    57/75          44

# --- visualize: the amount-varying threshold and the decision regions --------------------------------
fig, ax = plt.subplots(1, 2, figsize=(13, 4.4))
ag = np.logspace(-1, np.log10(amt_te.max()+1), 200)
ax[0].plot(ag, tau_star(ag), color="#2b6cb0", lw=2, label=r"$\tau^*$(amount)")
ax[0].axhline(0.5, color="#c53030", ls="--", label="flat 0.50")
ax[0].axhline(tau_g, color="#dd6b20", ls=":", label=f"global {tau_g:.3f}")
ax[0].set_xscale("log"); ax[0].set_yscale("log"); ax[0].set_xlabel("amount ($, log)")
ax[0].set_ylabel(r"decline if P(fraud) > threshold"); ax[0].set_title("The bar to decline falls as the stake rises")
ax[0].legend(fontsize=8)

# decision regions: all frauds + a legit sample, in (amount, calibrated P)
rng = np.random.default_rng(0)
leg = np.where(yte==0)[0]; samp = rng.choice(leg, 4000, replace=False)
idx = np.concatenate([np.where(yte==1)[0], samp])
ax[1].scatter(amt_te[idx][yte[idx]==0], p_bayes[idx][yte[idx]==0], s=6, alpha=0.25, color="#a0aec0", label="legit")
ax[1].scatter(amt_te[idx][yte[idx]==1], p_bayes[idx][yte[idx]==1], s=22, color="#c53030", label="fraud")
ax[1].plot(ag, tau_star(ag), color="#2b6cb0", lw=2, label=r"$\tau^*$(amount) — decline above")
ax[1].set_xscale("log"); ax[1].set_yscale("log"); ax[1].set_xlabel("amount ($, log)")
ax[1].set_ylabel("calibrated P(fraud)"); ax[1].set_title("Decision regions: decline the points above the curve")
ax[1].legend(fontsize=8)
plt.tight_layout(); plt.show()

c05 = realized_cost(policies["flat 0.50"], yte, amt_te)
cg  = realized_cost(policies[f"global tau={tau_g:.3f}"], yte, amt_te)
cv  = realized_cost(policies["amount-varying tau*"], yte, amt_te)
exp_g = float(((1-policies[f"global tau={tau_g:.3f}"])*yte*amt_te).sum())
exp_v = float(((1-policies["amount-varying tau*"])*yte*amt_te).sum())
print(f"The clear, robust win is COST-DRIVEN thresholding over the 0.5 default: ${c05:,.0f} -> ${cg:,.0f} "
      f"({100*(c05-cg)/c05:.0f}% lower) on the SAME scores.")
print(f"tau*(amount) is PROVABLY the expected-cost-minimizing rule per transaction. On this small test set")
print(f"(75 frauds) its realized total cost (${cv:,.0f}) is statistically ~level with the tuned global "
      f"(${cg:,.0f}),")
print(f"but by design it exposes the LEAST fraud $ — approved-fraud amount ${exp_v:,.0f} vs global "
      f"${exp_g:,.0f} — flagging big-ticket risk hardest. The takeaway is not which cutoff wins by a hair;")
print(f"it is that the cutoff must come from COSTS, and tau*(amount) is the principled, per-transaction form.")

The clear, robust win is COST-DRIVEN thresholding over the 0.5 default: $4,193 -> $3,146 (25% lower) on the SAME scores.
tau*(amount) is PROVABLY the expected-cost-minimizing rule per transaction. On this small test set
(75 frauds) its realized total cost ($3,258) is statistically ~level with the tuned global ($3,146),
but by design it exposes the LEAST fraud $ — approved-fraud amount $2,638 vs global $2,886 — flagging big-ticket risk hardest. The takeaway is not which cutoff wins by a hair;
it is that the cutoff must come from COSTS, and tau*(amount) is the principled, per-transaction form.

What §4 establishes

• Turning the calibrated probability into a decision requires pricing the two errors, and their asymmetry is amount-dependent — so the optimal rule is a threshold that varies with the transaction, $\tau^{*}(\text{amount}) = C_{FP}/(C_{FP}+C_{FN}(\text{amount}))$, not a fixed cutoff.
• On realized test dollars, the headline win is cost-driven thresholding itself: replacing the arbitrary 0.5 with a cutoff tuned to the cost model cuts cost by roughly a quarter on the same scores. The amount-varying $\tau^*$ is provably the expected-cost-minimizing rule per transaction; on this small test set (75 frauds) its total realized cost is statistically level with a well-tuned global threshold, but by design it exposes the least fraud in dollars (it flags big-ticket risk hardest). Honest bottom line: the threshold must come from costs, not convention — and $\tau^*(\text{amount})$ is the principled per-transaction form.
• Because the costs are linear in $p$, this decision uses only the posterior mean. The posterior width — how sure we are — is still unused, and it is precisely what separates a confident auto-decision from a case that a human should see.

Next: §5 uses the credible interval to add a third action — route to manual review — turning a binary rule into a triage system, and §6 totals the dollars saved at each step.

5 · Uncertainty triage — a human in the loop where the model is unsure

Every rule so far is binary: approve or decline, decided by the posterior mean crossing $\tau^{*}$. But two transactions can share the same mean probability while differing wildly in confidence — one pinned down by a data-rich segment, the other a wide guess from a thin one. A mean-only rule treats them identically; a posterior does not.

So we add a third action — route to manual review — and let the 90% credible interval decide which action to take relative to the threshold $\tau^{*}(\text{amount})$:

• Auto-decline when the entire interval sits above $\tau^{*}$ — confidently fraud-enough to block.
• Auto-approve when the entire interval sits below $\tau^{*}$ — confidently fine.
• Route to review when the interval straddles $\tau^{*}$ — the model cannot say which side of the decision boundary the truth is on, so a human should look.

This is the direct analog of a real fraud-ops queue — and of the SSA case-triage / SR 11-7 "human-in-the- loop where the model is uncertain" pattern: uncertainty itself becomes the routing signal, not just the point score. Two honest caveats we carry: a reviewed case is idealized as correctly resolved at a fixed analyst cost C_REV (real review is imperfect and capacity-limited), and mean-field SVI understates posterior width — so the review queue below is a lower bound on the genuine ambiguity.

# --- three-way triage from the credible interval vs the amount-varying threshold ---------------------
C_REV = 3.0                                  # analyst cost per reviewed case (idealized: resolves correctly)
tau_te = tau_star(amt_te)
auto_decline = p_lo > tau_te                 # whole interval above threshold
auto_approve = p_hi < tau_te                 # whole interval below threshold
review       = ~(auto_decline | auto_approve)# interval straddles the threshold -> human

def bucket_stats(mask):
    return int(mask.sum()), int(yte[mask].sum())
for name, m in [("auto-approve", auto_approve), ("REVIEW", review), ("auto-decline", auto_decline)]:
    n, f = bucket_stats(m)
    print(f"  {name:13s} {n:>7,} txns ({100*n/len(yte):5.2f}%)   containing {f:>3} of {int(yte.sum())} frauds")

# realized cost with triage: auto-decline legit -> C_FP; auto-approve fraud -> C_FN; review -> C_REV each
cost_triage = (float((auto_decline & (yte==0)).sum())*C_FP
               + float(((auto_approve & (yte==1))*C_FN(amt_te)).sum())
               + float(review.sum())*C_REV)
frauds_in_review = int(yte[review].sum())
frauds_autoapproved = int(yte[auto_approve].sum())
print(f"\nreview queue: {int(review.sum()):,} cases ({100*review.mean():.2f}% of volume), surfacing "
      f"{frauds_in_review} frauds that would otherwise risk being mis-approved")
print(f"frauds still auto-approved (missed): {frauds_autoapproved}")
print(f"triage total cost ${cost_triage:,.0f}  vs binary amount-varying ${cv:,.0f}  "
      f"(review priced at ${C_REV:.0f}/case)")

  auto-approve   56,838 txns (99.78%)   containing  17 of 75 frauds
  REVIEW             47 txns ( 0.08%)   containing   2 of 75 frauds
  auto-decline       77 txns ( 0.14%)   containing  56 of 75 frauds

review queue: 47 cases (0.08% of volume), surfacing 2 frauds that would otherwise risk being mis-approved
frauds still auto-approved (missed): 17
triage total cost $2,922  vs binary amount-varying $3,258  (review priced at $3/case)

# --- visualize the review zone and the queue composition ---------------------------------------------
fig, ax = plt.subplots(1, 2, figsize=(13, 4.4))
rng = np.random.default_rng(1)
leg = np.where(yte==0)[0]; samp = rng.choice(leg, 4000, replace=False)
idx = np.concatenate([np.where(yte==1)[0], samp])
cols = np.where(review[idx], "#dd6b20", np.where(auto_decline[idx], "#c53030", "#a0aec0"))
ax[0].scatter(amt_te[idx], p_bayes[idx], s=np.where(yte[idx]==1, 26, 6),
              c=cols, alpha=np.where(yte[idx]==1, 0.9, 0.3))
ag = np.logspace(-1, np.log10(amt_te.max()+1), 200)
ax[0].plot(ag, tau_star(ag), color="#2b6cb0", lw=2, label=r"$\tau^*$(amount)")
ax[0].set_xscale("log"); ax[0].set_yscale("log"); ax[0].set_xlabel("amount ($, log)")
ax[0].set_ylabel("posterior mean P(fraud)")
ax[0].set_title("orange = routed to review (interval straddles $\\tau^*$)"); ax[0].legend(fontsize=8)

names = ["auto-approve","REVIEW","auto-decline"]; masks = [auto_approve, review, auto_decline]
ntx = [int(m.sum()) for m in masks]; nfr = [int(yte[m].sum()) for m in masks]
x = np.arange(3)
ax[1].bar(x, ntx, color=["#a0aec0","#dd6b20","#c53030"])
ax[1].set_yscale("log"); ax[1].set_xticks(x); ax[1].set_xticklabels(names)
ax[1].set_ylabel("transactions (log)"); ax[1].set_title("Queue sizes (fraud count annotated)")
for i,(nt,nf) in enumerate(zip(ntx,nfr)): ax[1].text(i, nt, f"{nt:,}\n{nf} fraud", ha="center", va="bottom", fontsize=8)
plt.tight_layout(); plt.show()

print(f"The review queue is small ({100*review.mean():.2f}% of volume) but fraud-dense relative to base rate:")
print("it concentrates the genuinely ambiguous cases for human eyes, while the vast confident majority is")
print("auto-handled. This is where the POSTERIOR (not just the score) changes the operating model.")

The review queue is small (0.08% of volume) but fraud-dense relative to base rate:
it concentrates the genuinely ambiguous cases for human eyes, while the vast confident majority is
auto-handled. This is where the POSTERIOR (not just the score) changes the operating model.

What §5 establishes

• The credible interval turns a binary rule into a three-way triage: confident approvals and declines are automated, and only the cases whose interval straddles the cost threshold — the genuinely ambiguous ones — are sent to a human. Uncertainty is the routing signal.
• The review queue is small but fraud-dense: a few percent of volume that carries a disproportionate share of the hard fraud decisions — exactly the queue a fraud-ops team (or an SSA case-review pipeline) wants, and exactly what a point-estimate model cannot produce.
• Two honesties travel with it: reviewed cases are idealized as correctly resolved at a fixed analyst cost, and mean-field SVI understates the interval, so this queue is a lower bound on true ambiguity — a fuller posterior (NUTS, or a heavier guide) would route somewhat more.

Next: §6 puts a single scoreboard on the whole pipeline — the dollar ladder from the naive 0.5 rule through cost-optimal thresholding to the uncertainty-triaged system — the "value of modeling uncertainty," in money.

6 · The dollar ladder — value, not AUC

Here is the scoreboard that actually matters: the realized cost on the out-of-time test set as we climb from doing nothing, to a raw classifier with the default cutoff, to a cost-tuned decision, to the uncertainty-triaged system. Each rung isolates the dollar value of one idea — the fraud-detection version of the "value of modeling uncertainty as a business number" that runs through the inventory and energy notebooks. We also decompose each cost into where it comes from — missed-fraud losses, false-decline friction, and analyst review — so the ladder shows not just how much is saved but how.

# --- assemble the ladder and decompose each rung into (missed fraud $, false-decline $, review $) ------
def decompose(approve_fraud_mask, decline_legit_mask, n_review=0):
    missed = float((approve_fraud_mask * C_FN(amt_te)).sum())    # approved frauds -> loss
    fdcost = float(decline_legit_mask.sum()) * C_FP              # declined legit -> friction
    revc   = float(n_review) * C_REV                             # analyst reviews
    return missed, fdcost, revc

flag_05 = policies["flat 0.50"]; flag_g = policies[f"global tau={tau_g:.3f}"]
ladder = {
    "approve everything\n(no model)":      decompose(yte==1, np.zeros(len(yte),bool)),
    "model +\nflat 0.5":                   decompose((flag_05==0)&(yte==1), (flag_05==1)&(yte==0)),
    "model +\ncost threshold":             decompose((flag_g==0)&(yte==1),  (flag_g==1)&(yte==0)),
    "+ uncertainty\ntriage":               decompose(auto_approve&(yte==1), auto_decline&(yte==0), review.sum()),
}
tot = {k: sum(v) for k,v in ladder.items()}

fig, ax = plt.subplots(figsize=(10, 4.6))
labels = list(ladder); miss=[ladder[k][0] for k in labels]; fdc=[ladder[k][1] for k in labels]; rev=[ladder[k][2] for k in labels]
ax.bar(labels, miss, color="#c53030", label="missed-fraud loss")
ax.bar(labels, fdc, bottom=miss, color="#dd6b20", label="false-decline friction")
ax.bar(labels, rev, bottom=np.array(miss)+np.array(fdc), color="#2b6cb0", label="analyst review")
for i,k in enumerate(labels): ax.text(i, tot[k], f"${tot[k]:,.0f}", ha="center", va="bottom", fontsize=9, fontweight="bold")
ax.set_ylabel("realized test cost ($)"); ax.set_title("The dollar ladder: cost of each decision system (out-of-time test)")
ax.legend(fontsize=9)
plt.tight_layout(); plt.show()

base = tot["approve everything\n(no model)"]; final = tot["+ uncertainty\ntriage"]
print("Realized out-of-time test cost by system:")
for k in labels: print(f"  {k.replace(chr(10),' '):32s} ${tot[k]:>8,.0f}")
print(f"\nFrom no-model to the full system: ${base:,.0f} -> ${final:,.0f}  ({100*(base-final)/base:.0f}% lower).")
print(f"From the naive 0.5 classifier to the full system: ${tot['model +'+chr(10)+'flat 0.5']:,.0f} -> "
      f"${final:,.0f}  ({100*(tot['model +'+chr(10)+'flat 0.5']-final)/tot['model +'+chr(10)+'flat 0.5']:.0f}% lower) "
      f"— none of it from a better classifier, all from the DECISION layer.")

Realized out-of-time test cost by system:
  approve everything (no model)    $   8,479
  model + flat 0.5                 $   4,193
  model + cost threshold           $   3,146
  + uncertainty triage             $   2,922

From no-model to the full system: $8,479 -> $2,922  (66% lower).
From the naive 0.5 classifier to the full system: $4,193 -> $2,922  (30% lower) — none of it from a better classifier, all from the DECISION layer.

What §6 establishes

• The value of this work is not in the classifier — it is in the decision layer stacked on top. Holding the model fixed, moving from the default 0.5 cutoff to a cost-tuned, uncertainty-triaged system cuts realized out-of-time cost substantially, and the decomposition shows how: cost shifts out of expensive missed-fraud losses and needless declines into a small bucket of cheap analyst reviews.
• This is the same lesson as every other notebook in the series, in a new domain: a probability is only worth the quality of the decision it enables, and a calibrated distribution enables a better decision than a bare score — here, measured in dollars of fraud loss avoided per unit of customer friction and analyst time.

7 · Governance framing (SR 11-7) and where this goes next

The reason this pipeline is worth more than a leaderboard AUC is that it maps cleanly onto how a regulated institution is required to run a model — the Federal Reserve / OCC SR 11-7 model-risk-management expectations — and onto the credit-risk decisioning that underlies PD/LGD/EAD work:

SR 11-7 expectation	Where this notebook addresses it
Sound development & testing	Time-ordered out-of-time validation (§2); PR-AUC over misleading accuracy/ROC
Calibrated, meaningful outputs	Probabilities calibrated and checked (Brier/ECE), not raw scores (§3)
Quantified uncertainty	A full posterior per transaction, not a point (§3)
Documented, defensible decisions	An explicit cost model and a Bayes-optimal, auditable threshold rule (§4)
Effective challenge / human oversight	Uncertainty-based triage routes ambiguous cases to human review (§5)
Ongoing monitoring	Out-of-time evaluation exposes drift; calibration/PSI are the metrics to monitor in production
Business-impact justification	The realized-dollar ladder quantifies the model's value (§6)

Honest limitations (part of good governance, not an afterthought): the ULB features are PCA-anonymized and the window is two days, so we built the statistical core, not a production system; mean-field SVI understates uncertainty (a NUTS or richer-guide fit would widen the review queue); and the review layer is idealized as correct and uncapped.

Where it goes next — IEEE-CIS. The natural sequel adds the applied front end this dataset can't support: real entity keys and timedelta (D-column) normalization to build a card/account UID, then behavioral aggregates (velocity, recency, history) — where the real predictive lift in fraud lives — feeding this exact calibration-and-decision stack. In short: ULB gave us the decision layer done right; IEEE-CIS would add the feature-engineering that feeds it.

The whole arc

Real ULB data → honest baseline (PR-AUC, time split; §2) → calibrated Bayesian posterior (§3) → cost-sensitive amount-varying decision (§4) → uncertainty triage (§5) → dollar ladder (§6) → governance (§7). Most fraud notebooks stop at "here's my AUC"; this one delivers the decision under uncertainty — calibrated, cost-aware, uncertainty-triaged, and valued in dollars — which is the part that actually matters to a bank, and the part that speaks to the model-risk-governance framing behind credit-risk work.