Fraud Detection — Cost-Sensitive Decisions

Python · LightGBM, NumPyro, scikit-learn  ·  Data: ULB credit card fraud and IEEE-CIS (821 MB, not stored here)

The Scoreboard Lies

Fraud detection is presented as a classification problem and graded like one. It is neither. Every score has to become an action — approve, decline, or send to a human — and the three cost wildly different amounts when they are wrong. This example takes the classifier as given and asks what the decision layer is worth on its own.

Start with why the usual scoreboard is useless. In the ULB card data, 492 frauds in 284,807 transactions — 0.173% — a model that approves everything is 99.83% accurate and worthless. ROC-AUC is barely better: both a gradient booster and a plain logistic regression post 0.98. The honest measure is PR-AUC, where the same two models separate to 0.792 and 0.752 against a baseline of 0.0013, and even that is a ranking measure, not a decision.

Before a score can be priced it has to mean something. Raw boosted scores are inflated rather than probabilistic, so the notebook calibrates them and checks the result: isotonic regression cuts expected calibration error from 0.183 to 0.042 (×10⁻³). A hierarchical Bayesian calibrator does not beat it on either Brier or ECE — reported as such — and earns its place for a different reason: it attaches a credible interval to every transaction, which is what the triage step later needs and a point estimate cannot supply.

A Threshold Derived From Costs

Now the decision. Declining a good customer costs about $10 in service and churn; approving a fraud costs the amount, plus $10. Those two numbers, not a convention, fix the threshold — and the expected-cost-minimising cutoff falls out per transaction:

τ⋆(amount)=Cfalse declineCfalse decline+amount+Cfixed\tau^\star(\text{amount}) = \frac{C_{\text{false decline}}}{C_{\text{false decline}} + \text{amount} + C_{\text{fixed}}}

Because the loss scales with the amount, τ⋆\tau^\star is lower for a large charge than a small one — the model should be more suspicious of a $2,000 transaction than a $20 one at the same score. The tuned global cutoff lands at 0.278, nowhere near the 0.5 that a classifier hands you by default.

Changing only the cutoff, on identical scores, cuts realized cost from $4,193 to $3,146 — 25%. The amount-varying rule is statistically level with it on total cost on this small test set (75 frauds), but by construction it exposes the least fraud value: $2,638 approved-fraud against $2,886. On a bigger sample that is the rule to prefer, and the notebook says so rather than declaring a winner from a hair's difference.

The last rung uses the uncertainty rather than the estimate. Where the posterior interval straddles the threshold, the model does not know — so route those cases to a person. The queue is 47 transactions, 0.08% of volume, and it contains 2 frauds that would otherwise have been approved. At a review cost of $3 a case, total cost falls to $2,922.

systemrealized cost, out-of-time test
approve everything — no model$8,479
model + flat 0.5 cutoff$4,193the classifier's contribution
model + cost-derived threshold$3,146−25%, same scores
+ uncertainty triage$2,92247 cases to a human, 0.08% of volume

The whole point of the section, in one column. From no model at all to the full system is $8,479 → $2,922, a 66% reduction. But the classifier accounts for only the first step. From the naive 0.5 cutoff to the full system is $4,193 → $2,922, 30% — and none of it comes from a better classifier. It is all calibration, costs, and knowing when to defer.

Where The Features Are The Hard Part

The companion notebook takes the same decision layer to a dataset where the features are the hard part: IEEE-CIS, 590,540 transactions over 182 days with a 3.50% fraud rate. Two problems there do not exist in the anonymized ULB data. First, drift — the raw D1 field means something different across the split, its mean moving 90.8 → 109.1, because it is a relative timedelta; re-anchoring it to a fixed calendar date removes the shift, and the repaired feature ranks 2nd of 93 by SHAP.

Second, entity resolution. Cards are not identified, so behaviour cannot be accumulated. Heuristically joining card, address and anchor turns 13,553 apparent cards into 199,070 usable entities, resolves 89% of transactions, and reveals that 69% are repeat sightings — and that 46% of test transactions are on a card already seen in training. That unlocks history, and history is predictive: fraud rises from 2.0% at first sighting to 3.8% at ten or more, because compromised cards are worked in bursts.

Both notebooks close on governance, which is where a bank actually lives. Under SR 11-7 a deployed model needs its assumptions documented, its performance monitored, and its overrides auditable — and a cost-threshold decision layer is unusually well suited to that, because the threshold is derived from two stated costs rather than tuned to a metric nobody can defend to a regulator.

Notebooks

Python ULB card fraud as a decision problem: why accuracy and ROC-AUC mislead at 0.173% prevalence, isotonic and hierarchical Bayesian calibration compared honestly, a cost-derived threshold that varies with transaction amount, uncertainty triage routing 0.08% of volume to a human, and the dollar ladder from no model to the full system (Python) Open → Python IEEE-CIS: the feature-engineering front end, where a relative-timedelta field drifts across the split until it is re-anchored, heuristic entity resolution turns 13,553 cards into 199,070 usable identities, and behavioural aggregates are built leak-safely from prior transactions only — then run back through the same decision layer (Python) Open →

References