Naive Bayes Classifier

Definition

The naive Bayes classifier predicts a class by applying Bayes' rule under the "naive" assumption that the attributes are conditionally independent given the class: $\Pr(C=c\mid x)\propto \Pr(C=c)\prod_j \Pr(x_j\mid C=c)$. It assigns the class with the highest posterior. Despite the independence assumption being false in most real data, it is a remarkably strong and cheap classifier — a puzzle explained by Domingos-Pazzani (1997).

Key Ideas

• Conditional-independence factorization. The joint class-conditional density is approximated as a product of per-attribute marginals, reducing estimation to one-dimensional densities (or counts) — fast to fit and robust in high dimensions.
• Calibration vs. decision. Under quadratic loss, the probability estimates are only accurate if independence holds. But under zero-one loss (misclassification rate) the classifier only needs the correct class to have the largest estimated posterior — a much weaker requirement.
• Optimality despite dependence. Domingos-Pazzani show naive Bayes can be zero-one-optimal even when attributes are strongly dependent; the region of correct classification is vastly larger than the region of accurate probability estimation. This is why crude probability estimates still yield good classifications.
• A generative baseline. As a generative model it is trivially trained (closed-form counts/means), handles missing data gracefully, and needs little data — the reason it remains a default baseline (text classification, spam filtering).

Why It Matters

• A strong, cheap baseline. Naive Bayes is a standard first classifier — fast, low-variance, and often competitive with far more complex methods, especially in high dimensions with limited data.
• The calibration–classification lesson. The result generalizes: a model can be a bad density/probability estimator yet a good classifier, because classification is an argmax robust to probability error — a caution against judging classifiers by likelihood/calibration alone.
• Part of the classifier toolkit. It sits beside discriminative methods (SVMs, logistic regression) and trees as the canonical generative classifier.

Open Questions

• When does it fail? Strong dependence can break classification when it flips the argmax; characterizing exactly which dependence structures do so is subtle.
• Probability calibration. Naive Bayes probabilities are often over-confident (pushed toward 0/1); recalibration is needed when calibrated probabilities — not just labels — matter.
• Feature engineering. Performance depends on the attribute representation and discretization of continuous features; smoothing (Laplace) is needed for unseen combinations.

Related

• Support Vector Machine
• Classification and Regression Trees (CART)
• Proper Scoring Rule
• Machine Learning in Asset Pricing
• Domingos-Pazzani (1997) On the Optimality of the Simple Bayesian Classifier under Zero-One Loss
• Pedro Domingos