naive bayes

Naive Bayes classifies by applying Bayes' theorem — published posthumously in 1763 from Thomas Bayes' own notes — while deliberately assuming every feature is independent of every other feature given the class, an assumption that is almost never literally true and barely matters in practice anyway. The method became the backbone of the first generation of effective spam filters in the early 2000s, popularized by Paul Graham's 2002 essay "A Plan for Spam" and built directly into Mozilla Thunderbird's adaptive junk filter, precisely because email text is exactly the kind of high-dimensional, word-count-heavy data where treating "free" and "win" as independent signals — even though spam emails obviously contain both together — still ranks spam above legitimate mail correctly almost every time.

A concrete worked example shows exactly where the independence assumption gets smuggled in. Say 30% of training emails are spam, so the prior probability of spam is 0.3 and of a legitimate "ham" email is 0.7; "free" appears in 80% of spam but only 10% of ham, and "meeting" appears in just 5% of spam but 60% of ham. For a new email containing both words, naive Bayes multiplies the prior by each word's likelihood as if they were unrelated: spam scores 0.3 times 0.8 times 0.05, which comes to 0.012, while ham scores 0.7 times 0.1 times 0.6, which comes to 0.042 — ham wins, correctly, because "meeting" is such a strong ham signal that it overwhelms "free" even though the two words' true joint probability (which the model never actually computes) would have looked different.

The reason this keeps working despite an assumption that's provably false is that classification only needs the right order, not the right numbers. Multiplying independent-looking likelihoods together distorts the actual probability values — the true P(spam | both words) is almost never exactly 0.012 — but what naive Bayes needs to get right is just which class's score comes out on top, and that ranking survives the independence shortcut far more often than the underlying numbers do. This is also exactly why naive Bayes is notorious for poorly calibrated probabilities (a naive Bayes model might output 99.97% confidence on something that's really only 85% likely) while still being reliably correct about which class to predict.

Naive Bayes earns its place as the model every more complex approach has to justify beating, because it is almost free to train and surprisingly hard to outperform badly. Training requires only a single pass through the data to count word frequencies per class, it handles bag-of-words feature spaces with 10,000 or more dimensions without the curse-of-dimensionality problems that plague distance-based methods like k-nearest-neighbors, and it remains competitive on datasets with only a few hundred labeled examples where a deep model would have nowhere near enough data to train. The practical discipline this creates is simple: train a naive Bayes baseline first, and treat any more complex model as having to earn its additional complexity against that baseline's accuracy, not against zero.

The independence assumption that gives naive Bayes its name is also the cleanest illustration of a broader lesson that shows up across machine learning: a model doesn't need to describe reality correctly to be useful, it needs to make the decision reality actually calls for — and naive Bayes's entire career, from 1763's original theorem to a 2002 spam filter still recognizable in inboxes decades later, has been built on a simplification everyone studying it knows is wrong, used anyway because the thing it gets wrong (the exact probability) matters less than the thing it gets right (which class wins).