Definition
Which words make a message spam, and how sure can you be from counting them?
Naive Bayes classifiers apply Bayes' theorem with the 'naive' assumption that features are conditionally independent given the class, so P(c | x) ∝ P(c) Πⱼ P(xⱼ | c). Gaussian Naive Bayes models continuous features with normal distributions; multinomial Naive Bayes models word counts; Bernoulli models presence or absence. Laplace smoothing adds a pseudo-count so unseen words don't give zero probability.
In simple words
Count how often each word appears in spam and in normal messages. For a new message, multiply the chances of its words under each class, times how common each class is, and pick the larger. The independence assumption is wrong ('free' and 'prize' often appear together), yet the ranking is usually right.
Why it matters
Naive Bayes trains in one pass, works with very little data, scales to huge vocabularies and is a classic baseline for text classification and spam filtering.
Key terms
- Bayes' theorem
- P(class | features) = P(features | class) × P(class) ÷ P(features).
- Prior
- P(class): how common each class is before looking at the features.
- Likelihood
- P(features | class): how probable the observed features are within a class.
- Naive independence assumption
- Treating features as independent given the class, so likelihoods multiply.
- Laplace smoothing
- Adding a small count to every word or value so unseen ones do not give zero probability.
- Gaussian, multinomial and Bernoulli NB
- Variants for continuous features, word counts and binary presence features.
Explained in detail
Bayes' theorem as a classifier
Naive Bayes chooses the class with the highest posterior probability. Instead of estimating the joint probability of all features, which would need enormous data, it assumes features are independent within each class and multiplies their individual probabilities. Consider a spam filter where 40 per cent of training emails are spam. The word "lottery" appears in 20 per cent of spam and 0.5 per cent of normal emails; "meeting" appears in 1 per cent of spam and 10 per cent of normal emails. For an email containing "lottery" but not "meeting", the spam score is 0.4 × 0.2 × 0.99 = 0.0792 and the normal score is 0.6 × 0.005 × 0.9 = 0.0027; normalising gives about 97 per cent spam.
Variants and practical details
| Variant | Feature type | Typical use |
|---|---|---|
| Multinomial NB | Word counts or TF-IDF | Spam, topic and sentiment classification |
| Bernoulli NB | Binary: word present or absent | Short texts such as SMS |
| Gaussian NB | Continuous features assumed normal within each class | Quick baselines on sensor or medical measurements |
| Complement NB | Word counts with imbalanced classes | Text where one category dominates |
Two technical points matter in practice. First, a word never seen in the spam training set would make P(word | spam) = 0 and wipe out the whole product; Laplace or additive smoothing adds 1 (or a small α) to every count to avoid this. Second, multiplying many small probabilities underflows to zero on a computer, so implementations add log-probabilities instead. The independence assumption is clearly false, since "free" and "offer" tend to appear together, but naive Bayes often ranks classes correctly even when its probabilities are badly calibrated, overconfidently pushed towards 0 or 1.
- Strengths: trains in a single pass, needs little data, scales to millions of features and is a strong text baseline.
- Weaknesses: correlated features are double-counted, and probability outputs should not be trusted without calibration.
Real-world scenarios: early email spam filters, such as those popularised after Paul Graham's 2002 essay, used naive Bayes; it still serves as a fast first filter in many systems, for routing support tickets, tagging news articles by topic and detecting the language of a short message. Because training simply counts words per class, the model can be updated instantly with each new labelled message.
How it works
- Prior: P(c) = share of training messages in class c.
- Likelihood: P(word | c) = (count of word in c + α) / (total words in c + α·V).
- Score each class with log P(c) + Σ log P(wordᵢ | c), adding logs to avoid underflow.
- Predict the class with the highest score.
The maths
- Count of word w in class c
- Vocabulary size
- Smoothing pseudo-count, often 1
In code
A multinomial Naive Bayes spam filter from scratch with Laplace smoothing, checked against scikit-learn.
import math
from collections import Counter
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
train = [("win a free prize now", "spam"), ("free money click now", "spam"), ("claim your free gift", "spam"),
("meeting at noon tomorrow", "ham"), ("lunch with the team", "ham"), ("project notes for the meeting", "ham")]
classes = ["ham", "spam"]
counts = {c: Counter() for c in classes}
for text, label in train:
counts[label].update(text.split())
vocab = {w for c in classes for w in counts[c]}
prior = {c: sum(1 for _, l in train if l == c) / len(train) for c in classes}
def scores(text, alpha=1.0):
result = {}
for c in classes:
total = sum(counts[c].values())
s = math.log(prior[c])
for word in text.split():
s += math.log((counts[c][word] + alpha) / (total + alpha * len(vocab)))
result[c] = round(s, 3)
return result
for message in ["free prize meeting", "team meeting notes", "click to win"]:
s = scores(message)
print(f"{message!r:24} log-scores {s} -> {max(s, key=s.get)}")
vec = CountVectorizer().fit([t for t, _ in train])
nb = MultinomialNB(alpha=1.0).fit(vec.transform([t for t, _ in train]), [l for _, l in train])
print("scikit-learn agrees:", list(nb.predict(vec.transform(["free prize meeting", "team meeting notes", "click to win"]))))Output
'free prize meeting' log-scores {'ham': -10.174, 'spam': -9.193} -> spam
'team meeting notes' log-scores {'ham': -8.787, 'spam': -11.272} -> ham
'click to win' log-scores {'ham': -11.272, 'spam': -9.886} -> spam
scikit-learn agrees: [np.str_('spam'), np.str_('ham'), np.str_('spam')]Try it yourself
Which variant?
Match each data type to the Naive Bayes variant that suits it.
- Word counts in emails
- Heights and weights in centimetres and kilograms
- Whether each keyword appears or not
Common mistakes
- Forgetting smoothing, so one unseen word sets a class probability to zero.
- Multiplying many small probabilities instead of adding logs, causing underflow.
- Trusting its probabilities as calibrated. The ranking is good; the numbers are often too extreme.
Check yourself
Summary
- Naive Bayes multiplies the prior by per-feature likelihoods.
- Gaussian, multinomial and Bernoulli variants suit different features.
- Use Laplace smoothing and log probabilities.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: scikit-learn: Naive Bayes · Jurafsky & Martin, Speech and Language Processing, chapter 4