A1 · Supervised Machine Learning in Depth

Naive Bayes Classifiers

Naive Bayes classifiers apply Bayes' theorem with the 'naive' assumption that features are conditionally independent given the class, so P(c | x) ∝ P(c) Πⱼ P(xⱼ | c). Gaussian Naive Bayes models continuous features with normal distributions; multinomial Naive Bayes models word counts; Bernoulli models presence or absence. Laplace smoothing adds a pseudo-count so unseen words don't give zero probability.

15 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Which words make a message spam, and how sure can you be from counting them?

Naive Bayes classifiers apply Bayes' theorem with the 'naive' assumption that features are conditionally independent given the class, so P(c | x) ∝ P(c) Πⱼ P(xⱼ | c). Gaussian Naive Bayes models continuous features with normal distributions; multinomial Naive Bayes models word counts; Bernoulli models presence or absence. Laplace smoothing adds a pseudo-count so unseen words don't give zero probability.

In simple words

Count how often each word appears in spam and in normal messages. For a new message, multiply the chances of its words under each class, times how common each class is, and pick the larger. The independence assumption is wrong ('free' and 'prize' often appear together), yet the ranking is usually right.

Why it matters

Naive Bayes trains in one pass, works with very little data, scales to huge vocabularies and is a classic baseline for text classification and spam filtering.

Key terms

Bayes' theorem
P(class | features) = P(features | class) × P(class) ÷ P(features).
Prior
P(class): how common each class is before looking at the features.
Likelihood
P(features | class): how probable the observed features are within a class.
Naive independence assumption
Treating features as independent given the class, so likelihoods multiply.
Laplace smoothing
Adding a small count to every word or value so unseen ones do not give zero probability.
Gaussian, multinomial and Bernoulli NB
Variants for continuous features, word counts and binary presence features.

Explained in detail

Bayes' theorem as a classifier

Naive Bayes chooses the class with the highest posterior probability. Instead of estimating the joint probability of all features, which would need enormous data, it assumes features are independent within each class and multiplies their individual probabilities. Consider a spam filter where 40 per cent of training emails are spam. The word "lottery" appears in 20 per cent of spam and 0.5 per cent of normal emails; "meeting" appears in 1 per cent of spam and 10 per cent of normal emails. For an email containing "lottery" but not "meeting", the spam score is 0.4 × 0.2 × 0.99 = 0.0792 and the normal score is 0.6 × 0.005 × 0.9 = 0.0027; normalising gives about 97 per cent spam.

Variants and practical details

VariantFeature typeTypical use
Multinomial NBWord counts or TF-IDFSpam, topic and sentiment classification
Bernoulli NBBinary: word present or absentShort texts such as SMS
Gaussian NBContinuous features assumed normal within each classQuick baselines on sensor or medical measurements
Complement NBWord counts with imbalanced classesText where one category dominates

Two technical points matter in practice. First, a word never seen in the spam training set would make P(word | spam) = 0 and wipe out the whole product; Laplace or additive smoothing adds 1 (or a small α) to every count to avoid this. Second, multiplying many small probabilities underflows to zero on a computer, so implementations add log-probabilities instead. The independence assumption is clearly false, since "free" and "offer" tend to appear together, but naive Bayes often ranks classes correctly even when its probabilities are badly calibrated, overconfidently pushed towards 0 or 1.

  • Strengths: trains in a single pass, needs little data, scales to millions of features and is a strong text baseline.
  • Weaknesses: correlated features are double-counted, and probability outputs should not be trusted without calibration.

Real-world scenarios: early email spam filters, such as those popularised after Paul Graham's 2002 essay, used naive Bayes; it still serves as a fast first filter in many systems, for routing support tickets, tagging news articles by topic and detecting the language of a short message. Because training simply counts words per class, the model can be updated instantly with each new labelled message.

How it works

  1. Prior: P(c) = share of training messages in class c.
  2. Likelihood: P(word | c) = (count of word in c + α) / (total words in c + α·V).
  3. Score each class with log P(c) + Σ log P(wordᵢ | c), adding logs to avoid underflow.
  4. Predict the class with the highest score.

The maths

The Naive Bayes decision rule
Laplace-smoothed word probability
Count of word w in class c
Vocabulary size
Smoothing pseudo-count, often 1

In code

A multinomial Naive Bayes spam filter from scratch with Laplace smoothing, checked against scikit-learn.

naive_bayes.py
import math
from collections import Counter
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

train = [("win a free prize now", "spam"), ("free money click now", "spam"), ("claim your free gift", "spam"),
         ("meeting at noon tomorrow", "ham"), ("lunch with the team", "ham"), ("project notes for the meeting", "ham")]
classes = ["ham", "spam"]
counts = {c: Counter() for c in classes}
for text, label in train:
    counts[label].update(text.split())
vocab = {w for c in classes for w in counts[c]}
prior = {c: sum(1 for _, l in train if l == c) / len(train) for c in classes}

def scores(text, alpha=1.0):
    result = {}
    for c in classes:
        total = sum(counts[c].values())
        s = math.log(prior[c])
        for word in text.split():
            s += math.log((counts[c][word] + alpha) / (total + alpha * len(vocab)))
        result[c] = round(s, 3)
    return result

for message in ["free prize meeting", "team meeting notes", "click to win"]:
    s = scores(message)
    print(f"{message!r:24} log-scores {s} -> {max(s, key=s.get)}")

vec = CountVectorizer().fit([t for t, _ in train])
nb = MultinomialNB(alpha=1.0).fit(vec.transform([t for t, _ in train]), [l for _, l in train])
print("scikit-learn agrees:", list(nb.predict(vec.transform(["free prize meeting", "team meeting notes", "click to win"]))))

Output

'free prize meeting'     log-scores {'ham': -10.174, 'spam': -9.193} -> spam
'team meeting notes'     log-scores {'ham': -8.787, 'spam': -11.272} -> ham
'click to win'           log-scores {'ham': -11.272, 'spam': -9.886} -> spam
scikit-learn agrees: [np.str_('spam'), np.str_('ham'), np.str_('spam')]

Try it yourself

Which variant?

Match each data type to the Naive Bayes variant that suits it.

  • Word counts in emails
  • Heights and weights in centimetres and kilograms
  • Whether each keyword appears or not

Common mistakes

  • Forgetting smoothing, so one unseen word sets a class probability to zero.
  • Multiplying many small probabilities instead of adding logs, causing underflow.
  • Trusting its probabilities as calibrated. The ranking is good; the numbers are often too extreme.

Check yourself

Question 1

1. What is 'naive' about Naive Bayes?

Question 2

2. Why add logs instead of multiplying?

Question 3

3. With α = 1, an unseen word gets probability…

Summary

  • Naive Bayes multiplies the prior by per-feature likelihoods.
  • Gaussian, multinomial and Bernoulli variants suit different features.
  • Use Laplace smoothing and log probabilities.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: scikit-learn: Naive Bayes · Jurafsky & Martin, Speech and Language Processing, chapter 4