Definition
An email contains "free", "winner" and "click". Each word on its own is weak evidence of spam. How can a model combine them, quickly, over millions of emails?
Naive Bayes is a family of probabilistic classifiers based on Bayes' theorem. It predicts the class with the highest posterior probability, combining the prior probability of each class with the likelihood of each feature given that class, under the "naive" assumption that features are independent of each other once the class is known.
In simple words
Imagine a judge who listens to each witness separately and multiplies how much each one makes guilt more or less likely, without asking whether the witnesses talked to each other. It is a simplification, but it is fast and often surprisingly accurate.
Overview
- Naive Bayes classifies using Bayes' theorem with the assumption that features are independent given the class.
- It multiplies the class prior by each feature's likelihood and picks the class with the highest result.
- Training is just counting or computing averages, so it is extremely fast.
- Laplace (add-one) smoothing stops unseen words from giving zero probability.
- Despite the naive assumption, it works well for text classification.
- Its probabilities are often poorly calibrated even when its class choices are good.
Why it matters
Naive Bayes trains in a single pass, needs little data, handles thousands of word features easily and gives probabilities. It is a classic, strong baseline for text classification, such as spam filtering, sentiment and topic labelling.
Key terms
- Prior
- P(class): how common each class is before looking at features.
- Likelihood
- P(feature | class): how often a feature appears in each class.
- Conditional independence
- The assumption that, given the class, features do not influence each other.
- Laplace smoothing
- Adding a small count to every feature so unseen words do not give zero probability.
- Multinomial naive Bayes
- The variant for word counts, used for text.
- Gaussian naive Bayes
- The variant for continuous features, assuming each is normally distributed within a class.
Types
Naive Bayes has several types, each assuming a different distribution for the features.
1. Gaussian naive Bayes
For continuous features assumed to follow a normal distribution within each class.
- How it works: It estimates a mean and variance per feature per class and uses the normal density as the likelihood.
- Example: Classifying iris flowers from petal and sepal measurements.
- When to use it: Use it for numeric features as a fast baseline.
2. Multinomial naive Bayes
For count features such as how many times each word appears.
- How it works: Likelihoods are word frequencies per class with additive smoothing; documents are bags of words.
- Example: Sorting support tickets into billing, delivery and technical categories.
- When to use it: Use it for text classification with word counts or TF-IDF. It trains in seconds even on millions of documents.
3. Bernoulli naive Bayes
For binary features recording whether something is present or absent.
- How it works: Each feature's probability of being 1 is learned per class, and absent features count as evidence too.
- Example: Spam detection from whether words such as "free" or "winner" appear at all.
- When to use it: Use it for short texts and yes/no features.
4. Complement naive Bayes
A variant of multinomial naive Bayes designed for imbalanced text classes.
- How it works: It estimates parameters from all classes except the one in question, which makes weights more stable.
- Example: Classifying rare complaint types that make up only 2% of messages.
- When to use it: Use it for text data with very uneven class sizes.
5. Categorical naive Bayes
For categorical features with several values each.
- How it works: It estimates the probability of each category value per class directly from counts.
- Example: Predicting whether to play cricket from outlook (sunny, overcast, rain), wind and humidity levels.
- When to use it: Use it when features are labels rather than numbers or word counts.
Explained in detail
How it combines evidence
Bayes' theorem says P(class | features) is proportional to P(features | class) × P(class). Estimating P(features | class) directly for every possible combination of thousands of words is impossible; you would never see enough emails. The naive assumption breaks it into a product: P(free | spam) × P(winner | spam) × P(click | spam) and so on. Each of these is easy to estimate by counting how often the word appears in spam emails. The model computes this score for spam and for not spam, and predicts whichever is higher.
A tiny worked example
| Spam | Not spam | |
|---|---|---|
| Prior | 0.4 | 0.6 |
| P("free" | class) | 0.30 | 0.02 |
| P("meeting" | class) | 0.01 | 0.10 |
| Score for "free meeting" | 0.4 × 0.30 × 0.01 = 0.0012 | 0.6 × 0.02 × 0.10 = 0.0012 |
Here the evidence balances exactly, so the model is unsure. Add "winner" and the spam score rises sharply. In practice, multiplying many small probabilities gives numbers too tiny for a computer to store, so implementations add logarithms of probabilities instead of multiplying them.
Smoothing
If the word "invoice" never appeared in spam during training, P(invoice | spam) would be 0, and any email containing it could never be classified as spam, however many other spammy words it had. Laplace smoothing adds 1, or a small value alpha, to every count, so unseen words get a small but non-zero probability. In scikit-learn this is the alpha parameter of MultinomialNB.
Why naive is often good enough
- The independence assumption is clearly false: "click" and "here" often appear together. Yet classification only needs the right class to score highest, not accurate probabilities, and naive Bayes often ranks classes correctly.
- Its probabilities tend to be overconfident, pushed close to 0 or 1, so do not read them as calibrated chances.
- It needs far less data than many models, which makes it valuable for small or new problems.
- Paul Graham's 2002 essay "A Plan for Spam" popularised Bayesian spam filtering, and filters based on it cut spam dramatically for early email users.
Naive Bayes also trains incrementally: new emails simply update the counts, so the filter can keep learning without retraining from scratch.
How it works
- Turn each text into word counts, for example with CountVectorizer.
- Count how often each word appears in each class, and the share of each class.
- Apply smoothing so no probability is zero.
- For a new text, add the log prior and the log likelihoods of its words for each class.
- Predict the class with the highest total.
In code
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
texts = [
"win a free prize now", "free money click here", "winner claim your free gift",
"meeting at ten tomorrow", "please review the project report", "lunch with the team today",
]
labels = ["spam", "spam", "spam", "ham", "ham", "ham"]
vectoriser = CountVectorizer()
X = vectoriser.fit_transform(texts)
model = MultinomialNB(alpha=1.0).fit(X, labels)
for message in ["free prize for the winner", "project meeting tomorrow", "free lunch with the team"]:
probs = model.predict_proba(vectoriser.transform([message]))[0]
scores = {str(c): round(float(pr), 2) for c, pr in zip(model.classes_, probs)}
print(f"{message!r}: {scores}")Output
'free prize for the winner': {'ham': 0.14, 'spam': 0.86}
'project meeting tomorrow': {'ham': 0.88, 'spam': 0.12}
'free lunch with the team': {'ham': 0.84, 'spam': 0.16}Try it yourself
How the filter weighed three words
A naive Bayes spam filter reads "Claim your free prize, winner!"
Match the variant to the data
Which naive Bayes suits each feature type?
- Word counts in documents
- Whether each word appears or not
- Height and weight measurements
Common mistakes
- Forgetting smoothing, so unseen words give zero probability.
- Trusting naive Bayes probabilities as calibrated.
- Using Gaussian naive Bayes on word counts.
Check yourself
Summary
- Naive Bayes applies Bayes' theorem with a feature-independence assumption.
- It is fast, needs little data and is a strong text baseline.
- Use smoothing and log-probabilities; do not trust its probabilities as calibrated.
Finished reading?
Sources: scikit-learn: Naive Bayes · An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani)