Definition
Shake a sentence in a bag until the words fall out in random order, then count them. You have lost the grammar but kept a surprising amount of the meaning. That simple trick powered spam filters for decades.
The bag-of-words (BoW) model represents a document as a vector of word counts over a fixed vocabulary, ignoring word order. Each position in the vector corresponds to one vocabulary word, and its value is how many times that word appears, or simply 1 or 0 for presence. Extensions include n-grams, which count short sequences such as "not good" to recover some order, and TF-IDF weighting, which down-weights words that appear in almost every document. These vectors can be fed to classifiers such as naive Bayes or logistic regression.
In simple words
Imagine describing a recipe only by the list of ingredients and how much of each, without the steps. You can still guess if it is a dessert or a curry. Bag of words describes text the same way.
Overview
- Bag of words represents a document by how often each vocabulary word appears, ignoring order.
- Each document becomes a long, mostly zero vector called a sparse vector.
- CountVectorizer in scikit-learn builds the vocabulary and the counts.
- Losing word order means "dog bites man" and "man bites dog" look the same.
- Despite its simplicity, it works well for spam filters and topic classification.
- TF-IDF weighting and n-grams are common improvements.
Why it matters
Bag of words is the simplest way to turn text into numbers that machine learning can use. It is fast, explainable and often a strong baseline: with enough data, a bag-of-words classifier can sort spam, topics or sentiment surprisingly well.
Key terms
- Vocabulary
- The list of distinct words the model counts.
- Document-term matrix
- A table with one row per document and one column per word, holding counts.
- Sparse vector
- A vector that is mostly zeros, because each document uses few of the vocabulary's words.
- n-gram
- A sequence of n consecutive tokens, such as the bigram "not good".
- Binary features
- Recording only whether a word appears (1) or not (0).
- Feature hashing
- Mapping words to a fixed number of columns with a hash function to save memory.
Types
Bag-of-words representations come in several types depending on what is counted and how.
1. Count (frequency) vectors
Vectors holding how many times each word appears.
- How it works: Build a vocabulary from training documents, then count each word per document; unseen words at prediction time are ignored.
- Example: "good food, good service" gives good = 2, food = 1, service = 1.
- When to use it: Use them with multinomial naive Bayes and as a simple baseline.
2. Binary (presence) vectors
Vectors recording only whether each word appears.
- How it works: CountVectorizer(binary=True) sets each entry to 0 or 1, so repeating a word adds nothing.
- Example: Short messages where whether "refund" appears matters more than how often.
- When to use it: Use them for short texts and with Bernoulli naive Bayes. They also reduce the effect of words repeated for emphasis.
3. Normalised frequency vectors
Counts divided by document length, so long and short documents are comparable.
- How it works: Each count becomes a proportion, or the whole vector is scaled to unit length.
- Example: Comparing word use in a 200-word review and a 2,000-word article.
- When to use it: Use them when documents vary greatly in length.
4. Bag of n-grams
Counting sequences of two or more words as well as single words.
- How it works: ngram_range=(1, 2) adds bigrams such as "not good", recovering some word order at the cost of a larger vocabulary.
- Example: Distinguishing "not good" from "good" in sentiment analysis.
- When to use it: Use it when short phrases carry meaning, especially negations.
5. Hashed bag of words
Mapping words into a fixed number of columns with a hash function.
- How it works: HashingVectorizer needs no stored vocabulary and handles new words, but different words can collide in the same column.
- Example: Classifying a live stream of millions of social-media posts.
- When to use it: Use it for very large or streaming text where storing a vocabulary is impractical.
Explained in detail
Building the vectors
Take three short reviews: "good phone", "bad phone" and "not good not bad". The vocabulary, sorted alphabetically, is bad, good, not, phone. The first review becomes [0, 1, 0, 1], the second [1, 0, 0, 1] and the third [1, 1, 2, 0]. Stacked together these rows form the document-term matrix. Real vocabularies contain tens of thousands of words, so each row is mostly zeros and is stored as a sparse matrix that records only the non-zero entries. In scikit-learn, CountVectorizer builds the vocabulary from training text and transforms any document into such a row; words not in the vocabulary are simply ignored.
Strengths and weaknesses
| Strength | Weakness |
|---|---|
| Simple, fast and easy to explain: you can list the words that pushed a decision | Ignores order: "dog bites man" equals "man bites dog" |
| Works well for topic and spam detection with enough data | No notion of meaning: "great" and "excellent" are unrelated columns |
| Needs little computation; runs on any laptop | Vocabulary grows huge; unseen words are dropped |
| Strong baseline to compare fancy models against | Negation and sarcasm are hard without n-grams |
n-grams and weighting
Adding bigrams, pairs of adjacent words, lets the model see "not good" as one feature, which captures negation. Trigrams add more context but multiply the vocabulary size, so they need more data. Very common words such as "the" dominate raw counts; TF-IDF fixes this by multiplying each count by a weight that is high for words concentrated in a few documents and low for words that appear everywhere. Limiting the vocabulary, for example to words appearing in at least two documents, removes typos and reduces overfitting. Paul Graham's 2002 essay "A Plan for Spam" popularised naive Bayes on word counts for spam filtering, and similar methods still run inside many email systems.
Bag of words also powers simple document search. Each document and the query become count vectors over the same vocabulary, and documents are ranked by how many query words they contain, usually weighted with TF-IDF or the related BM25 formula. This keyword search is fast, transparent and still used inside many search engines, often combined with embedding-based search that matches meaning. Its weakness is vocabulary mismatch: a query for "car" misses a page that only says "automobile", which is exactly the gap that word embeddings, the next topic, address. Knowing both lets you build hybrid search that uses each where it is strongest.
How it works
- Clean and tokenise the training texts.
- Build the vocabulary from training data only.
- Convert each document into a count or binary vector, optionally with bigrams.
- Train a classifier such as naive Bayes or logistic regression.
- Inspect the most influential words to check that the model makes sense.
In code
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
docs = ["good phone", "bad phone", "not good not bad"]
vec = CountVectorizer()
X = vec.fit_transform(docs)
print("Vocabulary:", list(vec.get_feature_names_out()))
print("Document-term matrix:\n", X.toarray())
train = ["great battery and great camera", "loved the screen", "battery is good", "camera is excellent",
"battery died fast", "screen cracked in a week", "not good at all", "terrible camera"]
labels = ["pos", "pos", "pos", "pos", "neg", "neg", "neg", "neg"]
for ngrams in [(1, 1), (1, 2)]:
v = CountVectorizer(ngram_range=ngrams)
model = MultinomialNB().fit(v.fit_transform(train), labels)
test = ["not good battery", "good camera"]
print(f"ngram_range={ngrams} vocabulary={len(v.vocabulary_):2d} predictions={[str(p) for p in model.predict(v.transform(test))]}")Output
Vocabulary: ['bad', 'good', 'not', 'phone'] Document-term matrix: [[0 1 0 1] [1 0 0 1] [1 1 2 0]] ngram_range=(1, 1) vocabulary=19 predictions=['neg', 'pos'] ngram_range=(1, 2) vocabulary=38 predictions=['neg', 'pos']
Try it yourself
The village post office spam filter
A rural co-operative bank's SMS inbox is flooded with fake prize messages.
Can bag of words tell these apart?
Using single-word counts only.
"dog bites man" vs "man bites dog"
"great movie" vs "boring movie"
"cricket match today" vs "budget speech today"
"not bad, good" vs "not good, bad"
Common mistakes
- Building the vocabulary on the test set as well as training.
- Ignoring negation that single words cannot capture.
- Keeping every rare typo in the vocabulary.
Check yourself
Summary
- Bag of words turns a document into word counts over a vocabulary, ignoring order.
- n-grams recover short phrases; TF-IDF down-weights common words.
- It is fast, explainable and a strong baseline, but has no sense of meaning or order.
Finished reading?
Sources: scikit-learn: text feature extraction · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft)