Definition
TF-IDF (Term Frequency-Inverse Document Frequency) scores how important a word is to one document within a collection. A word scores high if it appears often in this document and rarely in the others.
In simple words
In a collection of cricket articles, "the" appears everywhere, so it says nothing. "Cricket" appears in all of them too. But "Dhoni" appears often in only a few, and tells you a lot about those. TF-IDF rewards the words that make a document different.
Overview
- TF-IDF weights a word by how often it appears in a document and how rare it is across documents.
- TF is term frequency; IDF is log(N ÷ number of documents containing the word).
- Common words such as "the" get low weights; distinctive words get high weights.
- Karen Spärck Jones introduced the idea of IDF in 1972.
- TF-IDF vectors are compared with cosine similarity for search and recommendations.
- It remains a strong, fast baseline for search and text classification.
Why it matters
It's a simple, fast and explainable way to turn text into numbers. It's still a strong baseline for text classification, and related ideas such as BM25 power keyword search and hybrid RAG.
Key terms
- Term frequency (TF)
- How often a word appears in a document, often divided by the document's length.
- Document frequency
- How many documents in the collection contain the word.
- Inverse document frequency (IDF)
- log(N / df): large for rare words, small for words found everywhere.
- Corpus
- The whole collection of documents.
- Stop words
- Very common words such as "the" and "is", which carry little meaning.
- Sparse vector
- A vector that is mostly zeros, as each document uses only a few of all possible words.
Types
TF-IDF has several variants, and related weighting schemes are widely used. These are the main types.
1. Raw and log-scaled term frequency
TF as the raw count, or as 1 + log(count) to dampen repeated words.
- How it works: sublinear_tf=True in TfidfVectorizer uses the log form, so a word appearing 20 times is not weighted 20 times as much.
- Example: A product review repeating "amazing" ten times does not dominate its vector.
- When to use it: Use log scaling for longer documents where words repeat a lot.
2. Smoothed inverse document frequency
IDF with 1 added to counts to avoid division by zero and zero weights.
- How it works: scikit-learn uses idf = ln((1 + N) ÷ (1 + df)) + 1 by default.
- Example: A word found in all 100 documents still gets a small positive weight.
- When to use it: Use the default smoothing unless you have a reason to change it.
3. Normalised TF-IDF vectors
TF-IDF vectors scaled to unit length (L2 norm).
- How it works: Normalisation makes the dot product equal cosine similarity and removes the effect of document length.
- Example: Finding the stored FAQ question closest to a user's question.
- When to use it: Use it for search, clustering and any comparison between documents. scikit-learn applies L2 normalisation by default.
4. BM25
A ranking function developed from TF-IDF ideas (Robertson and colleagues, 1990s) used by search engines.
- How it works: It saturates term frequency and adjusts for document length with two parameters, k₁ and b.
- Example: Elasticsearch and Lucene ranking results with BM25 by default.
- When to use it: Use it for keyword search and as the first stage of retrieval-augmented generation.
5. TF-IDF with n-grams and characters
Applying TF-IDF to word n-grams or character n-grams.
- How it works: Character n-grams (for example 3 to 5 characters) handle spelling variants and transliteration.
- Example: Matching "Bengaluru" with "Bangalore" style variations and typos in Hinglish reviews.
- When to use it: Use it for noisy, misspelled or transliterated text. Use analyzer="char_wb" in TfidfVectorizer to build character n-grams within word boundaries.
Explained in detail
Why counting words is not enough
If you represent documents by raw word counts, the most frequent words are "the", "is" and "and", which appear in every document and say nothing about its topic. TF-IDF fixes this by multiplying how often a word appears in this document (term frequency) by how rare it is across the collection (inverse document frequency). A word like "chlorophyll" that appears five times in one biology article but in few other documents gets a high score for that article; "the" gets almost zero everywhere. The idea of inverse document frequency was introduced by Karen Spärck Jones in 1972 and has been a foundation of search engines ever since.
A worked example
| Word | Count in document A | Documents containing it (of 100) | IDF = log(100 / df) | TF × IDF |
|---|---|---|---|---|
| the | 20 | 100 | 0 | 0 |
| cricket | 6 | 10 | about 2.3 | about 13.8 |
| wicket | 3 | 2 | about 3.9 | about 11.7 |
| match | 4 | 30 | about 1.2 | about 4.8 |
Using natural logarithms, "cricket" and "wicket" score highest, correctly identifying document A as being about cricket, while "the" disappears. Libraries such as scikit-learn's TfidfVectorizer add small adjustments, such as smoothing the IDF and normalising each document vector to length 1, but the idea is the same.
Uses and limits
- Search: rank documents by how well their TF-IDF vectors match the query's, often with cosine similarity.
- Classification: TF-IDF features with logistic regression or naive Bayes make strong, fast baselines for spam, sentiment and topic tasks.
- Keyword extraction: the highest-scoring words in a document summarise what it is about.
- Limit: it ignores word order and meaning, so "not good" and "good" look similar, and "car" and "automobile" are unrelated.
- Limit: vectors are very long and sparse, one dimension per vocabulary word. Word embeddings and transformers, later in this module, address both limits.
TF-IDF also works for Indian languages, but tokenisation needs care: Hindi words written in Devanagari and in Roman script are different strings, and word endings vary with grammar, so the same idea may be split across several tokens. Preprocessing, such as normalising scripts and spellings, matters even more than for English. Despite newer methods, TF-IDF remains a strong first baseline: it is fast, transparent and needs no training data beyond the documents themselves.
How it works
- TF: count how often the word appears in this document.
- IDF: check how rare the word is across all documents.
- Multiply the two.
The maths
- How many times term t appears in document d
- The total number of documents
- How many documents contain t
10 documents. "cricket" appears 3 times in document 1, and in 2 of the 10 documents. "the" appears 20 times in document 1, and in all 10. Using log base 10:
cricket: 3 × log(10/2) = 3 × 0.699 = 2.10 the: 20 × log(10/10) = 20 × 0 = 0
"the" scores zero however often it appears.
In code
scikit-learn uses a smoothed version of the formula, so a word in every document gets a small score instead of exactly zero. It still ranks far below the rare words.
from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
"the match was great and dhoni played well",
"the weather was great today",
"the budget speech was long",
]
vectorizer = TfidfVectorizer()
matrix = vectorizer.fit_transform(documents).toarray()
words = vectorizer.get_feature_names_out()
scores = sorted(zip(matrix[0], words), reverse=True)
print("Top words in document 1:", [word for score, word in scores[:3]])
print("Score of 'the':", round(matrix[0][list(words).index("the")], 3))
print("Score of 'dhoni':", round(matrix[0][list(words).index("dhoni")], 3))Output
Top words in document 1: ['well', 'played', 'match'] Score of 'the': 0.236 Score of 'dhoni': 0.399
Common mistakes
- Fitting the vectoriser on test data. Fit on the training set only.
- Using it where meaning matters more than keywords. Consider embeddings.
Check yourself
Summary
- It rewards words that are frequent here and rare elsewhere.
- tf-idf = tf × log(N/df).
- It's a strong, simple baseline that ignores meaning.
Finished reading?
Sources: Jurafsky & Martin, Speech and Language Processing, ch. 5