Definition
"Run", "runs", "running" and "ran" are four strings to a computer but one idea to you. If a search engine treated them as unrelated, searching for "running shoes" would miss a page about "shoes for runners".
Stemming and lemmatisation reduce words to a common base form so that different forms of a word are treated alike. Stemming chops off endings using simple rules, so "studies" may become "studi"; it is fast but can produce non-words and mistakes. Lemmatisation uses a dictionary and the word's grammatical role to return the proper base form, called the lemma, so "studies" becomes "study" and "better" becomes "good". The Porter stemmer, published by Martin Porter in 1980, is the best-known stemming algorithm.
In simple words
Stemming is like trimming every word with scissors at a fixed place: quick, but sometimes you cut off too much. Lemmatisation is like asking a grammar teacher for the dictionary form: slower, but correct.
Overview
- Stemming and lemmatisation reduce words to a base form so that variants are treated alike.
- Stemming chops off endings with rules and may produce non-words.
- Lemmatisation uses a dictionary and the part of speech to return a real word, the lemma.
- Both shrink the vocabulary and help search and bag-of-words models.
- Transformer models usually skip these steps because subword tokens handle variants.
- Indian languages with rich inflection need language-specific tools.
Why it matters
Reducing words to base forms shrinks the vocabulary, combines evidence from related words and improves search and classification on small data sets. It is especially important for languages with many word forms, such as Hindi, Tamil and Marathi.
Key terms
- Morphology
- How words are built from smaller meaningful parts, such as "un-" + "happy" + "-ness".
- Inflection
- Changes to a word for tense, number or gender, such as "walk" → "walked" or "ladka" → "ladke".
- Stem
- The result of chopping affixes by rule; it need not be a real word.
- Lemma
- The dictionary form of a word, such as "go" for "went".
- Over-stemming
- Cutting so much that unrelated words share a stem, such as "university" and "universe".
- Under-stemming
- Failing to join related forms, such as "ran" and "run".
Types
There are several types of stemmer and lemmatiser, ranging from aggressive rules to dictionary-based tools.
1. Porter stemmer
A rule-based English stemmer published by Martin Porter in 1980.
- How it works: It applies about 60 suffix-stripping rules in steps, such as removing "-ing" and "-ational".
- Example: "connection", "connected" and "connecting" all become "connect".
- When to use it: Use it for fast English search and classic text classification. Its output may not be a real word, such as "univers".
2. Snowball and Lancaster stemmers
Snowball (Porter2) is an improved, multilingual stemmer; Lancaster is a more aggressive one.
- How it works: Snowball has rules for many European languages; Lancaster cuts more and can merge unrelated words.
- Example: Lancaster reducing "maximum" to "maxim", while Snowball keeps "maximum".
- When to use it: Use Snowball as a better default; use Lancaster only when maximum shrinking is wanted.
3. Dictionary-based lemmatisation
Looking up the base form in a dictionary such as WordNet.
- How it works: The lemmatiser needs the part of speech: "better" as an adjective becomes "good", and "meeting" as a noun stays "meeting".
- Example: spaCy and NLTK's WordNet lemmatiser turning "was" into "be" and "mice" into "mouse".
- When to use it: Use it when results must be real, readable words, as in topic modelling and analytics.
4. Morphological analysers for Indian languages
Tools that split words into root and affixes using the grammar of a language.
- How it works: Hindi verbs and nouns change for gender, number and tense, and Dravidian languages add long chains of suffixes; analysers handle these rules.
- Example: "ladkiyon" analysed as root "ladki" plus a plural oblique suffix.
- When to use it: Use language-specific tools, such as those in the Indic NLP Library, rather than English stemmers.
5. Subword-based normalisation
Letting subword tokenisers and embeddings handle word variants.
- How it works: Variants share subword pieces and end up with similar embeddings, so explicit stemming is unnecessary.
- Example: BERT treating "playing" and "played" similarly through the shared piece "play".
- When to use it: Use it with modern neural models; keep stemming for classic, count-based methods.
Explained in detail
Stemming versus lemmatisation
| Word | Porter stem | Lemma |
|---|---|---|
| studies | studi | study |
| running | run | run |
| ran | ran | run (as a verb) |
| better | better | good (as an adjective) |
| meeting | meet | meeting (noun) or meet (verb) |
| universities | univers | university |
The Porter stemmer applies about 60 suffix rules in five steps, for example replacing "-ies" with "-i" and removing "-ing" when a vowel remains in the stem. It never looks words up, so it is very fast and works on unknown words, but it produces stems such as "studi" and can merge unrelated words. A lemmatiser, such as the one in spaCy or NLTK's WordNet lemmatiser, needs the part of speech: "meeting" is a noun in "the meeting was long" but a verb form in "we are meeting today". Lemmatisation is more accurate but slower and needs a dictionary for each language.
Indian languages
Hindi nouns and verbs change with gender, number, case and tense: "ladka", "ladke", "ladkon"; "khaata", "khaati", "khaaye", "khaayenge". Tamil, Telugu, Kannada and Malayalam are agglutinative: many suffixes stack onto one root, so a single word can mean "from the houses that were in the village". Without normalising these forms, a search for one form misses the others, and a classifier sees each form as a separate, rare word. Tools such as the IndicNLP library and lemmatisers built by Indian research groups handle many of these languages, though quality varies, especially for text written in Roman script.
When to use which, or neither
- Search engines and small classifiers benefit from stemming or lemmatisation because they merge evidence across word forms.
- Lemmatisation suits tasks where readable base forms matter, such as building glossaries or analysing text for reports.
- Transformer models usually skip both: their sub-word tokens and context already capture the link between "run" and "running", and the exact form carries useful information such as tense.
- Always test whether normalisation actually improves your validation score; sometimes it hurts.
A practical example is a government grievance portal searching complaints. Someone searching for "payment delayed" should also find complaints that say "payments delaying" or "delay in paying". Stemming the complaint text and the search query with the same rules makes these forms meet, so officers see all related complaints. Measuring how many relevant complaints are found before and after stemming shows whether the change actually helps.
How it works
- Tokenise and lowercase the text.
- Choose a stemmer for speed or a lemmatiser for accuracy, suited to the language.
- For lemmatisation, tag parts of speech first.
- Apply it to both training and new text.
- Compare validation results with and without normalisation.
In code
import re
def simple_stem(word):
"""A tiny rule-based stemmer in the spirit of Porter's first steps."""
for suffix, repl in [("sses", "ss"), ("ies", "i"), ("ing", ""), ("ed", ""), ("ly", ""), ("s", "")]:
if word.endswith(suffix) and len(word) - len(suffix) >= 3:
word = word[: len(word) - len(suffix)] + repl
break
return re.sub(r"(.)\1$", r"\1", word) if word.endswith(("nn", "tt", "pp")) else word
LEMMAS = {"ran": "run", "running": "run", "runs": "run", "studies": "study", "studied": "study",
"better": "good", "went": "go", "mice": "mouse", "ladke": "ladka", "ladkon": "ladka"}
words = ["running", "runs", "ran", "studies", "studied", "better", "went", "mice", "ladkon"]
print(f"{'word':<9}{'stem':<9}lemma")
for w in words:
print(f"{w:<9}{simple_stem(w):<9}{LEMMAS.get(w, w)}")
docs = ["she runs daily", "he was running late", "they ran home"]
stemmed = [{simple_stem(t) for t in d.split()} for d in docs]
lemmatised = [{LEMMAS.get(t, t) for t in d.split()} for d in docs]
print("Docs matching 'run' after stemming:", sum("run" in s for s in stemmed))
print("Docs matching 'run' after lemmatising:", sum("run" in s for s in lemmatised))Output
word stem lemma running run run runs run run ran ran run studies studi study studied studi study better better good went went go mice mice mouse ladkon ladkon ladka Docs matching 'run' after stemming: 2 Docs matching 'run' after lemmatising: 3
Try it yourself
Match the word to its lemma
What is the dictionary form?
- went
- mice
- better
- ladkon
- studies
Stemming or lemmatisation?
Which suits each situation better?
Indexing millions of web pages very quickly
Mapping "went" to "go"
A quick baseline search for a language without a lemmatiser
Building a readable word list for a report
Common mistakes
- Expecting stems to be real words.
- Lemmatising without part-of-speech information.
- Applying English stemmers to Hindi or Tamil.
Check yourself
Summary
- Stemming chops endings by rule; lemmatisation returns dictionary forms.
- Both merge word forms, helping search and small classifiers.
- Indian languages need language-specific tools; transformers usually skip both.
Finished reading?
Sources: Bird, Klein and Loper, Natural Language Processing with Python (NLTK book) · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft)