Definition
"Sachin hit a century in Chennai on Sunday for India." A news app instantly links the story to a cricketer, a city, a day and a country. It does this by tagging each word's grammatical role and spotting the names in the sentence.
Part-of-speech (POS) tagging labels each word with its grammatical category, such as noun, verb, adjective, pronoun or preposition, using the surrounding context. Named entity recognition (NER) finds spans of text that name real-world things and labels their type, such as PERSON, ORGANISATION, LOCATION, DATE, MONEY or PRODUCT. Both are sequence-labelling tasks: a label is predicted for every token, and modern systems use trained neural models, often transformers, while older ones used hidden Markov models and conditional random fields.
In simple words
POS tagging is like the grammar exercise where you underline nouns in blue and verbs in red. NER is like using a highlighter to mark every name of a person, place, company or date in a newspaper article.
Overview
- Part-of-speech (POS) tagging labels each word with its grammatical role, such as noun, verb or adjective.
- Named-entity recognition (NER) finds names of people, places, organisations, dates and amounts.
- Both are sequence-labelling tasks: one label per token.
- The Universal Dependencies project defines 17 POS tags used across many languages.
- NER often uses BIO tags to mark the beginning and inside of multi-word names.
- They feed information extraction, search and question answering.
Why it matters
Entities turn unstructured text into structured data: extracting names and amounts from invoices, linking news to companies for investors, finding drug names in medical notes, or pulling dates and places from complaints. POS tags help lemmatisation, grammar checking and parsing.
Key terms
- Part of speech
- A word's grammatical category: noun, verb, adjective, adverb, pronoun and so on.
- Named entity
- A real-world object with a name, such as a person, place, organisation or date.
- Sequence labelling
- Predicting a label for every token in a sequence.
- BIO tagging
- Marking tokens as Beginning, Inside or Outside an entity, so multi-word names can be labelled.
- Universal POS tags
- A shared set of 17 tags, such as NOUN, VERB and ADJ, used across many languages.
- Entity linking
- Connecting a mention to a specific entry in a knowledge base, such as the right "Sachin".
Types
Several types of tagging and entity recognition exist. These are the main ones and the methods that perform them.
1. Universal POS tags
A standard set of 17 tags such as NOUN, VERB, ADJ, ADV, PRON, ADP and PROPN.
- How it works: A tagger reads the sentence and assigns a tag to each word using context, since words like "book" can be noun or verb.
- Example: "Please book a ticket": "book" is VERB; "a good book": "book" is NOUN.
- When to use it: Use them for lemmatisation, grammar tools and as features for other tasks.
2. Standard entity types
Common NER categories: PERSON, ORGANIZATION, LOCATION (GPE), DATE, TIME, MONEY and PERCENT.
- How it works: The model marks each span and its type, using BIO tags such as B-PER, I-PER and O.
- Example: "Sundar Pichai visited Bengaluru on 5 March" gives PERSON, LOCATION and DATE.
- When to use it: Use them to pull structured facts from news, emails and documents.
3. Domain-specific entities
Custom entity types for particular fields.
- How it works: Models are trained or fine-tuned on labelled text from the domain, such as medicines, diseases, legal sections or product names.
- Example: Extracting drug names and dosages from prescriptions; section numbers from court judgements.
- When to use it: Use them when general categories miss what your application needs.
4. Rule-based and statistical taggers
Older methods using hand-written rules, gazetteers and probabilistic models such as HMMs and CRFs.
- How it works: Rules and name lists catch known patterns; conditional random fields use features of neighbouring words to choose the best label sequence.
- Example: A regular expression catching Indian phone numbers and PIN codes reliably.
- When to use it: Use rules for fixed formats such as dates and IDs, and as a fallback when data is scarce.
5. Neural sequence labellers
BiLSTM-CRF models and fine-tuned transformers such as BERT.
- How it works: Contextual embeddings capture each word's meaning in its sentence, giving high accuracy on POS tagging and NER.
- Example: spaCy's transformer pipelines tagging news articles in many languages.
- When to use it: Use them as the default for accurate tagging today. Fine-tune them on domain data when general models miss specialised names.
Explained in detail
Tagging in context
The same word can play different roles. In "I will book a ticket", "book" is a verb; in "I read a book", it is a noun. A tagger uses the surrounding words: after "will" a verb is likely, after "a" a noun is likely. Statistical taggers learned these patterns from hand-tagged corpora such as the Penn Treebank, an English corpus of about a million words released in the early 1990s. Modern neural taggers reach around 97 to 98 per cent accuracy on standard English text, but accuracy drops on social media, code-mixed text and less-resourced languages.
Named entities and BIO tags
| Token | BIO tag | Meaning |
|---|---|---|
| Narendra | B-PER | Beginning of a person's name |
| Modi | I-PER | Inside the same name |
| visited | O | Outside any entity |
| Tata | B-ORG | Beginning of an organisation |
| Steel | I-ORG | Inside the organisation name |
| in | O | Outside |
| Jamshedpur | B-LOC | A location |
NER is evaluated at the entity level: a prediction counts only if the whole span and its type are correct, and precision, recall and F1 are reported for each entity type. Typical challenges include ambiguous names ("Jordan" can be a person or a country), lowercase or misspelled names in chat messages, new names that never appeared in training, and Indian names that are also common words, such as "Asha" (hope) or "Kiran" (ray). Domain-specific NER, for medicine, law or finance, usually needs fine-tuning on annotated examples from that domain. Libraries such as spaCy provide ready NER models, and the IndicNER models from the AI4Bharat group cover many Indian languages.
- Entity linking goes one step further, deciding which "Sachin" is meant by linking to a knowledge base entry such as Wikidata.
- Extracted entities can be personal data: names, phone numbers and addresses must be handled under privacy law.
Building a custom NER model follows a familiar machine-learning loop. First define the entity types precisely, with written guidelines and examples, because annotators disagree when types are vague: is "Mumbai Indians" a place or an organisation? Then annotate a few hundred to a few thousand sentences, ideally with two annotators on a sample to measure agreement. Fine-tune a pre-trained model such as a BERT variant on the BIO-tagged data, evaluate per entity type on held-out sentences, study the errors, add examples for the weak types and repeat. Rules and gazetteers can cover very regular patterns, such as PIN codes, PAN numbers or dates, more reliably than a model.
How it works
- Decide which entity types matter for your task.
- Try a pre-trained tagger or NER model on sample text.
- Annotate a few hundred domain examples with BIO tags if accuracy is too low.
- Fine-tune and evaluate entity-level precision, recall and F1 per type.
- Check privacy rules before storing extracted personal information.
In code
import re
sentence = "Sachin Tendulkar scored 100 runs in Chennai on 12 March for India"
tokens = sentence.split()
PEOPLE = {("Sachin", "Tendulkar")}
PLACES = {"Chennai", "India", "Mumbai"}
MONTHS = {"January", "February", "March", "April"}
VERBS = {"scored", "hit", "visited"}
PREPS = {"in", "on", "for", "at"}
tags, i = [], 0
while i < len(tokens):
t = tokens[i]
if i + 1 < len(tokens) and (t, tokens[i + 1]) in PEOPLE:
tags += [(t, "PROPN", "B-PER"), (tokens[i + 1], "PROPN", "I-PER")]
i += 2
continue
if re.fullmatch(r"\d+", t) and i + 1 < len(tokens) and tokens[i + 1] in MONTHS:
tags += [(t, "NUM", "B-DATE"), (tokens[i + 1], "PROPN", "I-DATE")]
i += 2
continue
pos = "VERB" if t in VERBS else "ADP" if t in PREPS else "NUM" if t.isdigit() else "PROPN" if t[0].isupper() else "NOUN"
tags.append((t, pos, "B-LOC" if t in PLACES else "O"))
i += 1
for word, pos, ent in tags:
print(f"{word:<11}{pos:<6}{ent}")
print("Entities:", [w for w, _, e in tags if e != "O"])Output
Sachin PROPN B-PER Tendulkar PROPN I-PER scored VERB O 100 NUM O runs NOUN O in ADP O Chennai PROPN B-LOC on ADP O 12 NUM B-DATE March PROPN I-DATE for ADP O India PROPN B-LOC Entities: ['Sachin', 'Tendulkar', 'Chennai', '12', 'March', 'India']
Try it yourself
Sorting complaints by place
A city helpline receives thousands of text complaints a day and must route them to the right ward office.
Match the word to its entity type
Which label fits?
- Infosys
- Kolkata
- 15 August 1947
- ₹2,500
- Kalpana Chawla
Common mistakes
- Counting partly correct entity spans as correct.
- Using a news-trained NER model on medical or legal text without checking.
- Storing extracted names and phone numbers without privacy safeguards.
Check yourself
Summary
- POS tagging labels each word's grammatical role using context.
- NER finds and types names of people, places, organisations, dates and amounts, often with BIO tags.
- Evaluate per entity type, adapt to the domain and protect personal data.
Finished reading?
Sources: Jurafsky and Martin, Speech and Language Processing (3rd ed. draft) · spaCy 101: everything you need to know