I8 · Natural Language Processing in Practice

Parts of Speech and Named Entities

Part-of-speech (POS) tagging labels each word with its grammatical category, such as noun, verb, adjective, pronoun or preposition, using the surrounding context. Named entity recognition (NER) finds spans of text that name real-world things and labels their type, such as PERSON, ORGANISATION, LOCATION, DATE, MONEY or PRODUCT. Both are sequence-labelling tasks: a label is predicted for every token, and modern systems use trained neural models, often transformers, while older ones used hidden Markov models and conditional random fields.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

"Sachin hit a century in Chennai on Sunday for India." A news app instantly links the story to a cricketer, a city, a day and a country. It does this by tagging each word's grammatical role and spotting the names in the sentence.

Part-of-speech (POS) tagging labels each word with its grammatical category, such as noun, verb, adjective, pronoun or preposition, using the surrounding context. Named entity recognition (NER) finds spans of text that name real-world things and labels their type, such as PERSON, ORGANISATION, LOCATION, DATE, MONEY or PRODUCT. Both are sequence-labelling tasks: a label is predicted for every token, and modern systems use trained neural models, often transformers, while older ones used hidden Markov models and conditional random fields.

In simple words

POS tagging is like the grammar exercise where you underline nouns in blue and verbs in red. NER is like using a highlighter to mark every name of a person, place, company or date in a newspaper article.

Overview

  • Part-of-speech (POS) tagging labels each word with its grammatical role, such as noun, verb or adjective.
  • Named-entity recognition (NER) finds names of people, places, organisations, dates and amounts.
  • Both are sequence-labelling tasks: one label per token.
  • The Universal Dependencies project defines 17 POS tags used across many languages.
  • NER often uses BIO tags to mark the beginning and inside of multi-word names.
  • They feed information extraction, search and question answering.

Why it matters

Entities turn unstructured text into structured data: extracting names and amounts from invoices, linking news to companies for investors, finding drug names in medical notes, or pulling dates and places from complaints. POS tags help lemmatisation, grammar checking and parsing.

Key terms

Part of speech
A word's grammatical category: noun, verb, adjective, adverb, pronoun and so on.
Named entity
A real-world object with a name, such as a person, place, organisation or date.
Sequence labelling
Predicting a label for every token in a sequence.
BIO tagging
Marking tokens as Beginning, Inside or Outside an entity, so multi-word names can be labelled.
Universal POS tags
A shared set of 17 tags, such as NOUN, VERB and ADJ, used across many languages.
Entity linking
Connecting a mention to a specific entry in a knowledge base, such as the right "Sachin".

Types

Several types of tagging and entity recognition exist. These are the main ones and the methods that perform them.

1. Universal POS tags

A standard set of 17 tags such as NOUN, VERB, ADJ, ADV, PRON, ADP and PROPN.

  • How it works: A tagger reads the sentence and assigns a tag to each word using context, since words like "book" can be noun or verb.
  • Example: "Please book a ticket": "book" is VERB; "a good book": "book" is NOUN.
  • When to use it: Use them for lemmatisation, grammar tools and as features for other tasks.

2. Standard entity types

Common NER categories: PERSON, ORGANIZATION, LOCATION (GPE), DATE, TIME, MONEY and PERCENT.

  • How it works: The model marks each span and its type, using BIO tags such as B-PER, I-PER and O.
  • Example: "Sundar Pichai visited Bengaluru on 5 March" gives PERSON, LOCATION and DATE.
  • When to use it: Use them to pull structured facts from news, emails and documents.

3. Domain-specific entities

Custom entity types for particular fields.

  • How it works: Models are trained or fine-tuned on labelled text from the domain, such as medicines, diseases, legal sections or product names.
  • Example: Extracting drug names and dosages from prescriptions; section numbers from court judgements.
  • When to use it: Use them when general categories miss what your application needs.

4. Rule-based and statistical taggers

Older methods using hand-written rules, gazetteers and probabilistic models such as HMMs and CRFs.

  • How it works: Rules and name lists catch known patterns; conditional random fields use features of neighbouring words to choose the best label sequence.
  • Example: A regular expression catching Indian phone numbers and PIN codes reliably.
  • When to use it: Use rules for fixed formats such as dates and IDs, and as a fallback when data is scarce.

5. Neural sequence labellers

BiLSTM-CRF models and fine-tuned transformers such as BERT.

  • How it works: Contextual embeddings capture each word's meaning in its sentence, giving high accuracy on POS tagging and NER.
  • Example: spaCy's transformer pipelines tagging news articles in many languages.
  • When to use it: Use them as the default for accurate tagging today. Fine-tune them on domain data when general models miss specialised names.

Explained in detail

Tagging in context

The same word can play different roles. In "I will book a ticket", "book" is a verb; in "I read a book", it is a noun. A tagger uses the surrounding words: after "will" a verb is likely, after "a" a noun is likely. Statistical taggers learned these patterns from hand-tagged corpora such as the Penn Treebank, an English corpus of about a million words released in the early 1990s. Modern neural taggers reach around 97 to 98 per cent accuracy on standard English text, but accuracy drops on social media, code-mixed text and less-resourced languages.

Named entities and BIO tags

TokenBIO tagMeaning
NarendraB-PERBeginning of a person's name
ModiI-PERInside the same name
visitedOOutside any entity
TataB-ORGBeginning of an organisation
SteelI-ORGInside the organisation name
inOOutside
JamshedpurB-LOCA location

NER is evaluated at the entity level: a prediction counts only if the whole span and its type are correct, and precision, recall and F1 are reported for each entity type. Typical challenges include ambiguous names ("Jordan" can be a person or a country), lowercase or misspelled names in chat messages, new names that never appeared in training, and Indian names that are also common words, such as "Asha" (hope) or "Kiran" (ray). Domain-specific NER, for medicine, law or finance, usually needs fine-tuning on annotated examples from that domain. Libraries such as spaCy provide ready NER models, and the IndicNER models from the AI4Bharat group cover many Indian languages.

  • Entity linking goes one step further, deciding which "Sachin" is meant by linking to a knowledge base entry such as Wikidata.
  • Extracted entities can be personal data: names, phone numbers and addresses must be handled under privacy law.

Building a custom NER model follows a familiar machine-learning loop. First define the entity types precisely, with written guidelines and examples, because annotators disagree when types are vague: is "Mumbai Indians" a place or an organisation? Then annotate a few hundred to a few thousand sentences, ideally with two annotators on a sample to measure agreement. Fine-tune a pre-trained model such as a BERT variant on the BIO-tagged data, evaluate per entity type on held-out sentences, study the errors, add examples for the weak types and repeat. Rules and gazetteers can cover very regular patterns, such as PIN codes, PAN numbers or dates, more reliably than a model.

How it works

  1. Decide which entity types matter for your task.
  2. Try a pre-trained tagger or NER model on sample text.
  3. Annotate a few hundred domain examples with BIO tags if accuracy is too low.
  4. Fine-tune and evaluate entity-level precision, recall and F1 per type.
  5. Check privacy rules before storing extracted personal information.

In code

pos_ner.py
import re

sentence = "Sachin Tendulkar scored 100 runs in Chennai on 12 March for India"
tokens = sentence.split()

PEOPLE = {("Sachin", "Tendulkar")}
PLACES = {"Chennai", "India", "Mumbai"}
MONTHS = {"January", "February", "March", "April"}
VERBS = {"scored", "hit", "visited"}
PREPS = {"in", "on", "for", "at"}

tags, i = [], 0
while i < len(tokens):
    t = tokens[i]
    if i + 1 < len(tokens) and (t, tokens[i + 1]) in PEOPLE:
        tags += [(t, "PROPN", "B-PER"), (tokens[i + 1], "PROPN", "I-PER")]
        i += 2
        continue
    if re.fullmatch(r"\d+", t) and i + 1 < len(tokens) and tokens[i + 1] in MONTHS:
        tags += [(t, "NUM", "B-DATE"), (tokens[i + 1], "PROPN", "I-DATE")]
        i += 2
        continue
    pos = "VERB" if t in VERBS else "ADP" if t in PREPS else "NUM" if t.isdigit() else "PROPN" if t[0].isupper() else "NOUN"
    tags.append((t, pos, "B-LOC" if t in PLACES else "O"))
    i += 1

for word, pos, ent in tags:
    print(f"{word:<11}{pos:<6}{ent}")
print("Entities:", [w for w, _, e in tags if e != "O"])

Output

Sachin     PROPN B-PER
Tendulkar  PROPN I-PER
scored     VERB  O
100        NUM   O
runs       NOUN  O
in         ADP   O
Chennai    PROPN B-LOC
on         ADP   O
12         NUM   B-DATE
March      PROPN I-DATE
for        ADP   O
India      PROPN B-LOC
Entities: ['Sachin', 'Tendulkar', 'Chennai', '12', 'March', 'India']

Try it yourself

Sorting complaints by place

A city helpline receives thousands of text complaints a day and must route them to the right ward office.

Match the word to its entity type

Which label fits?

  • Infosys
  • Kolkata
  • 15 August 1947
  • ₹2,500
  • Kalpana Chawla

Common mistakes

  • Counting partly correct entity spans as correct.
  • Using a news-trained NER model on medical or legal text without checking.
  • Storing extracted names and phone numbers without privacy safeguards.

Check yourself

Question 1

1. In "I will book a ticket", "book" is a…

Question 2

2. In BIO tagging, what does I-ORG mean?

Question 3

3. Why is "Asha" tricky for NER?

Summary

  • POS tagging labels each word's grammatical role using context.
  • NER finds and types names of people, places, organisations, dates and amounts, often with BIO tags.
  • Evaluate per entity type, adapt to the domain and protect personal data.

Finished reading?

Sources: Jurafsky and Martin, Speech and Language Processing (3rd ed. draft) · spaCy 101: everything you need to know