I8 · Natural Language Processing in Practice

Stages of NLP

Natural language processing (NLP) is the part of AI that lets computers read, understand and produce human language. A classic NLP pipeline has stages: collecting and cleaning text, splitting it into tokens, normalising words, tagging grammar and entities (syntax), working out meaning (semantics), using context across sentences (discourse and pragmatics) and finally performing a task such as classification, translation, question answering or generation. Modern transformer models learn many of these stages internally, but the stages still describe what any language system must handle.

8 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

When you type "kal ka weather kaisa rahega?" into a phone assistant, the words pass through a chain of steps before an answer comes back. Each step turns messy human language into something a little more structured for a computer.

Natural language processing (NLP) is the part of AI that lets computers read, understand and produce human language. A classic NLP pipeline has stages: collecting and cleaning text, splitting it into tokens, normalising words, tagging grammar and entities (syntax), working out meaning (semantics), using context across sentences (discourse and pragmatics) and finally performing a task such as classification, translation, question answering or generation. Modern transformer models learn many of these stages internally, but the stages still describe what any language system must handle.

In simple words

Understanding a sentence is like solving a puzzle in layers. First you find the pieces (words), then which pieces fit together (grammar), then what picture they make (meaning), and finally why someone showed you the puzzle at all (intention).

Overview

  • Natural language processing (NLP) lets computers read, understand and generate human language.
  • Classic NLP works in stages, from raw text to meaning.
  • Each stage resolves a different kind of ambiguity, such as which meaning of "bank" is meant.
  • Modern neural models learn many stages together end to end, but the stages still describe what must be understood.
  • NLP powers search, translation, chatbots, spam filters and voice assistants.
  • Indian languages add challenges such as many scripts and code-mixing.

Why it matters

Language is how people search, complain, ask for help, write laws and share knowledge. NLP powers search engines, spam filters, translation, chatbots, voice assistants, document summarisers and large language models. Knowing the stages helps you see where a system can go wrong.

Key terms

Corpus
A large collection of text used to study language or train models.
Token
A unit of text, such as a word, part of a word or punctuation mark.
Syntax
The grammatical structure of a sentence: which words relate to which.
Semantics
The meaning of words and sentences.
Pragmatics
Meaning that depends on context and intention, such as sarcasm or polite requests.
Ambiguity
When text can be understood in more than one way.

Types

NLP is traditionally divided into five levels of analysis. Each type of analysis builds on the one before.

1. Lexical analysis

Breaking text into words, sentences and tokens and normalising them.

  • How it works: Tokenisation, lower-casing, stemming and lemmatisation turn raw characters into clean units.
  • Example: "Running late!!" becomes the tokens ["running", "late", "!"], then the lemma "run".
  • When to use it: Use it as the first step of almost every NLP pipeline. Mistakes here carry into every later stage.

2. Syntactic analysis (parsing)

Finding the grammatical structure of a sentence.

  • How it works: Part-of-speech tagging labels words as nouns, verbs and so on; dependency parsing links each word to the word it modifies.
  • Example: In "Asha reads books", "Asha" is the subject and "books" the object of "reads".
  • When to use it: Use it for grammar checking, information extraction and question answering.

3. Semantic analysis

Working out the meaning of words and sentences.

  • How it works: Word-sense disambiguation, named-entity recognition and embeddings capture meaning; semantic role labelling finds who did what to whom.
  • Example: Telling whether "bank" means a river bank or a money bank from the rest of the sentence.
  • When to use it: Use it for search, classification and anything that depends on meaning rather than exact words.

4. Discourse analysis

Understanding how sentences connect across a text.

  • How it works: Coreference resolution links pronouns to the things they refer to; discourse relations mark cause, contrast and sequence.
  • Example: In "Ravi missed the bus. He was late.", "He" refers to Ravi.
  • When to use it: Use it for summarisation, chatbots with memory and reading comprehension.

5. Pragmatic analysis

Understanding what a speaker intends, beyond the literal words.

  • How it works: Context, tone and world knowledge reveal requests, sarcasm and politeness.
  • Example: "Can you pass the salt?" is a request, not a question about ability; "Great, another Monday!" is sarcasm.
  • When to use it: Use it for dialogue systems and sentiment analysis, where intent matters most.

Explained in detail

The stages one by one

StageWhat happensExample
1. Text acquisition and cleaningCollect text; remove HTML, fix encoding, handle emojis and spellingTurning a scraped review page into plain sentences
2. TokenisationSplit text into sentences and tokens"Don't go!" → Do, n't, go, !
3. NormalisationLowercase, stem or lemmatise, handle numbers"Running", "ran" → "run"
4. SyntaxTag parts of speech; parse the sentence structure"book" as a noun or a verb
5. SemanticsFind entities, word senses and relations"Apple" the company or the fruit
6. Discourse and pragmaticsLink pronouns, track the conversation, read intention"Can you pass the salt?" is a request, not a question about ability
7. TaskClassify, extract, translate, answer or generateLabelling a review as positive

Why language is hard for computers

Human language is full of ambiguity. "I saw the man with the telescope" can mean that you used a telescope or that the man had one. "Bank" can be a river bank or a money bank. Words change meaning with context, people use slang, sarcasm and emojis, and in India many people mix languages in one sentence, such as Hindi and English, and write Hindi in Roman letters. Spelling varies ("kya", "kyaa", "kia"), and the same idea can be said in countless ways. Early NLP systems from the 1960s, such as the ELIZA chatbot of 1966, used hand-written rules and pattern matching. From the 1990s, statistical methods learned from large corpora. Since about 2013, neural methods, word embeddings and then transformers, have become dominant.

Pipelines then and now

In a classic pipeline, each stage is a separate tool, and an error early on, such as a wrong tokenisation, flows into every later stage. Libraries such as NLTK and spaCy provide these tools in Python. A modern transformer model takes raw tokens and learns grammar, meaning and context together inside its layers, which is why it often outperforms hand-built pipelines. Even so, cleaning, tokenisation and careful evaluation remain your responsibility, and small, explainable pipelines are still useful when data is limited, speed matters or decisions must be justified.

  • Natural language understanding (NLU) covers reading and interpreting; natural language generation (NLG) covers producing text.
  • Speech systems add stages before and after: converting audio to text, and text back to audio.

How it works

  1. Define the task and the language or languages involved.
  2. Collect and clean representative text.
  3. Tokenise and normalise in a way that suits the task.
  4. Choose features or a pre-trained model, then train and evaluate.
  5. Test on tricky cases: ambiguity, sarcasm, code-mixing and spelling variants.

In code

nlp_pipeline.py
import re

text = "Kal ka weather kaisa rahega? I'm going to Pune!! :)"

# 1. Clean: lowercase and drop emoticons
clean = re.sub(r"[:;]-?[)(]", "", text.lower())
# 2. Split into sentences, then tokens
sentences = [s.strip() for s in re.split(r"(?<=[.?!])\s+", clean) if s.strip()]
tokens = [re.findall(r"[a-z]+'[a-z]+|[a-z]+|[?!.]", s) for s in sentences]
# 3. Normalise a few common spelling variants
variants = {"kia": "kya", "kyaa": "kya", "i'm": "i am"}
print("Sentences:", sentences)
for t in tokens:
    print("Tokens:", [variants.get(w, w) for w in t])
# 4. A tiny "task": spot a question about weather
is_weather_question = any("weather" in t for t in tokens) and "?" in clean
print("Weather question detected:", is_weather_question)

Output

Sentences: ['kal ka weather kaisa rahega?', "i'm going to pune!!"]
Tokens: ['kal', 'ka', 'weather', 'kaisa', 'rahega', '?']
Tokens: ['i am', 'going', 'to', 'pune', '!', '!']
Weather question detected: True

Try it yourself

A question's journey

A farmer asks a voice assistant: "Kal baarish hogi kya Nashik mein?"

Put the NLP stages in order

Arrange the classic pipeline.

  1. 1Normalise words
  2. 2Split into tokens
  3. 3Collect and clean the text
  4. 4Work out meaning and context
  5. 5Tag grammar and entities
  6. 6Perform the task

Common mistakes

  • Assuming English-only tools will work on Hinglish or other Indian languages.
  • Skipping text cleaning and getting garbage tokens.
  • Ignoring ambiguity and sarcasm when evaluating.

Check yourself

Question 1

1. Which stage splits text into words and punctuation?

Question 2

2. "Can you open the window?" is usually a request. Understanding this is…

Question 3

3. Why is Hinglish hard for many NLP tools?

Summary

  • NLP lets computers read, understand and produce human language.
  • The classic stages are cleaning, tokenisation, normalisation, syntax, semantics, pragmatics and the task.
  • Ambiguity, context and code-mixing make language hard; transformers learn many stages internally.

Finished reading?

Sources: Jurafsky and Martin, Speech and Language Processing (3rd ed. draft) · spaCy 101: everything you need to know