Definition
When you type "kal ka weather kaisa rahega?" into a phone assistant, the words pass through a chain of steps before an answer comes back. Each step turns messy human language into something a little more structured for a computer.
Natural language processing (NLP) is the part of AI that lets computers read, understand and produce human language. A classic NLP pipeline has stages: collecting and cleaning text, splitting it into tokens, normalising words, tagging grammar and entities (syntax), working out meaning (semantics), using context across sentences (discourse and pragmatics) and finally performing a task such as classification, translation, question answering or generation. Modern transformer models learn many of these stages internally, but the stages still describe what any language system must handle.
In simple words
Understanding a sentence is like solving a puzzle in layers. First you find the pieces (words), then which pieces fit together (grammar), then what picture they make (meaning), and finally why someone showed you the puzzle at all (intention).
Overview
- Natural language processing (NLP) lets computers read, understand and generate human language.
- Classic NLP works in stages, from raw text to meaning.
- Each stage resolves a different kind of ambiguity, such as which meaning of "bank" is meant.
- Modern neural models learn many stages together end to end, but the stages still describe what must be understood.
- NLP powers search, translation, chatbots, spam filters and voice assistants.
- Indian languages add challenges such as many scripts and code-mixing.
Why it matters
Language is how people search, complain, ask for help, write laws and share knowledge. NLP powers search engines, spam filters, translation, chatbots, voice assistants, document summarisers and large language models. Knowing the stages helps you see where a system can go wrong.
Key terms
- Corpus
- A large collection of text used to study language or train models.
- Token
- A unit of text, such as a word, part of a word or punctuation mark.
- Syntax
- The grammatical structure of a sentence: which words relate to which.
- Semantics
- The meaning of words and sentences.
- Pragmatics
- Meaning that depends on context and intention, such as sarcasm or polite requests.
- Ambiguity
- When text can be understood in more than one way.
Types
NLP is traditionally divided into five levels of analysis. Each type of analysis builds on the one before.
1. Lexical analysis
Breaking text into words, sentences and tokens and normalising them.
- How it works: Tokenisation, lower-casing, stemming and lemmatisation turn raw characters into clean units.
- Example: "Running late!!" becomes the tokens ["running", "late", "!"], then the lemma "run".
- When to use it: Use it as the first step of almost every NLP pipeline. Mistakes here carry into every later stage.
2. Syntactic analysis (parsing)
Finding the grammatical structure of a sentence.
- How it works: Part-of-speech tagging labels words as nouns, verbs and so on; dependency parsing links each word to the word it modifies.
- Example: In "Asha reads books", "Asha" is the subject and "books" the object of "reads".
- When to use it: Use it for grammar checking, information extraction and question answering.
3. Semantic analysis
Working out the meaning of words and sentences.
- How it works: Word-sense disambiguation, named-entity recognition and embeddings capture meaning; semantic role labelling finds who did what to whom.
- Example: Telling whether "bank" means a river bank or a money bank from the rest of the sentence.
- When to use it: Use it for search, classification and anything that depends on meaning rather than exact words.
4. Discourse analysis
Understanding how sentences connect across a text.
- How it works: Coreference resolution links pronouns to the things they refer to; discourse relations mark cause, contrast and sequence.
- Example: In "Ravi missed the bus. He was late.", "He" refers to Ravi.
- When to use it: Use it for summarisation, chatbots with memory and reading comprehension.
5. Pragmatic analysis
Understanding what a speaker intends, beyond the literal words.
- How it works: Context, tone and world knowledge reveal requests, sarcasm and politeness.
- Example: "Can you pass the salt?" is a request, not a question about ability; "Great, another Monday!" is sarcasm.
- When to use it: Use it for dialogue systems and sentiment analysis, where intent matters most.
Explained in detail
The stages one by one
| Stage | What happens | Example |
|---|---|---|
| 1. Text acquisition and cleaning | Collect text; remove HTML, fix encoding, handle emojis and spelling | Turning a scraped review page into plain sentences |
| 2. Tokenisation | Split text into sentences and tokens | "Don't go!" → Do, n't, go, ! |
| 3. Normalisation | Lowercase, stem or lemmatise, handle numbers | "Running", "ran" → "run" |
| 4. Syntax | Tag parts of speech; parse the sentence structure | "book" as a noun or a verb |
| 5. Semantics | Find entities, word senses and relations | "Apple" the company or the fruit |
| 6. Discourse and pragmatics | Link pronouns, track the conversation, read intention | "Can you pass the salt?" is a request, not a question about ability |
| 7. Task | Classify, extract, translate, answer or generate | Labelling a review as positive |
Why language is hard for computers
Human language is full of ambiguity. "I saw the man with the telescope" can mean that you used a telescope or that the man had one. "Bank" can be a river bank or a money bank. Words change meaning with context, people use slang, sarcasm and emojis, and in India many people mix languages in one sentence, such as Hindi and English, and write Hindi in Roman letters. Spelling varies ("kya", "kyaa", "kia"), and the same idea can be said in countless ways. Early NLP systems from the 1960s, such as the ELIZA chatbot of 1966, used hand-written rules and pattern matching. From the 1990s, statistical methods learned from large corpora. Since about 2013, neural methods, word embeddings and then transformers, have become dominant.
Pipelines then and now
In a classic pipeline, each stage is a separate tool, and an error early on, such as a wrong tokenisation, flows into every later stage. Libraries such as NLTK and spaCy provide these tools in Python. A modern transformer model takes raw tokens and learns grammar, meaning and context together inside its layers, which is why it often outperforms hand-built pipelines. Even so, cleaning, tokenisation and careful evaluation remain your responsibility, and small, explainable pipelines are still useful when data is limited, speed matters or decisions must be justified.
- Natural language understanding (NLU) covers reading and interpreting; natural language generation (NLG) covers producing text.
- Speech systems add stages before and after: converting audio to text, and text back to audio.
How it works
- Define the task and the language or languages involved.
- Collect and clean representative text.
- Tokenise and normalise in a way that suits the task.
- Choose features or a pre-trained model, then train and evaluate.
- Test on tricky cases: ambiguity, sarcasm, code-mixing and spelling variants.
In code
import re
text = "Kal ka weather kaisa rahega? I'm going to Pune!! :)"
# 1. Clean: lowercase and drop emoticons
clean = re.sub(r"[:;]-?[)(]", "", text.lower())
# 2. Split into sentences, then tokens
sentences = [s.strip() for s in re.split(r"(?<=[.?!])\s+", clean) if s.strip()]
tokens = [re.findall(r"[a-z]+'[a-z]+|[a-z]+|[?!.]", s) for s in sentences]
# 3. Normalise a few common spelling variants
variants = {"kia": "kya", "kyaa": "kya", "i'm": "i am"}
print("Sentences:", sentences)
for t in tokens:
print("Tokens:", [variants.get(w, w) for w in t])
# 4. A tiny "task": spot a question about weather
is_weather_question = any("weather" in t for t in tokens) and "?" in clean
print("Weather question detected:", is_weather_question)Output
Sentences: ['kal ka weather kaisa rahega?', "i'm going to pune!!"] Tokens: ['kal', 'ka', 'weather', 'kaisa', 'rahega', '?'] Tokens: ['i am', 'going', 'to', 'pune', '!', '!'] Weather question detected: True
Try it yourself
A question's journey
A farmer asks a voice assistant: "Kal baarish hogi kya Nashik mein?"
Put the NLP stages in order
Arrange the classic pipeline.
- 1Normalise words
- 2Split into tokens
- 3Collect and clean the text
- 4Work out meaning and context
- 5Tag grammar and entities
- 6Perform the task
Common mistakes
- Assuming English-only tools will work on Hinglish or other Indian languages.
- Skipping text cleaning and getting garbage tokens.
- Ignoring ambiguity and sarcasm when evaluating.
Check yourself
Summary
- NLP lets computers read, understand and produce human language.
- The classic stages are cleaning, tokenisation, normalisation, syntax, semantics, pragmatics and the task.
- Ambiguity, context and code-mixing make language hard; transformers learn many stages internally.
Finished reading?
Sources: Jurafsky and Martin, Speech and Language Processing (3rd ed. draft) · spaCy 101: everything you need to know