I8 · Natural Language Processing in Practice

Module Review and Assessment

This review brings together NLP in Practice: the stages of NLP, tokenisation and cleaning, stemming and lemmatisation, parts of speech and named entities, bag of words, TF-IDF, word embeddings, sentiment analysis, text classification, intent-based chatbots, speech-to-text and text-to-speech and NLP for Indian languages, followed by a scored quiz.

15 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Could you now take raw text in any Indian language, clean and tokenise it, turn it into numbers, classify its topic or sentiment, build a chatbot around it and add a voice?

This review brings together NLP in Practice: the stages of NLP, tokenisation and cleaning, stemming and lemmatisation, parts of speech and named entities, bag of words, TF-IDF, word embeddings, sentiment analysis, text classification, intent-based chatbots, speech-to-text and text-to-speech and NLP for Indian languages, followed by a scored quiz.

In simple words

Read the recap table, try the activities, then take the quiz. Each answer comes with an explanation. If you miss one, go back to that topic.

Why it matters

Language is the most common way people interact with AI. These skills let you build useful text and voice tools and understand what large language models build on.

How it works

TopicKey idea
StagesClean, tokenise, normalise, tag, understand meaning and context, perform the task
TokensWords, sub-words or characters; keep negations and emojis for sentiment
Normalising wordsStemming chops by rule; lemmatisation returns dictionary forms
POS and NERGrammatical tags and named entities with BIO labels
RepresentationsBag of words and TF-IDF count words; embeddings capture meaning
ClassificationSentiment and topic labels; evaluate per class with macro F1
ChatbotsIntents, slots, fallbacks and human hand-over
SpeechASR measured by WER; TTS with a vocoder; consent for voices
Indian languagesScripts, code-mixing, Roman typing, Unicode normalisation and native-speaker review

Try it yourself

Match the task to the technique

Which technique fits best?

  • Find all company names in news
  • Search help articles by meaning
  • Route complaints to departments
  • Let users block a card by chat
  • Turn a voice question into text

True or false?

Decide for each statement.

  • Bag of words keeps word order

  • Lemmatisation turns "went" into "go"

  • Removing "not" can flip sentiment

  • WER can never exceed 100%

Common mistakes

  • Skipping a simple baseline before trying large models.
  • Evaluating only in English or Hindi.
  • Letting chatbots act without confirmation or human hand-over.

Check yourself

Question 1

1. Which NLP stage deals with intention and context, such as polite requests?

Question 2

2. Why do LLMs use sub-word tokens?

Question 3

3. "Studies" → "studi" is an example of…

Question 4

4. In BIO tags, "O" means…

Question 5

5. TF-IDF gives a low weight to words that…

Question 6

6. Word embeddings place similar words…

Question 7

7. A sentiment model labels all reviews positive on data that is 90% positive. Check…

Question 8

8. In a chatbot, "Pune" in "ticket to Pune" is a…

Question 9

9. Reference of 8 words, transcript with 2 errors. WER is…

Question 10

10. "aap kaise ho" typed for Hindi is an example of…

Question 11

11. In a bigram model, "i" appears 5 times as a previous word and is followed by "like" twice. What is P(like | i)?

Question 12

12. Topic models such as LDA represent each document as…

Question 13

13. Which metric is commonly used to evaluate machine translation by comparing n-gram overlap with reference translations?

Question 14

14. How does abstractive summarisation differ from extractive summarisation?

Summary

  • NLP turns text into structure: tokens, tags, entities, vectors and labels.
  • Start with simple baselines, evaluate per class and per language, and add context-aware models where needed.
  • Chatbots and voice systems need confirmations, fallbacks, consent and human hand-over.

Finished reading?

Sources: Jurafsky and Martin, Speech and Language Processing (3rd ed. draft) · AI4Bharat: open-source AI for Indian languages