I8 · Natural Language Processing in Practice

NLP for Indian Languages

NLP for Indian languages covers building tools such as tokenisers, translators, speech recognisers, classifiers and language models for languages including Hindi, Bengali, Telugu, Marathi, Tamil, Urdu, Gujarati, Kannada, Odia, Malayalam, Punjabi, Assamese and many more. Key challenges are limited digital text for many languages, multiple scripts, rich word forms, code-mixing with English, text written in Roman script, and dialect variation. Projects such as AI4Bharat at IIT Madras and the government's Bhashini mission build open data sets and models to close the gap.

10 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

India's Constitution recognises 22 scheduled languages, and the 2011 Census recorded 121 languages spoken by 10,000 or more people each. Most AI tools were built for English first. Making NLP work for a Santhali speaker or a Tulu speaker is one of the most important challenges in Indian AI.

NLP for Indian languages covers building tools such as tokenisers, translators, speech recognisers, classifiers and language models for languages including Hindi, Bengali, Telugu, Marathi, Tamil, Urdu, Gujarati, Kannada, Odia, Malayalam, Punjabi, Assamese and many more. Key challenges are limited digital text for many languages, multiple scripts, rich word forms, code-mixing with English, text written in Roman script, and dialect variation. Projects such as AI4Bharat at IIT Madras and the government's Bhashini mission build open data sets and models to close the gap.

In simple words

Most AI tools were raised mainly on English books. To speak well with someone in Odia or Kannada, they need to read and listen to much more of those languages, written in their own scripts and also in the mixed way people actually chat.

Overview

  • India's Constitution lists 22 scheduled languages, and the 2011 census recorded hundreds of mother tongues.
  • Indian languages use many scripts, including Devanagari, Bengali, Gurmukhi, Tamil, Telugu, Kannada, Malayalam and Urdu.
  • Code-mixing (Hinglish) and romanised writing are very common online.
  • Many languages are low-resource: little labelled data exists.
  • Rich morphology and free word order make tasks harder.
  • AI4Bharat, Bhashini and IIT research groups release open models and data sets.

Why it matters

Only a small share of Indians are comfortable using English online. Government services, health information, farming advice, courts and education reach far more people when AI works in their languages. Building for Indian languages is also a major career and research opportunity.

Key terms

Low-resource language
A language with little digital text or labelled data for training models.
Script
A writing system, such as Devanagari, Bengali, Tamil, Gurmukhi or Perso-Arabic.
Transliteration
Converting text from one script to another, such as "namaste" to नमस्ते.
Code-mixing
Mixing two languages in one sentence, such as Hinglish or Tanglish.
Multilingual model
One model trained on many languages, which can transfer knowledge between them.
Unicode normalisation
Making text that looks identical use the same underlying characters.

Types

NLP for Indian languages faces several types of challenge, each with its own tools and remedies.

1. Script diversity and Unicode

Handling many writing systems with vowel signs, conjuncts and different number forms.

  • How it works: Text must be Unicode-normalised so that the same word is stored the same way; tokenisers must not split vowel signs from their consonants.
  • Example: "क्षेत्र" stored with different character sequences on different keyboards until normalised.
  • When to use it: Use Indic NLP Library tools for normalisation and script conversion.

2. Code-mixing and transliteration

Text that mixes languages, often written in the Roman script.

  • How it works: Models must handle words from two languages in one sentence and many spellings of the same word; transliteration tools such as IndicXlit convert between scripts.
  • Example: "Kal meeting cancel ho gayi, so I'm free" in a WhatsApp message.
  • When to use it: Use code-mixed training data and character-level or multilingual models.

3. Low-resource learning

Building models when labelled data is scarce.

  • How it works: Multilingual pre-trained models (IndicBERT, MuRIL, mBERT) transfer knowledge from high-resource languages; translation and synthetic data add examples.
  • Example: Fine-tuning MuRIL on 2,000 labelled Odia news headlines.
  • When to use it: Use multilingual models and community-built data sets such as IndicCorp and Samanantar. Evaluate on native-speaker test sets.

4. Morphology and word order

Languages where words change form a lot and order is flexible.

  • How it works: Dravidian languages join many suffixes into long words; Hindi marks case with postpositions; subword tokenisers and morphological analysers help.
  • Example: A single Malayalam word expressing what takes a whole English phrase.
  • When to use it: Use subword or character models, and language-specific analysers for tasks such as lemmatisation.

5. National platforms and speech for Indian languages

Government and open-source systems that provide translation, ASR and TTS in Indian languages.

  • How it works: Bhashini (launched 2022) offers APIs and data for translation and speech; AI4Bharat releases models such as IndicTrans2, IndicWhisper and Indic TTS voices.
  • Example: A government helpline answering farmers' spoken questions in their own language.
  • When to use it: Use these resources to build inclusive services that reach people in their own languages.

Explained in detail

The main challenges

ChallengeExampleCommon approach
Little training dataFar less web text in Maithili or Santhali than in EnglishCollect and release open corpora; multilingual transfer from related languages
Many scriptsHindi in Devanagari, Punjabi in Gurmukhi, Urdu in Perso-Arabic scriptScript-aware tokenisers; transliteration to a common script
Roman-script typing"aap kaise ho", "kaise ho aap", "kese ho"Transliteration models; training on romanised text
Code-mixing"Meeting cancel ho gayi, reschedule karte hain"Data sets of real mixed text; multilingual models
Rich morphologyLong agglutinative words in Tamil, Malayalam and KannadaSub-word tokenisers trained on these languages
DialectsBhojpuri, Awadhi and Haryanvi alongside standard HindiDialect-specific data collection, especially for speech

Who is building the tools

AI4Bharat, a research lab at IIT Madras, has released large open resources: IndicCorp, a multi-billion-token text corpus across many Indian languages; IndicTrans2, translation models covering all 22 scheduled languages; IndicBERT and other language models; and speech data sets and models for recognition and synthesis. The Government of India launched the Bhashini mission in 2022 under the National Language Translation Mission to make translation, speech recognition and text-to-speech available as public services and to crowdsource data through a programme called Bhasha Daan, in which citizens contribute recordings and translations. Companies and research groups have also released models with tokenisers designed for Indian scripts, and global multilingual models such as Google's MuRIL, built for 17 Indian languages, and Meta's No Language Left Behind translation model include many Indian languages.

Practical tips for builders

  • Normalise Unicode first: the same-looking word can be stored with different character sequences, which breaks matching.
  • Detect both language and script; a Hindi message may arrive in Devanagari or in Roman letters.
  • Evaluate separately for each language and dialect your users speak, not only for Hindi.
  • Work with native speakers for annotation and testing; machine translation of English data misses local idioms and culture.
  • Respect the communities whose language data you use: credit contributors and follow the licences of open data sets.

Evaluation data is as scarce as training data. Standard benchmarks such as IndicGLUE and IndicXTREME let researchers compare models across Indian languages on tasks like classification, entity recognition and question answering, and they regularly show large gaps between Hindi and lower-resource languages.

How it works

  1. List the languages, scripts and dialects of your real users.
  2. Normalise Unicode and detect language and script.
  3. Start from multilingual or Indian-language models such as IndicBERT, MuRIL or IndicTrans2.
  4. Collect and label local data with native speakers, including code-mixed and romanised text.
  5. Report results per language and fix the weakest before launch.

In code

indian_scripts.py
import unicodedata

SCRIPTS = [("Devanagari", 0x0900, 0x097F), ("Bengali", 0x0980, 0x09FF), ("Gurmukhi", 0x0A00, 0x0A7F),
           ("Gujarati", 0x0A80, 0x0AFF), ("Tamil", 0x0B80, 0x0BFF), ("Telugu", 0x0C00, 0x0C7F),
           ("Kannada", 0x0C80, 0x0CFF), ("Malayalam", 0x0D00, 0x0D7F), ("Latin", 0x0041, 0x007A)]

def script_of(text):
    counts = {}
    for ch in text:
        for name, lo, hi in SCRIPTS:
            if lo <= ord(ch) <= hi:
                counts[name] = counts.get(name, 0) + 1
    return max(counts, key=counts.get) if counts else "unknown"

samples = ["नमस्ते, आप कैसे हैं?", "வணக்கம்", "ನಮಸ್ಕಾರ", "ਸਤ ਸ੍ਰੀ ਅਕਾਲ", "kya haal hai bhai", "Meeting cancel ho gayi"]
for s in samples:
    print(f"{s:<24} script={script_of(s):<10} characters={len(s):>2}  utf-8 bytes={len(s.encode('utf-8'))}")

# The same-looking word stored two ways: precomposed vs base letter + nukta
a = "\u0958"            # क़ as one code point
b = "\u0915\u093C"      # क + nukta
print("Look the same:", a, b, "| equal before normalising:", a == b,
      "| equal after NFC:", unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))

Output

नमस्ते, आप कैसे हैं?     script=Devanagari characters=20  utf-8 bytes=50
வணக்கம்                  script=Tamil      characters= 7  utf-8 bytes=21
ನಮಸ್ಕಾರ                  script=Kannada    characters= 7  utf-8 bytes=21
ਸਤ ਸ੍ਰੀ ਅਕਾਲ             script=Gurmukhi   characters=12  utf-8 bytes=32
kya haal hai bhai        script=Latin      characters=17  utf-8 bytes=17
Meeting cancel ho gayi   script=Latin      characters=22  utf-8 bytes=22
Look the same: क़ क़ | equal before normalising: False | equal after NFC: True

Try it yourself

A health message in 12 languages

A state health department wants vaccination reminders in every major language of its districts.

Name the challenge

Which challenge does each example show?

  • "tumi kemon acho" typed for Bengali

  • Very little online text exists in Kurukh

  • "Naan office ku late ah varuven"

  • "Kal ka plan cancel, I'm too tired yaar"

Common mistakes

  • Testing only in Hindi and assuming other Indian languages work too.
  • Ignoring Roman-script and code-mixed messages that users actually type.
  • Publishing machine translations without native-speaker review.

Check yourself

Question 1

1. How many languages are in the Eighth Schedule of the Indian Constitution?

Question 2

2. What is transliteration?

Question 3

3. Why normalise Unicode in Indian-language text?

Summary

  • Indian languages bring many scripts, rich word forms, code-mixing, Roman typing and limited data.
  • AI4Bharat, Bhashini and multilingual models such as MuRIL and IndicTrans2 provide open tools and data.
  • Normalise text, test every language separately and work with native speakers.

Finished reading?

Sources: AI4Bharat: open-source AI for Indian languages · Bhashini: National Language Translation Mission