Definition
India's Constitution recognises 22 scheduled languages, and the 2011 Census recorded 121 languages spoken by 10,000 or more people each. Most AI tools were built for English first. Making NLP work for a Santhali speaker or a Tulu speaker is one of the most important challenges in Indian AI.
NLP for Indian languages covers building tools such as tokenisers, translators, speech recognisers, classifiers and language models for languages including Hindi, Bengali, Telugu, Marathi, Tamil, Urdu, Gujarati, Kannada, Odia, Malayalam, Punjabi, Assamese and many more. Key challenges are limited digital text for many languages, multiple scripts, rich word forms, code-mixing with English, text written in Roman script, and dialect variation. Projects such as AI4Bharat at IIT Madras and the government's Bhashini mission build open data sets and models to close the gap.
In simple words
Most AI tools were raised mainly on English books. To speak well with someone in Odia or Kannada, they need to read and listen to much more of those languages, written in their own scripts and also in the mixed way people actually chat.
Overview
- India's Constitution lists 22 scheduled languages, and the 2011 census recorded hundreds of mother tongues.
- Indian languages use many scripts, including Devanagari, Bengali, Gurmukhi, Tamil, Telugu, Kannada, Malayalam and Urdu.
- Code-mixing (Hinglish) and romanised writing are very common online.
- Many languages are low-resource: little labelled data exists.
- Rich morphology and free word order make tasks harder.
- AI4Bharat, Bhashini and IIT research groups release open models and data sets.
Why it matters
Only a small share of Indians are comfortable using English online. Government services, health information, farming advice, courts and education reach far more people when AI works in their languages. Building for Indian languages is also a major career and research opportunity.
Key terms
- Low-resource language
- A language with little digital text or labelled data for training models.
- Script
- A writing system, such as Devanagari, Bengali, Tamil, Gurmukhi or Perso-Arabic.
- Transliteration
- Converting text from one script to another, such as "namaste" to नमस्ते.
- Code-mixing
- Mixing two languages in one sentence, such as Hinglish or Tanglish.
- Multilingual model
- One model trained on many languages, which can transfer knowledge between them.
- Unicode normalisation
- Making text that looks identical use the same underlying characters.
Types
NLP for Indian languages faces several types of challenge, each with its own tools and remedies.
1. Script diversity and Unicode
Handling many writing systems with vowel signs, conjuncts and different number forms.
- How it works: Text must be Unicode-normalised so that the same word is stored the same way; tokenisers must not split vowel signs from their consonants.
- Example: "क्षेत्र" stored with different character sequences on different keyboards until normalised.
- When to use it: Use Indic NLP Library tools for normalisation and script conversion.
2. Code-mixing and transliteration
Text that mixes languages, often written in the Roman script.
- How it works: Models must handle words from two languages in one sentence and many spellings of the same word; transliteration tools such as IndicXlit convert between scripts.
- Example: "Kal meeting cancel ho gayi, so I'm free" in a WhatsApp message.
- When to use it: Use code-mixed training data and character-level or multilingual models.
3. Low-resource learning
Building models when labelled data is scarce.
- How it works: Multilingual pre-trained models (IndicBERT, MuRIL, mBERT) transfer knowledge from high-resource languages; translation and synthetic data add examples.
- Example: Fine-tuning MuRIL on 2,000 labelled Odia news headlines.
- When to use it: Use multilingual models and community-built data sets such as IndicCorp and Samanantar. Evaluate on native-speaker test sets.
4. Morphology and word order
Languages where words change form a lot and order is flexible.
- How it works: Dravidian languages join many suffixes into long words; Hindi marks case with postpositions; subword tokenisers and morphological analysers help.
- Example: A single Malayalam word expressing what takes a whole English phrase.
- When to use it: Use subword or character models, and language-specific analysers for tasks such as lemmatisation.
5. National platforms and speech for Indian languages
Government and open-source systems that provide translation, ASR and TTS in Indian languages.
- How it works: Bhashini (launched 2022) offers APIs and data for translation and speech; AI4Bharat releases models such as IndicTrans2, IndicWhisper and Indic TTS voices.
- Example: A government helpline answering farmers' spoken questions in their own language.
- When to use it: Use these resources to build inclusive services that reach people in their own languages.
Explained in detail
The main challenges
| Challenge | Example | Common approach |
|---|---|---|
| Little training data | Far less web text in Maithili or Santhali than in English | Collect and release open corpora; multilingual transfer from related languages |
| Many scripts | Hindi in Devanagari, Punjabi in Gurmukhi, Urdu in Perso-Arabic script | Script-aware tokenisers; transliteration to a common script |
| Roman-script typing | "aap kaise ho", "kaise ho aap", "kese ho" | Transliteration models; training on romanised text |
| Code-mixing | "Meeting cancel ho gayi, reschedule karte hain" | Data sets of real mixed text; multilingual models |
| Rich morphology | Long agglutinative words in Tamil, Malayalam and Kannada | Sub-word tokenisers trained on these languages |
| Dialects | Bhojpuri, Awadhi and Haryanvi alongside standard Hindi | Dialect-specific data collection, especially for speech |
Who is building the tools
AI4Bharat, a research lab at IIT Madras, has released large open resources: IndicCorp, a multi-billion-token text corpus across many Indian languages; IndicTrans2, translation models covering all 22 scheduled languages; IndicBERT and other language models; and speech data sets and models for recognition and synthesis. The Government of India launched the Bhashini mission in 2022 under the National Language Translation Mission to make translation, speech recognition and text-to-speech available as public services and to crowdsource data through a programme called Bhasha Daan, in which citizens contribute recordings and translations. Companies and research groups have also released models with tokenisers designed for Indian scripts, and global multilingual models such as Google's MuRIL, built for 17 Indian languages, and Meta's No Language Left Behind translation model include many Indian languages.
Practical tips for builders
- Normalise Unicode first: the same-looking word can be stored with different character sequences, which breaks matching.
- Detect both language and script; a Hindi message may arrive in Devanagari or in Roman letters.
- Evaluate separately for each language and dialect your users speak, not only for Hindi.
- Work with native speakers for annotation and testing; machine translation of English data misses local idioms and culture.
- Respect the communities whose language data you use: credit contributors and follow the licences of open data sets.
Evaluation data is as scarce as training data. Standard benchmarks such as IndicGLUE and IndicXTREME let researchers compare models across Indian languages on tasks like classification, entity recognition and question answering, and they regularly show large gaps between Hindi and lower-resource languages.
How it works
- List the languages, scripts and dialects of your real users.
- Normalise Unicode and detect language and script.
- Start from multilingual or Indian-language models such as IndicBERT, MuRIL or IndicTrans2.
- Collect and label local data with native speakers, including code-mixed and romanised text.
- Report results per language and fix the weakest before launch.
In code
import unicodedata
SCRIPTS = [("Devanagari", 0x0900, 0x097F), ("Bengali", 0x0980, 0x09FF), ("Gurmukhi", 0x0A00, 0x0A7F),
("Gujarati", 0x0A80, 0x0AFF), ("Tamil", 0x0B80, 0x0BFF), ("Telugu", 0x0C00, 0x0C7F),
("Kannada", 0x0C80, 0x0CFF), ("Malayalam", 0x0D00, 0x0D7F), ("Latin", 0x0041, 0x007A)]
def script_of(text):
counts = {}
for ch in text:
for name, lo, hi in SCRIPTS:
if lo <= ord(ch) <= hi:
counts[name] = counts.get(name, 0) + 1
return max(counts, key=counts.get) if counts else "unknown"
samples = ["नमस्ते, आप कैसे हैं?", "வணக்கம்", "ನಮಸ್ಕಾರ", "ਸਤ ਸ੍ਰੀ ਅਕਾਲ", "kya haal hai bhai", "Meeting cancel ho gayi"]
for s in samples:
print(f"{s:<24} script={script_of(s):<10} characters={len(s):>2} utf-8 bytes={len(s.encode('utf-8'))}")
# The same-looking word stored two ways: precomposed vs base letter + nukta
a = "\u0958" # क़ as one code point
b = "\u0915\u093C" # क + nukta
print("Look the same:", a, b, "| equal before normalising:", a == b,
"| equal after NFC:", unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))Output
नमस्ते, आप कैसे हैं? script=Devanagari characters=20 utf-8 bytes=50 வணக்கம் script=Tamil characters= 7 utf-8 bytes=21 ನಮಸ್ಕಾರ script=Kannada characters= 7 utf-8 bytes=21 ਸਤ ਸ੍ਰੀ ਅਕਾਲ script=Gurmukhi characters=12 utf-8 bytes=32 kya haal hai bhai script=Latin characters=17 utf-8 bytes=17 Meeting cancel ho gayi script=Latin characters=22 utf-8 bytes=22 Look the same: क़ क़ | equal before normalising: False | equal after NFC: True
Try it yourself
A health message in 12 languages
A state health department wants vaccination reminders in every major language of its districts.
Name the challenge
Which challenge does each example show?
"tumi kemon acho" typed for Bengali
Very little online text exists in Kurukh
"Naan office ku late ah varuven"
"Kal ka plan cancel, I'm too tired yaar"
Common mistakes
- Testing only in Hindi and assuming other Indian languages work too.
- Ignoring Roman-script and code-mixed messages that users actually type.
- Publishing machine translations without native-speaker review.
Check yourself
Summary
- Indian languages bring many scripts, rich word forms, code-mixing, Roman typing and limited data.
- AI4Bharat, Bhashini and multilingual models such as MuRIL and IndicTrans2 provide open tools and data.
- Normalise text, test every language separately and work with native speakers.
Finished reading?
Sources: AI4Bharat: open-source AI for Indian languages · Bhashini: National Language Translation Mission