B4 · Domains of AI: Vision, Language, Speech and Robotics

What Is Natural Language Processing?

Natural Language Processing (NLP) is the field of AI that enables computers to read, understand and produce human language. It has two sides: understanding language (NLU) and generating it (NLG).

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

Natural Language Processing (NLP) is the field of AI that enables computers to read, understand and produce human language. It has two sides: understanding language (NLU) and generating it (NLG).

In simple words

When your phone suggests the next word, when Gmail flags a phishing mail, when Google translates Hindi to English: that's NLP. It turns messy human language into something a computer can work with, and back again.

Why it matters

Most of the world's knowledge is in text and speech. NLP powers search, translation, voice assistants, document analysis and LLMs. For India, with its many languages and its code-mixed Hinglish, it's especially important.

Key terms

Natural language
A language that people grew up speaking, such as Hindi, Tamil or English, as opposed to a language designed for computers.
Natural language processing (NLP)
The field of AI that enables computers to work with human language.
Natural language understanding
Getting meaning out of text or speech.
Natural language generation
Producing text or speech that people can understand.
Corpus
A large collection of text or speech used to train and test language systems.
Language model
A model that has learned which words are likely to follow which, and so can predict and produce text.

Explained in detail

Natural and formal languages

Computers have always been good at one kind of language: the strict languages designed for giving them instructions, in which every symbol has exactly one meaning. Human languages are different. Nobody designed Hindi or Tamil. They grew over thousands of years, they are full of exceptions, and they depend on shared knowledge and context. A sentence such as "Could you open the window?" is a question in its grammar and a request in its meaning. Teaching a machine to deal with language of this kind is the task of natural language processing.

An early promise

In January 1954 researchers in the United States gave a public demonstration in which a computer translated about sixty Russian sentences into English. The newspapers were amazed, and the researchers predicted that translation would be a solved problem within three to five years. The system knew only 250 words and six rules of grammar, and the sentences had been chosen carefully. Real translation turned out to need far more than a dictionary and rules, and the prediction missed by more than half a century.

Three ways the problem has been attacked

EraApproachWeakness
1950s to 1980sExperts wrote dictionaries and grammar rules by handLanguage has too many exceptions; the rules never ended
1990s to about 2015Programs counted patterns in millions of sentences and chose the most likely outputWorked phrase by phrase; lost the thread of long sentences
About 2015 onwardsDeep neural networks learn language from enormous collections of textNeed huge data and computing; can produce fluent errors

A widely used online translation service was launched in 2006 using the counting approach, and switched to neural networks in 2016. Users noticed the improvement immediately. In 2017 the Transformer design arrived, and by late 2022 chatbots built on it could hold a conversation on almost any subject.

How a machine holds meaning

A computer cannot work with a word directly, so each word or word piece is represented by a long list of numbers. These lists are learned from text, on a simple principle: words that appear in similar surroundings get similar lists. "Mango" and "guava" turn up in similar sentences, about eating, ripening and orchards, so their lists end up close together. "Mango" and "bicycle" end up far apart. No one tells the model that mangoes and guavas are fruit. It works out the closeness from how people use the words. This is how a search engine can find a page about "automobiles" when you typed "cars".

Understanding and generating

Language work has two sides. Understanding takes language in and extracts something from it: the topic, the feeling, the request, the names and dates mentioned. Generating goes the other way and produces language: a translation, a summary, an answer. A voice assistant does both in a few seconds, first understanding what you asked and then generating a reply.

How it works

Why language is hard

  • Ambiguity: "bank" may be a river bank or a money bank.
  • Context: "It's cold" could be weather, food or a person's manner.
  • Sarcasm: "Great, another Monday" isn't praise.
  • Variety: slang, spelling mistakes, and mixing languages.

A typical pipeline

  1. Clean the text and split it into tokens.
  2. Turn the tokens into numbers, using TF-IDF or embeddings.
  3. Train a model, or use a pre-trained one.
  4. Produce the output: a label, an answer or generated text.
TaskExample
ClassificationIs this review positive?
Named entity recognitionFind people, places and dates in text
TranslationHindi to English
SummarisationA long report into five lines
Question answeringAnswer from a document

Three eras

NLP began with hand-written rules. From the 1990s statistical methods took over, and from the 2010s neural networks did. Since 2018 Transformers have led most of the field.

Try it yourself

Inside a translation app

Follow one Hindi sentence on its way to English.

Why would an app get confused?

Each sentence is hard for a computer. Find the reason.

  • "The bat flew out of my hand."

  • "gr8 c u tmrw"

  • "I sat on the bank."

  • "Kal milte hain, ok?"

  • "Wonderful, my phone died again."

  • "Great, another Monday."

Understanding or generating?

NLP has two sides. Is each app understanding language, or producing it?

  • Writing a birthday message for you

  • Drafting a reply to a customer

  • Flagging a phishing email

  • Finding the dates in a long email

Common mistakes

  • Assuming a model trained on English works on Hindi or Hinglish. Test on your own language.
  • Removing stop words automatically with modern Transformers. They use those words for context.

Check yourself

Question 1

1. Producing text is called:

Question 2

2. Why can "bank" be hard for a computer?

Summary

  • NLP is how computers read and produce human language.
  • Ambiguity and context make it hard.
  • It moved from rules, to statistics, to neural networks and Transformers.

Finished reading?

Sources: Jurafsky & Martin, Speech and Language Processing