Definition
A company gets 50,000 reviews in one day. How can it tell whether people are happy?
Sentiment analysis is a natural language processing task that reads a piece of text and classifies the opinion or feeling it expresses, most often as positive, negative or neutral.
In simple words
When a friend sends you a message, you can tell from the words whether they are pleased, annoyed or just giving you a fact. Sentiment analysis teaches a computer to make the same judgement, for thousands of messages a second.
Why it matters
Organisations use it to hear large numbers of people at once: reviews of a product, feedback on a public service, replies to a helpline. It is also a clear example of where language models struggle, because feelings are often hidden in sarcasm and context.
Key terms
- Sentiment
- The opinion or feeling expressed in a piece of text.
- Polarity
- Whether a sentiment is positive, negative or neutral.
- Lexicon
- A list of words marked with the feeling they usually carry.
- Aspect
- One feature of the thing being reviewed, such as a phone's battery or camera.
- Negation
- A word such as "not" or "never" that reverses the meaning of what follows.
- Sarcasm
- Saying the opposite of what you mean, usually to mock or complain.
Explained in detail
Why anyone needs it
A small shop can read every comment its customers leave. A company with ten million customers cannot. In a single day it may receive more reviews, emails, call transcripts and social media posts than its staff could read in a year. Sentiment analysis turns that flood into something a person can act on: 82 per cent of yesterday's comments were positive, complaints about delivery doubled this week, the new packaging is disliked in the south. It is a way of listening at a scale that people cannot manage.
More than three labels
Positive, negative and neutral are only the simplest version. Some systems give a score, like stars from one to five. Some name the emotion: joy, anger, fear, sadness, surprise. The most useful version for a business is aspect-based sentiment, which works out what each opinion is about. The review "Camera is superb but the battery is terrible" is neither positive nor negative overall. It is positive about the camera and negative about the battery, and that is exactly what the phone maker needs to know.
Where the training labels come from
Supervised learning needs labelled examples, and sentiment analysis has a convenient source. Millions of people have written reviews and attached a star rating to them. The stars act as free labels: a five-star review is very probably positive, and a one-star review negative. Models trained on these learn which words and phrases go with which feeling. The weakness is that a model trained on film reviews learns the language of films. Move it to hospital feedback or farming advice and its accuracy drops, because the words that matter are different.
The same word, opposite feelings
| Word | Positive use | Negative use |
|---|---|---|
| Unpredictable | "The film's plot was unpredictable" | "The car's steering is unpredictable" |
| Long | "The battery life is long" | "The queue was long" |
| Cold | "The lassi was nice and cold" | "The food arrived cold" |
| Cheap | "Good quality and cheap" | "It looks and feels cheap" |
A list of feeling words cannot handle these. Only a model that reads the whole sentence can tell which meaning applies. Slang makes it harder still: among young speakers of English, "sick" and "wicked" are praise. Emoji often help, because a smiling or angry face is a strong clue. And much of what Indians write online mixes two languages in Roman script, which most tools were not trained on.
Who uses it
- Companies, to track opinion about products and to answer the angriest customers first.
- Governments and public services, to read feedback on schemes from very large numbers of citizens.
- News and finance organisations, to measure the mood of reports about a company or an event.
- Researchers, to study how public feeling about an issue changes over time.
Limits and proper use
Sentiment analysis is fairly reliable for the overall picture across thousands of texts, because individual errors average out. It is unreliable for any single message, especially a sarcastic, mixed or culturally specific one. It measures words, not inner feelings: a polite complaint and a furious one may score the same. Using it to judge an individual, for example to decide whether an employee or a student is "negative", goes beyond what the technology can honestly support.
How it works
Method 1: counting feeling words
The simplest method uses two lists, one of positive words (good, great, love, tasty) and one of negative words (bad, slow, broken, rude). Count how many of each appear. More positive words means a positive review. It is easy to build and easy to fool.
Method 2: learning from labelled examples
A better method is supervised learning. People label thousands of reviews as positive, negative or neutral. A model learns which patterns of words go with each label, including word order and combinations such as "not good". It then labels new reviews by itself.
| Review | Word counting says | Really | Why it is hard |
|---|---|---|---|
| The food was tasty and the service was great | Positive | Positive | Easy |
| The food was not good | Positive (sees 'good') | Negative | Negation flips the meaning |
| Great, the phone died in two days | Positive (sees 'great') | Negative | Sarcasm |
| Camera is superb but the battery is terrible | Unclear | Mixed | Two opinions in one sentence |
| The parcel arrived on Tuesday | Neutral | Neutral | A fact, with no feeling |
| Ekdum paisa vasool phone hai | Unknown words | Positive | Hinglish and slang are missing from many word lists |
Try it yourself
Three reviews go through the sentiment machine
A sweet shop wants to know how customers feel about a new mithai.
Be the sentiment model
Read each review as a person would. Which label fits?
I would not recommend this to anyone
Not bad at all
Oh wonderful, another two-hour delay
The package weighed two kilograms
Best dosa I have eaten this year
The shop opens at 9 a.m.
Common mistakes
- Thinking sentiment analysis reads minds. It only classifies the text.
- Trusting a method that only counts words. Negation and sarcasm defeat it.
- Forgetting mixed and neutral texts. Not everything is good or bad.
Check yourself
Summary
- Sentiment analysis labels text as positive, negative or neutral.
- Counting feeling words is simple; learning from labelled examples works better.
- Negation, sarcasm, mixed opinions and slang are the hard cases.
Finished reading?
Sources: AI4K12: Five Big Ideas in AI · Elements of AI (University of Helsinki)