B4 · Domains of AI: Vision, Language, Speech and Robotics

Sentiment Analysis

Sentiment analysis is a natural language processing task that reads a piece of text and classifies the opinion or feeling it expresses, most often as positive, negative or neutral.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A company gets 50,000 reviews in one day. How can it tell whether people are happy?

Sentiment analysis is a natural language processing task that reads a piece of text and classifies the opinion or feeling it expresses, most often as positive, negative or neutral.

In simple words

When a friend sends you a message, you can tell from the words whether they are pleased, annoyed or just giving you a fact. Sentiment analysis teaches a computer to make the same judgement, for thousands of messages a second.

Why it matters

Organisations use it to hear large numbers of people at once: reviews of a product, feedback on a public service, replies to a helpline. It is also a clear example of where language models struggle, because feelings are often hidden in sarcasm and context.

Key terms

Sentiment
The opinion or feeling expressed in a piece of text.
Polarity
Whether a sentiment is positive, negative or neutral.
Lexicon
A list of words marked with the feeling they usually carry.
Aspect
One feature of the thing being reviewed, such as a phone's battery or camera.
Negation
A word such as "not" or "never" that reverses the meaning of what follows.
Sarcasm
Saying the opposite of what you mean, usually to mock or complain.

Explained in detail

Why anyone needs it

A small shop can read every comment its customers leave. A company with ten million customers cannot. In a single day it may receive more reviews, emails, call transcripts and social media posts than its staff could read in a year. Sentiment analysis turns that flood into something a person can act on: 82 per cent of yesterday's comments were positive, complaints about delivery doubled this week, the new packaging is disliked in the south. It is a way of listening at a scale that people cannot manage.

More than three labels

Positive, negative and neutral are only the simplest version. Some systems give a score, like stars from one to five. Some name the emotion: joy, anger, fear, sadness, surprise. The most useful version for a business is aspect-based sentiment, which works out what each opinion is about. The review "Camera is superb but the battery is terrible" is neither positive nor negative overall. It is positive about the camera and negative about the battery, and that is exactly what the phone maker needs to know.

Where the training labels come from

Supervised learning needs labelled examples, and sentiment analysis has a convenient source. Millions of people have written reviews and attached a star rating to them. The stars act as free labels: a five-star review is very probably positive, and a one-star review negative. Models trained on these learn which words and phrases go with which feeling. The weakness is that a model trained on film reviews learns the language of films. Move it to hospital feedback or farming advice and its accuracy drops, because the words that matter are different.

The same word, opposite feelings

WordPositive useNegative use
Unpredictable"The film's plot was unpredictable""The car's steering is unpredictable"
Long"The battery life is long""The queue was long"
Cold"The lassi was nice and cold""The food arrived cold"
Cheap"Good quality and cheap""It looks and feels cheap"

A list of feeling words cannot handle these. Only a model that reads the whole sentence can tell which meaning applies. Slang makes it harder still: among young speakers of English, "sick" and "wicked" are praise. Emoji often help, because a smiling or angry face is a strong clue. And much of what Indians write online mixes two languages in Roman script, which most tools were not trained on.

Who uses it

  • Companies, to track opinion about products and to answer the angriest customers first.
  • Governments and public services, to read feedback on schemes from very large numbers of citizens.
  • News and finance organisations, to measure the mood of reports about a company or an event.
  • Researchers, to study how public feeling about an issue changes over time.

Limits and proper use

Sentiment analysis is fairly reliable for the overall picture across thousands of texts, because individual errors average out. It is unreliable for any single message, especially a sarcastic, mixed or culturally specific one. It measures words, not inner feelings: a polite complaint and a furious one may score the same. Using it to judge an individual, for example to decide whether an employee or a student is "negative", goes beyond what the technology can honestly support.

How it works

Method 1: counting feeling words

The simplest method uses two lists, one of positive words (good, great, love, tasty) and one of negative words (bad, slow, broken, rude). Count how many of each appear. More positive words means a positive review. It is easy to build and easy to fool.

Method 2: learning from labelled examples

A better method is supervised learning. People label thousands of reviews as positive, negative or neutral. A model learns which patterns of words go with each label, including word order and combinations such as "not good". It then labels new reviews by itself.

ReviewWord counting saysReallyWhy it is hard
The food was tasty and the service was greatPositivePositiveEasy
The food was not goodPositive (sees 'good')NegativeNegation flips the meaning
Great, the phone died in two daysPositive (sees 'great')NegativeSarcasm
Camera is superb but the battery is terribleUnclearMixedTwo opinions in one sentence
The parcel arrived on TuesdayNeutralNeutralA fact, with no feeling
Ekdum paisa vasool phone haiUnknown wordsPositiveHinglish and slang are missing from many word lists

Try it yourself

Three reviews go through the sentiment machine

A sweet shop wants to know how customers feel about a new mithai.

Be the sentiment model

Read each review as a person would. Which label fits?

  • I would not recommend this to anyone

  • Not bad at all

  • Oh wonderful, another two-hour delay

  • The package weighed two kilograms

  • Best dosa I have eaten this year

  • The shop opens at 9 a.m.

Common mistakes

  • Thinking sentiment analysis reads minds. It only classifies the text.
  • Trusting a method that only counts words. Negation and sarcasm defeat it.
  • Forgetting mixed and neutral texts. Not everything is good or bad.

Check yourself

Question 1

1. What does sentiment analysis classify?

Question 2

2. Why does counting positive words fail on "The film was not good"?

Question 3

3. "Great, my train is cancelled again." What makes this hard?

Summary

  • Sentiment analysis labels text as positive, negative or neutral.
  • Counting feeling words is simple; learning from labelled examples works better.
  • Negation, sarcasm, mixed opinions and slang are the hard cases.

Finished reading?

Sources: AI4K12: Five Big Ideas in AI · Elements of AI (University of Helsinki)