Definition
'Wah, phir se crash!' starts with a word of praise and ends with a complaint. If one person labels it positive and another negative, which label should a sentiment model learn from?
Annotation, or labelling, assigns each piece of data a category from a fixed scheme, such as a sentiment label for each review. Reliable annotation uses a written guideline, has at least two people label the same items independently, measures their agreement with statistics such as Cohen's kappa, and resolves disagreements by adjudication.
In simple words
Two teachers mark the same answer sheet without seeing each other's marks. If they agree, the marking scheme is clear. Where they differ, they talk, decide, and add a line to the marking scheme so the next teacher doesn't face the same doubt. Labelling reviews works exactly like that.
Overview
- A model learns from labels that people decided, so the labels' quality limits the model's quality.
- Hinglish adds its own challenges: Roman-script spellings vary, sentences switch language, and sarcasm, negation and emojis change the meaning.
- A guideline fixes the unit of annotation, defines each label with examples, and gives rules for the hard cases.
- At least two people label the same items independently; raw agreement and Cohen's kappa measure how consistent they are.
- Disagreements are adjudicated, turned into new rules, and the revised guideline is tested on fresh examples.
- At scale, teams use many annotators, gold questions, labelling tools and measures such as Krippendorff's alpha, and document how the data was labelled.
Why it matters
A sentiment model cannot be more consistent than the labels it learns from. If the same kind of sarcastic complaint is labelled positive in one place and negative in another, the model receives contradictory examples, and its test score is measured against labels that are partly arbitrary. Measuring agreement tells you how much of the 'truth' in a dataset is really a matter of interpretation.
Key terms
- Annotation guideline
- Written instructions saying when to assign each label, with examples and rules for hard cases.
- Label set
- The fixed list of labels, such as P, N, M and U, each with a definition.
- Unit of annotation
- The piece of text that receives one decision, such as a whole review or one sentence.
- Code-mixing
- Switching between two languages within a sentence or conversation, as Hinglish does with Hindi and English.
- Independent annotation
- Each person labels without seeing or discussing anyone else's labels.
- Observed agreement
- The fraction of shared items on which the annotators chose the same label.
- Chance agreement
- The agreement two annotators would reach by luck alone, given how often each uses each label.
- Cohen's kappa
- Agreement beyond chance for two annotators: 1 is perfect, 0 is no better than chance, below 0 is worse than chance.
- Adjudication
- Reviewing conflicting labels, deciding the final one and recording the reason as a rule.
- Gold set
- Items with agreed, checked labels, used to train annotators and to spot careless work.
- Aspect-based sentiment
- Labelling the opinion about each feature separately instead of one label per review.
Types
Sentiment can be labelled in several ways. The scheme must match the question the application needs answered, because each one gives a different kind of training signal.
1. Document-level sentiment
One label for the overall opinion of each whole review, chosen from a small fixed set such as P, N, M and U.
- How it works: Annotators read the entire review, apply the guideline's rules for sarcasm, negation and mixed opinions, and record a single label.
- Example: 'Design accha, search slow' gets M, because it clearly praises one thing and criticises another.
- When to use it: When the application needs one decision per message, such as tracking overall satisfaction with an app over time.
2. Aspect-based sentiment
A separate label for the opinion about each feature, or aspect, that a review mentions.
- How it works: Annotators first mark each aspect, such as design, search or battery, and then label the sentiment towards that aspect.
- Example: 'Layout simple, par battery bahut khaata hai' becomes layout: positive and battery: negative.
- When to use it: When the team must know which feature to fix, not only whether users are happy overall.
3. Ordinal ratings
Labels on an ordered scale, such as 1 to 5 stars, or very negative to very positive.
- How it works: Annotators choose a point on the scale, and agreement is measured with weighted kappa or Krippendorff's alpha, so near-misses count less than distant disagreements.
- Example: Calling a review 4 when a colleague chose 5 is a smaller disagreement than calling it 1.
- When to use it: When the strength of opinion matters, for example to find the angriest complaints first.
4. Multi-label tags
Several yes-or-no tags per review, describing what it contains rather than how the writer feels.
- How it works: Annotators decide each tag independently, such as bug report, feature request or praise, and agreement is measured tag by tag.
- Example: 'Dark mode kab aayega?' is tagged feature request, and 'Har baar login fail' is tagged bug report.
- When to use it: When reviews must reach the right team, such as developers for bugs and designers for requests.
Explained in detail
Why Hinglish reviews are hard to label
Hinglish mixes Hindi and English, usually in Roman script, and people switch languages inside a sentence, a habit linguists call code-mixing. Spelling is not standardised: the same word appears as bahut, bohot and bhot, and nahi as nahin or nai, so a list of 'positive words' misses many variants. Meaning depends on more than words. 'Wah, phir se crash!' uses a word of praise to complain, 'Not bad at all!' is praise built from a negative word, and 'Dark mode kab aayega?' is a request that may carry no opinion at all. The SemEval-2020 SentiMix shared task released about 20,000 Hinglish tweets labelled positive, negative or neutral, and 61 teams competed on them, which shows how seriously researchers take the problem.
| Hard case | Example | Guideline decision |
|---|---|---|
| Sarcasm | 'Great, ek aur spinning wheel' | Label the intended meaning: N |
| Negation | 'Not bad at all!' | Read the whole phrase: P |
| Mixed opinion | 'Design accha, search slow' | Clear praise and clear criticism: M |
| Question or request | 'Button kidhar hai?' | No opinion expressed: U |
| Hedged approval | 'Bas theek hai shayad' | Decide in the guideline, with an example |
| Emoji | 'Wow 🙄' | Treat the emoji as part of the text; an eye-roll often signals sarcasm |
Writing the guideline
Start by fixing the unit of annotation. Here the college wants the writer's overall opinion of its timetable app in each message, so one review gets one label. If the question were about individual features, an aspect-based scheme would record 'design: positive, battery: negative' instead. Then define each label with two examples: P for praise, N for criticism, M when a review clearly contains both, and U when the overall opinion is unclear or absent. Some teams split U into neutral, meaning no opinion, and unclear, meaning the reader cannot decide; that costs one more label but keeps two different situations apart.
Write rules for the hard cases in the table, plus one rule for doubt: what to do when no rule fits, such as choosing U and adding a comment. Avoid rules that guess the writer's mood from the form of the sentence, such as assuming that anyone who asks a question is unhappy. A guideline is a living document: every adjudicated disagreement should end as a new rule with an example.
Running an independent round
Give each reader the same numbered list, the guideline and an empty answer sheet, and ask them to label before discussing anything or looking at the demonstration answers in the code. Keep the order identical, or use review IDs, so that two decisions at position seven refer to the same review. Preserve both original sheets after discussion. If people edit their first labels to match the group decision, the agreement number will measure a conversation that has already happened and hide the ambiguity the exercise was meant to find. Twenty reviews make a good pilot round; a dataset for training a model needs hundreds or thousands, labelled with the guideline the pilot produced.
Measuring agreement: from raw percentage to kappa
Observed agreement is the share of reviews on which both readers chose the same label. It is easy to understand but flattering, because some agreement happens by luck. If two readers each mark 90 per cent of reviews positive at random, they agree on 0.9 × 0.9 + 0.1 × 0.1 = 82 per cent of reviews without reading a word. In 1960 the psychologist Jacob Cohen proposed kappa, which subtracts the agreement expected by chance from the agreement observed and divides by the most that could have been gained above chance. Kappa is 1 for perfect agreement, 0 for agreement no better than chance, and negative when readers agree less often than chance would predict.
A widely quoted scale from Landis and Koch (1977) calls 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and above 0.80 almost perfect, but its authors chose the bands for convenience, and kappa depends on how often each label is used. Alvan Feinstein and Domenic Cicchetti described in 1990 how high observed agreement can come with a low kappa when one label dominates. Always report observed agreement, kappa and the label counts together, and read the disagreements themselves.
Adjudication: turning disagreement into rules
In the demonstration the readers disagree on three reviews, and each points to a gap in the guideline. 'Wah, phir se crash!' was N for one reader and P for the other; the rule 'label sarcasm by its intended meaning' settles it as N. 'Bas theek hai shayad' was U and P; the guideline must say whether lukewarm approval counts as praise, and this team decides that a hedged 'theek hai ... shayad' means U. 'Layout simple, par battery bahut khaata hai' was M and N; one reader did not treat 'simple' as praise, and the team decides that 'simple' about a layout is praise, so the review is M. Record each case, the evidence each reader used, the decision and the new rule, then test the revised guideline independently on fresh reviews. Agreement on new examples, not on the ones already discussed, shows whether the guideline improved.
Labelling at scale
Real datasets use more than two annotators and far more than twenty items. Fleiss' kappa (1971) extends the idea to many raters, and Krippendorff's alpha handles any number of annotators, items labelled by only some of them, and ordered labels such as star ratings, where weighted kappa also helps by counting a 4-versus-5 disagreement as smaller than 1-versus-5. Teams train annotators on a gold set of agreed examples, mix hidden gold items into the work to spot careless labelling, and manage the work with tools such as Label Studio or doccano. Some researchers now keep every annotator's label instead of only the majority vote, because disagreement about sarcasm or offence is itself information about the text.
Reviews contain personal information, so remove names, phone numbers and similar details before sharing them with annotators, a habit that also fits India's Digital Personal Data Protection Act, 2023. Pay annotators fairly, warn them before they read abusive text, and document who labelled what, with which guideline and how agreement was measured, in a datasheet, the practice proposed by Timnit Gebru and colleagues in 2018.
Run the code below. It lists the three disagreements, counts each reader's labels and reports observed agreement 0.85, chance agreement 0.2625 and Cohen's kappa 0.80, matching the worked example. The two mostly-positive sheets agree on 0.90 of the reviews, yet their chance agreement is 0.905 and their kappa is −0.05.
How it works
- Choose the unit and write the guideline: each label, its definition, two examples, and rules for sarcasm, negation, mixed opinions and questions.
- Prepare one numbered list, and give each annotator the guideline and an empty sheet.
- Label independently, without discussion, and save the original sheets.
- Compute observed agreement, chance agreement and Cohen's kappa, and list every disagreement.
- Adjudicate each disagreement, record the decision and turn it into a reusable rule with an example.
- Test the revised guideline independently on fresh reviews before labelling the full dataset.
The maths
- The number of items both annotators labelled
- How many items annotator A gave label k
Reader A used P 6 times, N 5, M 4 and U 5; reader B used P 8, N 5, M 3 and U 4. Chance agreement adds the products of their shares for each label: (6 × 8 + 5 × 5 + 4 × 3 + 5 × 4) ÷ 20² = 105 ÷ 400 = 0.2625. They agreed on 17 of 20 reviews, so p_o = 0.85 and κ = (0.85 − 0.2625) ÷ (1 − 0.2625) = 0.5875 ÷ 0.7375 ≈ 0.797, about 0.80.
Now take two readers who each label 19 of 20 reviews P and disagree on just two reviews. Then p_o = 18 ÷ 20 = 0.90, but p_e = (19 × 19 + 1 × 1) ÷ 400 = 362 ÷ 400 = 0.905, so κ = (0.90 − 0.905) ÷ (1 − 0.905) ≈ −0.05. High observed agreement, yet no agreement beyond chance: this is the paradox Feinstein and Cicchetti described.
In code
The script checks both sheets, lists each disagreement for adjudication, and computes observed agreement, chance agreement and Cohen's kappa exactly as in the worked example. The last lines repeat the calculation for two readers who label almost everything positive.
from collections import Counter
reviews = [
"App mast hai", "Har baar login fail", "Design accha, search slow",
"Button kidhar hai?", "Wah, phir se crash!", "Ab timetable jaldi khulta hai",
"Update ke baad data gayab", "Calendar sahi, alerts bekaar", "Kal install kiya",
"Not bad at all!", "Bas theek hai shayad", "Offline mode kamaal hai",
"Great, ek aur spinning wheel", "Font clear hai par tabs confusing",
"Dark mode kab aayega?", "Kaafi time bach gaya", "Notifications late aate hain",
"Layout simple, par battery bahut khaata hai", "Version number kya hai?",
"Finally search works!",
]
LABELS = "PNMU" # Positive, Negative, Mixed, Unclear or no opinion
# Demonstration sheets. Cover them, label the reviews yourself, then paste your own letters here.
reader_a = "P N M U N P N M U P U P N M U P N M U P".split()
reader_b = "P N M U P P N M U P P P N M U P N N U P".split()
def agreement(a, b):
"""Observed agreement, agreement expected by chance, and Cohen's kappa for two sheets."""
if len(a) != len(b) or set(a + b) - set(LABELS):
raise ValueError("Both sheets need one of P, N, M or U for every review")
n = len(a)
p_o = sum(x == y for x, y in zip(a, b)) / n
count_a, count_b = Counter(a), Counter(b)
p_e = sum(count_a[k] * count_b[k] for k in LABELS) / (n * n)
return p_o, p_e, (p_o - p_e) / (1 - p_e)
if len(reader_a) != len(reviews):
raise ValueError("Give one label per review")
print("Disagreements to adjudicate:")
for number, (review, a, b) in enumerate(zip(reviews, reader_a, reader_b), 1):
if a != b:
print(f" {number:>2}. {review} (A: {a}, B: {b})")
print("Label counts A:", dict(Counter(reader_a)), " B:", dict(Counter(reader_b)))
p_o, p_e, kappa = agreement(reader_a, reader_b)
print(f"Observed {p_o:.2f}, expected by chance {p_e:.4f}, Cohen's kappa {kappa:.2f}")
# The kappa paradox: two readers who call almost every review positive
lazy_a = ["P"] * 18 + ["N", "P"]
lazy_b = ["P"] * 18 + ["P", "N"]
p_o, p_e, kappa = agreement(lazy_a, lazy_b)
print(f"Mostly-P sheets: observed {p_o:.2f}, expected by chance {p_e:.3f}, Cohen's kappa {kappa:.2f}")Output
Disagreements to adjudicate:
5. Wah, phir se crash! (A: N, B: P)
11. Bas theek hai shayad (A: U, B: P)
18. Layout simple, par battery bahut khaata hai (A: M, B: N)
Label counts A: {'P': 6, 'N': 5, 'M': 4, 'U': 5} B: {'P': 8, 'N': 5, 'M': 3, 'U': 4}
Observed 0.85, expected by chance 0.2625, Cohen's kappa 0.80
Mostly-P sheets: observed 0.90, expected by chance 0.905, Cohen's kappa -0.05Try it yourself
Two readers and an adjudicator
Follow one review from two independent labels to a new guideline rule.
Label these reviews
Use the guideline: P praise, N criticism, M both, U unclear or no opinion.
Notes feature mast, par sync slow
Great, ab login bhi nahi hota 🙄
Bohot badhiya app, roz use karta hoon
Bura nahi hai, kaafi helpful
App ka naya version kab aayega?
Paise waste, uninstall kar diya
Run a two-reader review
Use the twenty reviews in the code and the four labels.
- Cover the demonstration sheets. With a partner, label all twenty reviews independently. Working alone, label them twice, a day apart, and compare yourself with yourself.
- Paste both sheets into the code and run it. Note the observed agreement, the kappa and the label counts.
- Adjudicate one disagreement: write the evidence each of you used, the decision, and one rule with an example.
- Label these fresh reviews independently with your new rule: 'Kya speed hai, teen minute se load ho raha hai!' and 'Search fast hai par results adhoore hain.' Did you agree this time?
Think about it
- Which of your disagreements came from an unclear guideline, and which from a simple slip?
Common mistakes
- Discussing examples before labelling, so the agreement measures a conversation instead of the guideline.
- Editing the original sheets after discussion, which hides where the disagreement was.
- Reporting observed agreement alone, which looks high whenever one label dominates.
- Labelling sarcasm by its words, such as 'wah' or 'great', instead of its intended meaning.
- Treating every question or feature request as negative by default.
- Calling a kappa 'substantial' from a fixed table without looking at the label counts and the disagreements.
Check yourself
Summary
- Labels are human decisions; their consistency limits the model trained on them.
- Fix the unit and write a guideline with examples and rules for sarcasm, negation, mixed opinions and questions.
- Label independently, keep the original sheets, and measure observed agreement and Cohen's kappa together.
- Adjudicate disagreements into new rules and test the revised guideline on fresh reviews.
- At scale, use more annotators, gold items and measures such as Krippendorff's alpha, protect personal data and document the labelling.
Finished reading?
Sources: Artstein and Poesio · Inter-coder agreement for computational linguistics (2008) · Patwa et al. · SemEval-2020 Task 9: sentiment analysis of code-mixed tweets · scikit-learn · cohen_kappa_score · Gebru et al. · Datasheets for Datasets