Definition
Every day a state grievance portal receives thousands of complaints. Should this one go to the electricity board, the water department, the police or the health office? Sorting by hand takes days; a text classifier does it as each complaint arrives.
Text classification assigns one or more predefined labels to a piece of text. It covers spam detection, topic labelling, routing support tickets, language identification, toxicity detection and sentiment. In binary classification there are two labels; in multi-class classification exactly one of several; in multi-label classification a text can have several labels at once, such as a news story tagged both "sports" and "business". A typical pipeline turns text into features, such as TF-IDF vectors or embeddings, trains a classifier and evaluates it with precision, recall and F1 per label.
In simple words
A post office sorts letters into bags by PIN code. A text classifier sorts messages into bags by what they are about, reading the words instead of a code.
Overview
- Text classification assigns one or more labels to a piece of text.
- Typical tasks are spam detection, topic tagging, intent detection and language identification.
- A pipeline vectorises text (TF-IDF or embeddings) and trains a classifier.
- Labelled examples are needed; their quality limits the model.
- Precision, recall and F1 per class reveal weaknesses that accuracy hides.
- LLMs can classify with prompts alone, which is useful when labels are scarce.
Why it matters
Text classification is the most common NLP task in industry and government. It saves time, speeds up responses and lets organisations measure what people write about. It is also a template for many other ML problems.
Key terms
- Multi-class
- Each text gets exactly one label from several.
- Multi-label
- Each text can get any number of labels.
- Label taxonomy
- The agreed list and definitions of categories.
- Macro F1
- The average F1 across classes, giving small classes equal weight.
- Annotation guidelines
- Written rules that tell labellers how to choose labels consistently.
- Data drift
- Changes over time in the text or categories, such as new complaint types.
Types
Text classification comes in several task types and several model types. Both are described here.
1. Binary and multi-class classification
Choosing one label from two or more classes.
- How it works: The classifier outputs probabilities and picks the highest; class imbalance needs class weights or resampling.
- Example: Spam versus not spam; routing tickets to billing, delivery or technical support.
- When to use it: Use it when each text belongs to exactly one category.
2. Multi-label classification
Assigning any number of labels to one text.
- How it works: Each label gets its own yes/no output and threshold.
- Example: A news article tagged both "economy" and "agriculture".
- When to use it: Use it when texts can cover several topics at once. Evaluate with per-label F1 and with metrics such as Hamming loss, and tune each label's threshold separately on validation data.
3. Classic linear models
Naive Bayes, logistic regression and linear SVMs on bag-of-words or TF-IDF features.
- How it works: They train in seconds and their weights show which words drive each class.
- Example: A TF-IDF and logistic-regression news classifier reaching high accuracy with a few thousand articles.
- When to use it: Use them as fast, explainable baselines.
4. Neural and transformer classifiers
CNNs, LSTMs and fine-tuned transformers such as BERT, RoBERTa and IndicBERT.
- How it works: Fine-tuning adds a classification layer on top of a pre-trained model and trains on labelled examples.
- Example: IndicBERT classifying complaints written in Hindi, Marathi and Tamil.
- When to use it: Use them for the best accuracy, especially when context and word order matter.
5. Zero-shot and few-shot classification with LLMs
Classifying with instructions and perhaps a few examples in a prompt.
- How it works: The LLM reads the label definitions and returns a label, ideally in a structured format.
- Example: Sorting citizen feedback into ten categories with no labelled training data.
- When to use it: Use it to start quickly, then measure accuracy on a labelled sample and consider a cheaper fine-tuned model at scale.
Explained in detail
Building a classifier step by step
Start with the label taxonomy, because unclear categories doom a project: if "billing" and "payment failure" overlap, labellers will disagree and the model will too. Write short guidelines with examples, then label a sample of real texts, ideally with two people labelling the same 200 items to measure agreement. Split the data into training, validation and test sets, keeping the test set untouched until the end. A strong first model is TF-IDF with unigrams and bigrams plus logistic regression or a linear support vector machine; it trains in seconds and often reaches 80 to 90 per cent of the accuracy of much larger models. Then try fine-tuning a pre-trained transformer, and keep it only if it clearly helps.
Choosing features and models
| Option | Good when | Watch out for |
|---|---|---|
| TF-IDF + linear model | Thousands of labelled examples, need speed and explanations | Synonyms and new phrasing |
| Sentence embeddings + classifier | Few labels, many languages | Embedding model must suit the language |
| Fine-tuned transformer | Subtle categories, enough data and compute | Cost, latency, explainability |
| Zero-shot with an LLM | No labelled data yet; quick prototypes | Consistency, cost per message, privacy |
Evaluating and running it
Report precision, recall and F1 for every class and the macro average, and study the confusion matrix to see which categories are mixed up; often the fix is to merge or redefine categories rather than tune the model. For routing systems, set a confidence threshold: confident predictions are routed automatically, uncertain ones go to a person, and those human decisions become new training data. After launch, watch for data drift, such as a new government scheme generating a new type of complaint, and retrain regularly. Check that the classifier works equally well for messages in different languages and from different regions, so that some citizens' complaints are not systematically misrouted.
When you have no labelled data at all, large language models offer zero-shot classification: you describe the categories in a prompt and ask the model to choose one for each message. This is a fast way to prototype and to create a first set of labels, which people then check and correct. For high-volume, steady work, a small trained classifier is usually cheaper, faster and more consistent, and it keeps citizens' messages inside your own systems, which matters for privacy. Many teams use the language model to label data and then train the small model on those checked labels.
How it works
- Define clear, non-overlapping categories with written guidelines.
- Label real examples and measure agreement between labellers.
- Train a TF-IDF baseline, then try embeddings or a fine-tuned transformer.
- Evaluate per class with a confusion matrix and macro F1.
- Deploy with a confidence threshold, human review and regular retraining.
In code
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
train = [
("no electricity since morning in our lane", "electricity"), ("power cut every night transformer burnt", "electricity"),
("electric bill too high meter faulty", "electricity"), ("street light not working wire hanging", "electricity"),
("no water supply for three days", "water"), ("dirty water coming from tap", "water"),
("pipeline leaking on main road", "water"), ("water tanker did not come", "water"),
("garbage not collected for a week", "sanitation"), ("drain overflowing bad smell", "sanitation"),
("dead dog on road not removed", "sanitation"), ("public toilet very dirty", "sanitation"),
]
texts, labels = zip(*train)
model = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression(max_iter=1000)).fit(texts, labels)
new = ["tap water smells bad", "transformer sparking near school", "garbage dumped near drain", "road has big potholes"]
for text, probs in zip(new, model.predict_proba(new)):
best = probs.argmax()
label = model.classes_[best]
action = f"route to {label}" if probs[best] >= 0.38 else "send to a person"
print(f"{text:<34} {label:<12} p={probs[best]:.2f} -> {action}")Output
tap water smells bad water p=0.40 -> route to water transformer sparking near school electricity p=0.39 -> route to electricity garbage dumped near drain sanitation p=0.42 -> route to sanitation road has big potholes sanitation p=0.35 -> send to a person
Try it yourself
Routing grievances faster
A state grievance cell receives 8,000 complaints a day in English, Hindi and Marathi.
Match the task to its type
Binary, multi-class or multi-label?
- Spam or not spam
- Which one department should handle this complaint?
- Tag a news story with every relevant topic
- Which language is this message in?
Common mistakes
- Starting to label before agreeing clear category definitions.
- Judging only by overall accuracy when some classes are rare.
- Never retraining as new kinds of text appear.
Check yourself
Summary
- Text classification assigns predefined labels: binary, multi-class or multi-label.
- Clear categories and guidelines matter as much as the model; TF-IDF plus a linear model is a strong baseline.
- Evaluate per class, route uncertain cases to people, watch for drift and audit fairness.
Finished reading?
Sources: scikit-learn: text feature extraction · Hugging Face NLP course