Definition
If a friend tells you the sun rose this morning, you learn almost nothing. If they tell you it snowed in Chennai, you learn a lot. Information theory measures this surprise in bits, and the same measure helps decision trees choose questions and neural networks measure their errors.
Entropy, introduced by Claude Shannon in 1948, measures the average uncertainty in a random outcome: H = −Σ p log₂ p, in bits. A fair coin has entropy 1 bit; a certain outcome has entropy 0. Information gain is the drop in entropy after learning something, such as the answer to a decision-tree question. Cross-entropy measures how well predicted probabilities match the true outcomes, and is the standard loss for classification models. Kullback–Leibler (KL) divergence measures how different two probability distributions are.
In simple words
Entropy is how many yes-or-no questions you need, on average, to find the answer. Guessing a number from 1 to 8 takes 3 questions ("bigger than 4?" and so on), so its entropy is 3 bits. If the number is almost always 1, you need far fewer questions.
Overview
- Information theory, founded by Claude Shannon in 1948, measures information and uncertainty in bits.
- A rare event carries more information than a common one: −log₂ p bits.
- Entropy is the average information of a random outcome and is highest when all outcomes are equally likely.
- Decision trees choose splits by information gain; classifiers are trained by minimising cross-entropy.
- KL divergence measures how far one probability distribution is from another.
- Compression, communication and language models all rest on these ideas.
Why it matters
Entropy and information gain decide the splits in decision trees, cross-entropy is the loss minimised when training almost every classifier and language model, and KL divergence appears in variational autoencoders and in fine-tuning language models. These ideas also explain compression: predictable data compresses well.
Key terms
- Information content (surprise)
- −log₂ p bits for an outcome of probability p; rare outcomes carry more information.
- Entropy
- The average surprise of a distribution, H = −Σ p log₂ p.
- Bit
- The unit of information when logarithms are base 2: the answer to one fair yes-or-no question.
- Information gain
- Entropy before a split minus the weighted entropy after it.
- Cross-entropy
- The average surprise of the true outcomes under the model's predicted probabilities; a classification loss.
- KL divergence
- How many extra bits are needed when using distribution q in place of the true distribution p.
Types
Information theory defines several related quantities. Each type measures a different aspect of uncertainty or difference.
1. Self-information (surprise)
The information in one particular outcome: I(x) = −log₂ p(x).
- How it works: Rarer outcomes have smaller p and therefore larger surprise; a certain outcome (p = 1) carries 0 bits.
- Example: Rolling a six on a fair die carries log₂ 6 ≈ 2.58 bits; a one-in-a-thousand event about 10 bits.
- When to use it: Use it to reason about how informative an observation is, the building block of all the other measures.
2. Shannon entropy
The average surprise of a distribution: H = −Σ p log₂ p.
- How it works: Weight each outcome's surprise by its probability and add; for n equally likely outcomes H = log₂ n.
- Example: A fair coin has 1 bit; a coin landing heads 90% of the time has about 0.47 bits.
- When to use it: Use it to measure the impurity of groups in decision trees and how compressible data is.
3. Information gain (mutual information)
The reduction in entropy of one variable from learning another.
- How it works: Information gain = parent entropy minus the weighted entropy of the children after a split; mutual information generalises it to any pair of variables.
- Example: Splitting 10 emails on "contains free" reduces entropy from 1 bit to 0.39 bits, a gain of 0.61 bits.
- When to use it: Use it to choose decision-tree splits and to rank features (mutual_info_classif in scikit-learn).
4. Cross-entropy
The average surprise of true outcomes under a model's predicted probabilities.
- How it works: For classification it is −(1/N) Σ ln p̂(true class); confident wrong predictions get very large penalties.
- Example: A predicted probability of 0.9 for the true class costs 0.11 nats; 0.1 costs 2.30 nats.
- When to use it: Use it as the loss for logistic regression, neural-network classifiers and language models.
5. KL divergence
How many extra bits are needed when a distribution q is used in place of the true distribution p.
- How it works: D_KL(p ‖ q) = Σ p log (p ÷ q); it is zero only when p = q and is not symmetric.
- Example: Measuring how much a fine-tuned language model's word probabilities have drifted from the original model.
- When to use it: Use it in variational autoencoders, in keeping fine-tuned models close to their base, and in monitoring data drift.
Explained in detail
Measuring surprise and uncertainty
An outcome with probability p carries −log₂ p bits of information. A fair coin toss (p = 0.5) carries 1 bit; rolling a six on a die (p = 1/6) carries about 2.58 bits; an event with probability 1/1,024 carries 10 bits. Entropy averages this over all outcomes. It is highest when all outcomes are equally likely, because then you cannot predict anything, and zero when one outcome is certain. A biased coin that lands heads 90 per cent of the time has entropy of only about 0.47 bits. Logarithms in base 2 give bits; machine-learning libraries usually use natural logarithms, giving units called nats, but the ideas are identical.
Entropy in decision trees
A decision tree wants questions whose answers leave the groups as pure as possible. Before a split, a node has some entropy; after the split, each child has its own entropy. The weighted average of the children's entropies, weighted by how many examples go to each, is subtracted from the parent's entropy to give the information gain. The tree picks the split with the highest gain. Scikit-learn's DecisionTreeClassifier uses Gini impurity by default and entropy with criterion="entropy"; both usually choose similar splits. In the code below, a depth-one tree with criterion="entropy" chooses "contains_free" over "long_email" for its first split, matching the hand calculation of an information gain of 0.610 bits.
Cross-entropy as a loss
| Model's probability for the true class | Loss −ln p (nats) | Meaning |
|---|---|---|
| 0.99 | 0.01 | Confident and right: tiny loss |
| 0.9 | 0.11 | Fairly sure and right |
| 0.5 | 0.69 | Unsure |
| 0.1 | 2.30 | Confident and wrong: large loss |
| 0.01 | 4.61 | Very confident and wrong: very large loss |
Cross-entropy punishes confident mistakes heavily, which pushes models to give honest probabilities. A language model is trained to minimise the cross-entropy of the next token; its perplexity, e raised to that average loss, can be read as how many words it is effectively choosing between at each step.
Real-world scenarios: file compressors such as ZIP give short codes to common patterns and long codes to rare ones, so text with low entropy shrinks a lot while random data hardly shrinks at all; mobile networks use information theory to decide how much error-correcting data to add; and the game of twenty questions is won by asking questions that split the remaining possibilities as evenly as possible, which is exactly the question with the highest information gain.
How it works
- List the outcomes and their probabilities.
- Compute each outcome's surprise, −log₂ p.
- Average the surprises weighted by probability to get entropy.
- For a split, subtract the weighted child entropy from the parent entropy.
- For a classifier, average −ln p of the true class to get cross-entropy.
The maths
- Predicted probability of the true class for example i
Ten emails: 5 spam and 5 not, so the entropy is 1 bit. Split on "contains the word free": 4 emails contain it, all spam (entropy 0); 6 do not, 1 spam and 5 not. The second group's entropy is −(1/6)log₂(1/6) − (5/6)log₂(5/6) ≈ 0.431 + 0.219 = 0.650 bits. The weighted entropy after the split is 0.4 × 0 + 0.6 × 0.650 = 0.390, so the information gain is 1 − 0.390 = 0.610 bits. A split that left both groups half spam would have gain 0. In the same spirit, a box with 4 red, 2 blue and 2 green balls has entropy 0.5 × 1 + 0.25 × 2 + 0.25 × 2 = 1.5 bits.
In code
import numpy as np
from sklearn.tree import DecisionTreeClassifier
def entropy(counts):
p = np.array(counts, dtype=float) / sum(counts)
p = p[p > 0]
return float(-(p * np.log2(p)).sum())
print("Fair coin:", entropy([1, 1]), "| biased coin 90/10:", round(entropy([9, 1]), 3), "| balls 4/2/2:", entropy([4, 2, 2]))
parent = entropy([5, 5])
left, right = entropy([4, 0]), entropy([1, 5])
gain = parent - (4 / 10 * left + 6 / 10 * right)
print(f"Right child entropy {right:.3f}, information gain {gain:.3f} bits")
# A decision tree using entropy picks the same question
contains_free = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0])
long_email = np.array([1, 0, 1, 0, 1, 0, 1, 0, 1, 0])
is_spam = np.array([1, 1, 1, 1, 1, 0, 0, 0, 0, 0])
tree = DecisionTreeClassifier(criterion="entropy", max_depth=1).fit(np.column_stack([contains_free, long_email]), is_spam)
print("Feature chosen for the first split:", ["contains_free", "long_email"][tree.tree_.feature[0]])
for prob in [0.99, 0.9, 0.5, 0.1, 0.01]:
print(f"predicted probability of true class {prob:<5} -> cross-entropy loss {-np.log(prob):.2f}")Output
Fair coin: 1.0 | biased coin 90/10: 0.469 | balls 4/2/2: 1.5 Right child entropy 0.650, information gain 0.610 bits Feature chosen for the first split: contains_free predicted probability of true class 0.99 -> cross-entropy loss 0.01 predicted probability of true class 0.9 -> cross-entropy loss 0.11 predicted probability of true class 0.5 -> cross-entropy loss 0.69 predicted probability of true class 0.1 -> cross-entropy loss 2.30 predicted probability of true class 0.01 -> cross-entropy loss 4.61
Try it yourself
High or low entropy?
Which outcomes are hard to predict?
Whether the sun rises tomorrow
A fair die roll
Which of 1,000 equally likely tickets wins
A coin that lands heads 99% of the time
Computing information gain
Put the steps in order.
- 1Compute the entropy of the parent group
- 2Compute the entropy of each child group
- 3Split the examples by the answer to a question
- 4Subtract the weighted sum from the parent entropy
- 5Weight each child entropy by its share of examples
Common mistakes
- Mixing log base 2 and natural logarithms without noticing the units.
- Thinking higher entropy means better data; it means more uncertainty.
- Ignoring how heavily cross-entropy punishes confident wrong predictions.
Check yourself
Summary
- Entropy H = −Σ p log₂ p measures average uncertainty in bits.
- Information gain chooses decision-tree splits by how much entropy they remove.
- Cross-entropy is the standard classification loss and heavily punishes confident mistakes.
Finished reading?
Sources: Claude Shannon (1948), A Mathematical Theory of Communication · David MacKay, Information Theory, Inference, and Learning Algorithms (free book)