Definition
Despite its name, logistic regression is for classification. It answers yes-or-no questions with a probability.
Logistic regression is a classification algorithm that computes a weighted sum of the inputs and passes it through the sigmoid function, giving a probability between 0 and 1. It's trained by minimising log loss.
In simple words
Each clue about an email adds or takes away points: "free" adds 2, a known sender takes away 3. Total the points. Then the sigmoid squashes any total into a chance between 0% and 100%. A very high score gets close to 100%, a very low score gets close to 0%.
Overview
- Logistic regression is a classification model that predicts the probability of a class.
- It computes a weighted sum z = w · x + b and passes it through the sigmoid σ(z) = 1 ÷ (1 + e⁻ᶻ).
- It is trained by minimising log loss (cross-entropy), usually with regularisation.
- Each weight changes the log-odds; e^w is the odds ratio for a one-unit increase in that feature.
- Its decision boundary is linear, though polynomial features can bend it.
- It is fast, well calibrated and a standard baseline in medicine, finance and text classification.
Why it matters
It's fast and explainable, and it gives probabilities rather than just labels, which is why banks and hospitals use it. It's the standard baseline for classification. A single neuron with a sigmoid is exactly logistic regression, so it's also your first step to neural networks.
Key terms
- Logistic regression
- A classifier that turns a weighted sum of features into a probability with the sigmoid function.
- Logit or log-odds
- The weighted sum before the sigmoid: the log of p / (1 − p).
- Decision threshold
- The probability above which the model predicts the positive class, 0.5 by default.
- Log loss
- The loss that heavily punishes confident wrong predictions; also called cross-entropy.
- Decision boundary
- The line or surface where the predicted probability equals the threshold.
- Softmax
- The multi-class extension that turns several scores into probabilities adding to 1.
Types
Logistic regression has several types depending on the number of classes and how it is regularised.
1. Binary logistic regression
Predicting one of two classes.
- How it works: The sigmoid output is P(class = 1); predicting the positive class when it exceeds a threshold, usually 0.5.
- Example: Will a student pass (1) or not (0), from attendance and study hours.
- When to use it: Use it as the first model for any yes/no prediction.
2. Multinomial (softmax) logistic regression
Predicting one of three or more classes directly.
- How it works: Each class has its own weight vector; the softmax turns the class scores into probabilities that sum to 1.
- Example: Classifying handwritten digits 0–9 from pixel values.
- When to use it: Use it for multi-class problems; it is also the final layer of most neural-network classifiers.
3. One-vs-rest logistic regression
Handling many classes by training one binary model per class.
- How it works: Each model separates one class from all others; the class with the highest score wins.
- Example: Tagging news articles as sport, politics, business or science.
- When to use it: Use it when classes are not mutually exclusive or when binary models are easier to train.
4. Regularised logistic regression (L1, L2, elastic net)
Logistic regression with a penalty on weight size.
- How it works: In scikit-learn, C is the inverse of regularisation strength: smaller C means stronger regularisation; penalty="l1" can zero out features.
- Example: An L1 model keeping 30 of 10,000 words for spam detection.
- When to use it: Use it whenever there are many features, tuning C by cross-validation.
5. Ordinal logistic regression
Predicting ordered categories such as low, medium and high.
- How it works: It models cumulative probabilities, P(y ≤ k), with shared weights and separate thresholds for each level.
- Example: Predicting customer satisfaction ratings from 1 to 5 stars.
- When to use it: Use it when classes have a natural order and treating them as unrelated would waste information.
Explained in detail
Why not just use linear regression?
For a yes-or-no question, such as whether a customer will leave, a straight line can predict values below 0 or above 1, which make no sense as probabilities, and outliers can tilt the line badly. Logistic regression keeps the weighted sum z = w · x + b but passes it through the sigmoid, σ(z) = 1 / (1 + e⁻ᶻ), which squashes any number into the range 0 to 1. A large positive z gives a probability near 1, a large negative z gives a probability near 0, and z = 0 gives exactly 0.5. Despite its name, logistic regression is a classification method.
Reading the coefficients
Each coefficient changes the log-odds. Exponentiating it gives an odds ratio: a coefficient of 0.7 for "has a complaint in the last month" means e⁰·⁷ ≈ 2, so the odds of leaving roughly double for customers with a recent complaint, other features held fixed. This interpretability is why logistic regression is standard in medicine, credit scoring and social science, where people must understand and justify decisions.
Log loss
| True class | Predicted probability of positive | Log loss for this example |
|---|---|---|
| Positive | 0.9 | 0.11 (small) |
| Positive | 0.5 | 0.69 |
| Positive | 0.1 | 2.30 (large) |
| Positive | 0.01 | 4.61 (very large) |
Log loss is −log(p) for the probability the model gave to the true class. Being confidently wrong costs far more than being unsure, which pushes the model towards honest probabilities. Unlike squared error with a sigmoid, log loss gives a smooth, bowl-shaped surface for logistic regression, so gradient descent reliably finds the best weights.
Thresholds, classes and limits
- The 0.5 threshold is only a default. For disease screening you might flag anything above 0.2 to catch more cases; for spam you might require 0.9 to avoid blocking real email.
- For more than two classes, softmax regression gives a probability for each class.
- The decision boundary is a straight line or flat plane, so curved boundaries need extra features or a different model.
- Scale features before training so the optimiser converges well and coefficients are comparable.
- A logistic regression unit is exactly one neuron with a sigmoid activation, so it is the bridge to neural networks.
How it works
- Compute the score: a weighted sum of the features plus a bias.
- Apply the sigmoid to turn the score into a probability.
- Choose a threshold, usually 0.5, to get a yes or no.
- Train by reducing log loss with gradient descent.
Why not plain linear regression?
A straight line can predict values below 0 or above 1, which make no sense as probabilities. It's also pulled around badly by outliers. The sigmoid keeps the output in range.
The maths
- The score, any number
- A probability between 0 and 1
Weights: w₁ = 2 for the word "free", w₂ = −3 for a known sender, and b = −1. An email contains "free" from an unknown sender, so x = [1, 0].
z = 2(1) + (−3)(0) + (−1) = 1 σ(1) = 1 / (1 + e⁻¹) = 1 / 1.368 ≈ 0.731 P(spam) = 73.1%, above 0.5, so: spam If it really is spam, the loss = −ln(0.731) ≈ 0.313
In code
import math
from sklearn.linear_model import LogisticRegression
def sigmoid(z):
return 1 / (1 + math.exp(-z))
for z in (-5, -1, 0, 1, 5):
print(f"sigmoid({z:>2}) = {sigmoid(z):.3f}")
hours = [[1], [2], [3], [4], [5], [6], [7], [8]]
passed = [0, 0, 0, 0, 1, 1, 1, 1]
model = LogisticRegression().fit(hours, passed)
for h in (2, 4.5, 7):
print(f"Studied {h} hours: chance of passing = {model.predict_proba([[h]])[0][1]:.2f}")Output
sigmoid(-5) = 0.007 sigmoid(-1) = 0.269 sigmoid( 0) = 0.500 sigmoid( 1) = 0.731 sigmoid( 5) = 0.993 Studied 2 hours: chance of passing = 0.05 Studied 4.5 hours: chance of passing = 0.50 Studied 7 hours: chance of passing = 0.95
Common mistakes
- Thinking it does regression because of the name.
- Keeping a threshold of 0.5 without thinking about the cost of each error.
- Skipping feature scaling, which can slow training and make weights hard to compare.
Check yourself
Summary
- Weighted sum + sigmoid = a probability.
- Trained by minimising log loss.
- Choose the threshold based on what your errors cost.
Finished reading?
Sources: scikit-learn user guide · Google Machine Learning Crash Course