I5 · Machine Learning Foundations

Logistic Regression

Logistic regression is a classification algorithm that computes a weighted sum of the inputs and passes it through the sigmoid function, giving a probability between 0 and 1. It's trained by minimising log loss.

10 min Intermediate · Sublevel Intermediate

What you need: The earlier topics of this module.

Definition

Despite its name, logistic regression is for classification. It answers yes-or-no questions with a probability.

Logistic regression is a classification algorithm that computes a weighted sum of the inputs and passes it through the sigmoid function, giving a probability between 0 and 1. It's trained by minimising log loss.

In simple words

Each clue about an email adds or takes away points: "free" adds 2, a known sender takes away 3. Total the points. Then the sigmoid squashes any total into a chance between 0% and 100%. A very high score gets close to 100%, a very low score gets close to 0%.

Overview

  • Logistic regression is a classification model that predicts the probability of a class.
  • It computes a weighted sum z = w · x + b and passes it through the sigmoid σ(z) = 1 ÷ (1 + e⁻ᶻ).
  • It is trained by minimising log loss (cross-entropy), usually with regularisation.
  • Each weight changes the log-odds; e^w is the odds ratio for a one-unit increase in that feature.
  • Its decision boundary is linear, though polynomial features can bend it.
  • It is fast, well calibrated and a standard baseline in medicine, finance and text classification.

Why it matters

It's fast and explainable, and it gives probabilities rather than just labels, which is why banks and hospitals use it. It's the standard baseline for classification. A single neuron with a sigmoid is exactly logistic regression, so it's also your first step to neural networks.

Key terms

Logistic regression
A classifier that turns a weighted sum of features into a probability with the sigmoid function.
Logit or log-odds
The weighted sum before the sigmoid: the log of p / (1 − p).
Decision threshold
The probability above which the model predicts the positive class, 0.5 by default.
Log loss
The loss that heavily punishes confident wrong predictions; also called cross-entropy.
Decision boundary
The line or surface where the predicted probability equals the threshold.
Softmax
The multi-class extension that turns several scores into probabilities adding to 1.

Types

Logistic regression has several types depending on the number of classes and how it is regularised.

1. Binary logistic regression

Predicting one of two classes.

  • How it works: The sigmoid output is P(class = 1); predicting the positive class when it exceeds a threshold, usually 0.5.
  • Example: Will a student pass (1) or not (0), from attendance and study hours.
  • When to use it: Use it as the first model for any yes/no prediction.

2. Multinomial (softmax) logistic regression

Predicting one of three or more classes directly.

  • How it works: Each class has its own weight vector; the softmax turns the class scores into probabilities that sum to 1.
  • Example: Classifying handwritten digits 0–9 from pixel values.
  • When to use it: Use it for multi-class problems; it is also the final layer of most neural-network classifiers.

3. One-vs-rest logistic regression

Handling many classes by training one binary model per class.

  • How it works: Each model separates one class from all others; the class with the highest score wins.
  • Example: Tagging news articles as sport, politics, business or science.
  • When to use it: Use it when classes are not mutually exclusive or when binary models are easier to train.

4. Regularised logistic regression (L1, L2, elastic net)

Logistic regression with a penalty on weight size.

  • How it works: In scikit-learn, C is the inverse of regularisation strength: smaller C means stronger regularisation; penalty="l1" can zero out features.
  • Example: An L1 model keeping 30 of 10,000 words for spam detection.
  • When to use it: Use it whenever there are many features, tuning C by cross-validation.

5. Ordinal logistic regression

Predicting ordered categories such as low, medium and high.

  • How it works: It models cumulative probabilities, P(y ≤ k), with shared weights and separate thresholds for each level.
  • Example: Predicting customer satisfaction ratings from 1 to 5 stars.
  • When to use it: Use it when classes have a natural order and treating them as unrelated would waste information.

Explained in detail

Why not just use linear regression?

For a yes-or-no question, such as whether a customer will leave, a straight line can predict values below 0 or above 1, which make no sense as probabilities, and outliers can tilt the line badly. Logistic regression keeps the weighted sum z = w · x + b but passes it through the sigmoid, σ(z) = 1 / (1 + e⁻ᶻ), which squashes any number into the range 0 to 1. A large positive z gives a probability near 1, a large negative z gives a probability near 0, and z = 0 gives exactly 0.5. Despite its name, logistic regression is a classification method.

Reading the coefficients

Each coefficient changes the log-odds. Exponentiating it gives an odds ratio: a coefficient of 0.7 for "has a complaint in the last month" means e⁰·⁷ ≈ 2, so the odds of leaving roughly double for customers with a recent complaint, other features held fixed. This interpretability is why logistic regression is standard in medicine, credit scoring and social science, where people must understand and justify decisions.

Log loss

True classPredicted probability of positiveLog loss for this example
Positive0.90.11 (small)
Positive0.50.69
Positive0.12.30 (large)
Positive0.014.61 (very large)

Log loss is −log(p) for the probability the model gave to the true class. Being confidently wrong costs far more than being unsure, which pushes the model towards honest probabilities. Unlike squared error with a sigmoid, log loss gives a smooth, bowl-shaped surface for logistic regression, so gradient descent reliably finds the best weights.

Thresholds, classes and limits

  • The 0.5 threshold is only a default. For disease screening you might flag anything above 0.2 to catch more cases; for spam you might require 0.9 to avoid blocking real email.
  • For more than two classes, softmax regression gives a probability for each class.
  • The decision boundary is a straight line or flat plane, so curved boundaries need extra features or a different model.
  • Scale features before training so the optimiser converges well and coefficients are comparable.
  • A logistic regression unit is exactly one neuron with a sigmoid activation, so it is the bridge to neural networks.

How it works

  1. Compute the score: a weighted sum of the features plus a bias.
  2. Apply the sigmoid to turn the score into a probability.
  3. Choose a threshold, usually 0.5, to get a yes or no.
  4. Train by reducing log loss with gradient descent.

Why not plain linear regression?

A straight line can predict values below 0 or above 1, which make no sense as probabilities. It's also pulled around badly by outliers. The sigmoid keeps the output in range.

The maths

Sigmoid function
The score, any number
A probability between 0 and 1
The model
Log loss

Weights: w₁ = 2 for the word "free", w₂ = −3 for a known sender, and b = −1. An email contains "free" from an unknown sender, so x = [1, 0].

z = 2(1) + (−3)(0) + (−1) = 1
σ(1) = 1 / (1 + e⁻¹) = 1 / 1.368 ≈ 0.731
P(spam) = 73.1%, above 0.5, so: spam

If it really is spam, the loss = −ln(0.731) ≈ 0.313

In code

logistic.py
import math
from sklearn.linear_model import LogisticRegression

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

for z in (-5, -1, 0, 1, 5):
    print(f"sigmoid({z:>2}) = {sigmoid(z):.3f}")

hours = [[1], [2], [3], [4], [5], [6], [7], [8]]
passed = [0, 0, 0, 0, 1, 1, 1, 1]
model = LogisticRegression().fit(hours, passed)
for h in (2, 4.5, 7):
    print(f"Studied {h} hours: chance of passing = {model.predict_proba([[h]])[0][1]:.2f}")

Output

sigmoid(-5) = 0.007
sigmoid(-1) = 0.269
sigmoid( 0) = 0.500
sigmoid( 1) = 0.731
sigmoid( 5) = 0.993
Studied 2 hours: chance of passing = 0.05
Studied 4.5 hours: chance of passing = 0.50
Studied 7 hours: chance of passing = 0.95

Common mistakes

  • Thinking it does regression because of the name.
  • Keeping a threshold of 0.5 without thinking about the cost of each error.
  • Skipping feature scaling, which can slow training and make weights hard to compare.

Check yourself

Question 1

1. What is σ(0)?

Question 2

2. Logistic regression is used for:

Question 3

3. For cancer screening, should the threshold usually be lower or higher than 0.5?

Summary

  • Weighted sum + sigmoid = a probability.
  • Trained by minimising log loss.
  • Choose the threshold based on what your errors cost.

Finished reading?

Sources: scikit-learn user guide · Google Machine Learning Crash Course