A1 · Supervised Machine Learning in Depth

Logistic Regression

Logistic regression is a linear classifier that models the probability of the positive class as the sigmoid of a linear function of the features: p = σ(wᵀx + b). It is trained by minimising the log loss (binary cross-entropy), which is convex, so gradient methods find the global minimum. The decision boundary wᵀx + b = 0 is a hyperplane.

20 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

How can a weighted sum answer a yes-or-no question, with a probability?

Logistic regression is a linear classifier that models the probability of the positive class as the sigmoid of a linear function of the features: p = σ(wᵀx + b). It is trained by minimising the log loss (binary cross-entropy), which is convex, so gradient methods find the global minimum. The decision boundary wᵀx + b = 0 is a hyperplane.

In simple words

Compute a score, like linear regression, then squash it into 0 to 1 with the S-shaped sigmoid. A score of 0 becomes 50 per cent; large positive scores approach 100 per cent. The loss punishes confident wrong answers very heavily, so the model learns calibrated probabilities.

Why it matters

Logistic regression is the standard baseline for classification in medicine, finance and marketing, because it is fast, well understood and interpretable: each weight is a change in log-odds. It is also exactly a one-neuron neural network.

Key terms

Sigmoid function
σ(z) = 1 ÷ (1 + e⁻ᶻ), which turns any score into a probability between 0 and 1.
Log-odds (logit)
log(p ÷ (1 − p)); logistic regression models it as a linear function of the features.
Log loss (binary cross-entropy)
The training loss, −[y log p + (1 − y) log(1 − p)], which punishes confident mistakes heavily.
Decision boundary
Where the predicted probability equals the threshold; a straight line or flat plane for logistic regression.
Odds ratio
e^w: how the odds multiply for a one-unit increase in a feature.
Softmax (multinomial) regression
The extension to more than two classes, giving a probability for each.

Explained in detail

From a score to a probability

Despite its name, logistic regression is a classification method. It computes a linear score z = w₀ + w₁x₁ + … + wₚxₚ and passes it through the sigmoid to get a probability. A score of 0 gives probability 0.5, a score of 2 gives about 0.88 and a score of −2 about 0.12. Equivalently, the model says the log-odds of the positive class are a linear function of the features. A coefficient of 0.7 on "previous defaults" means each extra past default multiplies the odds of default by e^0.7 ≈ 2.0, holding other features fixed. This makes logistic regression one of the most interpretable classifiers, which is why it remains standard in credit scoring, medicine and social science.

How it is trained

There is no closed-form solution, so the weights are found by minimising log loss with gradient descent or a quasi-Newton method such as L-BFGS. Log loss rewards well-calibrated probabilities: predicting 0.9 for a case that turns out positive costs −log(0.9) ≈ 0.11, while predicting 0.01 costs −log(0.01) ≈ 4.6. Because the loss is convex, training finds the global optimum. Scikit-learn applies L2 regularisation by default with strength 1 ÷ C, so smaller C means stronger regularisation; features should be scaled. If the classes are perfectly separable, unregularised weights grow without limit, which is another reason regularisation matters.

StrengthsLimitations
Fast, stable and easy to interpret through odds ratiosOnly linear decision boundaries unless you add features
Gives probabilities that are usually reasonably calibratedStruggles with complex interactions that trees capture automatically
A strong baseline for tabular and text data with TF-IDFSensitive to outliers and highly correlated features

In practice, choose the decision threshold for the use case rather than defaulting to 0.5. For a disease screen, a threshold of 0.2 may be right because missing a case is costly; for approving loans automatically, 0.8 may be safer. For several classes, multinomial logistic regression uses softmax to give each class a probability that sums to 1, and "one-versus-rest" trains one binary model per class. Logistic regression is also the building block of neural networks: a single neuron with a sigmoid output trained with log loss is exactly logistic regression.

How it works

  1. Score: z = wᵀx + b.
  2. Probability: p = 1 / (1 + e^(−z)).
  3. Loss per example: −[y log p + (1 − y) log(1 − p)].
  4. Gradient: Xᵀ(p − y) / n, the same shape as linear regression.
  5. Predict the positive class (label 1) when p ≥ threshold, usually 0.5.

The maths

The sigmoid
Log loss (binary cross-entropy)
Its gradient
Log-odds interpretation

In code

Logistic regression trained from scratch with gradient descent, then compared with scikit-learn on the same data.

logistic_regression.py
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
scaler = StandardScaler().fit(X_tr)
X_tr, X_te = scaler.transform(X_tr), scaler.transform(X_te)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

Xb_tr = np.column_stack([np.ones(len(X_tr)), X_tr])
Xb_te = np.column_stack([np.ones(len(X_te)), X_te])
w = np.zeros(Xb_tr.shape[1])
for step in range(2000):
    p = sigmoid(Xb_tr @ w)
    w -= 0.1 * Xb_tr.T @ (p - y_tr) / len(y_tr)

p_tr = sigmoid(Xb_tr @ w)
loss = -np.mean(y_tr * np.log(p_tr) + (1 - y_tr) * np.log(1 - p_tr))
print("From scratch: training log loss", round(float(loss), 4))
print("From scratch: test accuracy", round(float(np.mean((sigmoid(Xb_te @ w) >= 0.5) == y_te)), 3))

sk = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("scikit-learn: test accuracy", round(sk.score(X_te, y_te), 3))
print("Probability for the first test case:", round(float(sk.predict_proba(X_te[:1])[0, 1]), 3))

Output

From scratch: training log loss 0.0478
From scratch: test accuracy 0.965
scikit-learn: test accuracy 0.958
Probability for the first test case: 0.996

Try it yourself

Compute a probability

Match each score z to its sigmoid probability (rounded).

  • z = 0
  • z = 2
  • z = −2
  • z = 5

Common mistakes

  • Using it for non-linear boundaries without adding features.
  • Assuming 0.5 is always the right threshold.
  • Interpreting weights of unscaled or correlated features as importance.

Check yourself

Question 1

1. σ(0) equals…

Question 2

2. Why log loss rather than squared error?

Question 3

3. A weight of 0.7 on a standardised feature means…

Summary

  • p = σ(wᵀx + b): a linear score squashed into a probability.
  • Trained by minimising convex log loss, with gradient Xᵀ(p − y)/n.
  • Weights are changes in log-odds; the boundary is linear.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Logistic regression