Definition
How can a weighted sum answer a yes-or-no question, with a probability?
Logistic regression is a linear classifier that models the probability of the positive class as the sigmoid of a linear function of the features: p = σ(wᵀx + b). It is trained by minimising the log loss (binary cross-entropy), which is convex, so gradient methods find the global minimum. The decision boundary wᵀx + b = 0 is a hyperplane.
In simple words
Compute a score, like linear regression, then squash it into 0 to 1 with the S-shaped sigmoid. A score of 0 becomes 50 per cent; large positive scores approach 100 per cent. The loss punishes confident wrong answers very heavily, so the model learns calibrated probabilities.
Why it matters
Logistic regression is the standard baseline for classification in medicine, finance and marketing, because it is fast, well understood and interpretable: each weight is a change in log-odds. It is also exactly a one-neuron neural network.
Key terms
- Sigmoid function
- σ(z) = 1 ÷ (1 + e⁻ᶻ), which turns any score into a probability between 0 and 1.
- Log-odds (logit)
- log(p ÷ (1 − p)); logistic regression models it as a linear function of the features.
- Log loss (binary cross-entropy)
- The training loss, −[y log p + (1 − y) log(1 − p)], which punishes confident mistakes heavily.
- Decision boundary
- Where the predicted probability equals the threshold; a straight line or flat plane for logistic regression.
- Odds ratio
- e^w: how the odds multiply for a one-unit increase in a feature.
- Softmax (multinomial) regression
- The extension to more than two classes, giving a probability for each.
Explained in detail
From a score to a probability
Despite its name, logistic regression is a classification method. It computes a linear score z = w₀ + w₁x₁ + … + wₚxₚ and passes it through the sigmoid to get a probability. A score of 0 gives probability 0.5, a score of 2 gives about 0.88 and a score of −2 about 0.12. Equivalently, the model says the log-odds of the positive class are a linear function of the features. A coefficient of 0.7 on "previous defaults" means each extra past default multiplies the odds of default by e^0.7 ≈ 2.0, holding other features fixed. This makes logistic regression one of the most interpretable classifiers, which is why it remains standard in credit scoring, medicine and social science.
How it is trained
There is no closed-form solution, so the weights are found by minimising log loss with gradient descent or a quasi-Newton method such as L-BFGS. Log loss rewards well-calibrated probabilities: predicting 0.9 for a case that turns out positive costs −log(0.9) ≈ 0.11, while predicting 0.01 costs −log(0.01) ≈ 4.6. Because the loss is convex, training finds the global optimum. Scikit-learn applies L2 regularisation by default with strength 1 ÷ C, so smaller C means stronger regularisation; features should be scaled. If the classes are perfectly separable, unregularised weights grow without limit, which is another reason regularisation matters.
| Strengths | Limitations |
|---|---|
| Fast, stable and easy to interpret through odds ratios | Only linear decision boundaries unless you add features |
| Gives probabilities that are usually reasonably calibrated | Struggles with complex interactions that trees capture automatically |
| A strong baseline for tabular and text data with TF-IDF | Sensitive to outliers and highly correlated features |
In practice, choose the decision threshold for the use case rather than defaulting to 0.5. For a disease screen, a threshold of 0.2 may be right because missing a case is costly; for approving loans automatically, 0.8 may be safer. For several classes, multinomial logistic regression uses softmax to give each class a probability that sums to 1, and "one-versus-rest" trains one binary model per class. Logistic regression is also the building block of neural networks: a single neuron with a sigmoid output trained with log loss is exactly logistic regression.
How it works
- Score: z = wᵀx + b.
- Probability: p = 1 / (1 + e^(−z)).
- Loss per example: −[y log p + (1 − y) log(1 − p)].
- Gradient: Xᵀ(p − y) / n, the same shape as linear regression.
- Predict the positive class (label 1) when p ≥ threshold, usually 0.5.
The maths
In code
Logistic regression trained from scratch with gradient descent, then compared with scikit-learn on the same data.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
scaler = StandardScaler().fit(X_tr)
X_tr, X_te = scaler.transform(X_tr), scaler.transform(X_te)
def sigmoid(z):
return 1 / (1 + np.exp(-z))
Xb_tr = np.column_stack([np.ones(len(X_tr)), X_tr])
Xb_te = np.column_stack([np.ones(len(X_te)), X_te])
w = np.zeros(Xb_tr.shape[1])
for step in range(2000):
p = sigmoid(Xb_tr @ w)
w -= 0.1 * Xb_tr.T @ (p - y_tr) / len(y_tr)
p_tr = sigmoid(Xb_tr @ w)
loss = -np.mean(y_tr * np.log(p_tr) + (1 - y_tr) * np.log(1 - p_tr))
print("From scratch: training log loss", round(float(loss), 4))
print("From scratch: test accuracy", round(float(np.mean((sigmoid(Xb_te @ w) >= 0.5) == y_te)), 3))
sk = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("scikit-learn: test accuracy", round(sk.score(X_te, y_te), 3))
print("Probability for the first test case:", round(float(sk.predict_proba(X_te[:1])[0, 1]), 3))Output
From scratch: training log loss 0.0478 From scratch: test accuracy 0.965 scikit-learn: test accuracy 0.958 Probability for the first test case: 0.996
Try it yourself
Compute a probability
Match each score z to its sigmoid probability (rounded).
- z = 0
- z = 2
- z = −2
- z = 5
Common mistakes
- Using it for non-linear boundaries without adding features.
- Assuming 0.5 is always the right threshold.
- Interpreting weights of unscaled or correlated features as importance.
Check yourself
Summary
- p = σ(wᵀx + b): a linear score squashed into a probability.
- Trained by minimising convex log loss, with gradient Xᵀ(p − y)/n.
- Weights are changes in log-odds; the boundary is linear.
Read it. Now try it yourself.
Open the interactive activityFinished reading?
Sources: James et al., An Introduction to Statistical Learning · scikit-learn: Logistic regression