Definition
An activation function is a non-linear function applied to a neuron's weighted sum. It lets a network learn curved, complex patterns. Common ones are ReLU in hidden layers, sigmoid for yes-or-no outputs, and softmax for multi-class outputs.
In simple words
It's a dimmer switch on each neuron that decides how strongly to pass on its signal. Without it, a network can only draw straight lines. With it, stacks of bends can build any shape, like making a curve out of many short straight pieces.
Overview
- An activation function transforms a neuron's weighted sum into its output.
- Non-linear activations let networks learn curved patterns; without them, many layers collapse into one linear layer.
- Good activations keep gradients healthy so deep networks can train.
- Hidden layers usually use ReLU or its relatives; output layers use activations matched to the task.
- Sigmoid and tanh saturate for large inputs, which causes vanishing gradients.
- Transformers commonly use GELU or SwiGLU activations.
Why it matters
The choice affects whether a network trains at all. Sigmoid in deep hidden layers caused vanishing gradients for years, and the switch to ReLU was a key step in the deep learning boom.
Key terms
- Activation function
- A non-linear function applied to a neuron's weighted sum.
- ReLU
- max(0, z): passes positive values, zeroes negative ones.
- Sigmoid
- 1 / (1 + e⁻ᶻ): squashes values into 0 to 1.
- Tanh
- Squashes values into −1 to 1, centred at zero.
- Softmax
- Turns a vector of scores into probabilities that add to 1.
- Vanishing gradient
- When gradients become tiny in early layers, so they barely learn.
Types
There are many activation functions. These are the main types and where each is used.
1. Sigmoid (logistic)
σ(z) = 1 ÷ (1 + e⁻ᶻ), squashing any input into the range 0 to 1.
- How it works: Its derivative is at most 0.25 and near zero for large |z|, so stacked sigmoids make gradients vanish.
- Example: σ(2) ≈ 0.88, read as an 88% probability of the positive class.
- When to use it: Use it in the output layer for binary classification and in LSTM gates; avoid it in deep hidden layers.
2. Tanh
The hyperbolic tangent, squashing inputs into −1 to 1.
- How it works: It is zero-centred, which helps learning compared with sigmoid, but it still saturates for large inputs.
- Example: tanh(1) ≈ 0.76 and tanh(−1) ≈ −0.76.
- When to use it: Use it in recurrent networks and small networks where outputs should be centred around zero.
3. ReLU (rectified linear unit)
ReLU(z) = max(0, z).
- How it works: It is cheap to compute and passes gradients of 1 for positive inputs, which made training deep networks practical around 2010–2012; units stuck at negative inputs can "die".
- Example: ReLU(−3) = 0, ReLU(2.5) = 2.5.
- When to use it: Use it as the default activation for hidden layers in CNNs and MLPs.
4. Leaky ReLU, ELU, GELU and Swish
ReLU variants that keep a small gradient for negative inputs or curve smoothly.
- How it works: Leaky ReLU uses a small slope such as 0.01 below zero; GELU and Swish multiply the input by a smooth gate, which often trains slightly better.
- Example: BERT and GPT models use GELU in their feed-forward layers.
- When to use it: Use them when ReLU units die or in transformer-style models.
5. Softmax
Turning a vector of scores into probabilities that add up to 1.
- How it works: softmax(zᵢ) = e^(zᵢ) ÷ Σ e^(zⱼ); larger scores get exponentially larger shares.
- Example: Scores 2, 1 and 0 become probabilities of about 0.67, 0.24 and 0.09.
- When to use it: Use it in the output layer for multi-class classification and inside attention.
6. Linear (identity) output
No activation: the output equals the weighted sum.
- How it works: It allows any real value, which is needed when predicting unbounded numbers.
- Example: The output neuron of a network predicting house prices or temperature.
- When to use it: Use it in the output layer for regression tasks. Pair it with a regression loss such as MSE or Huber, and scale the target if its values are very large.
Explained in detail
Why non-linearity is essential
A layer without an activation computes a linear function of its inputs. Stack ten such layers and the result is still just one linear function, because a linear function of a linear function is linear. Such a network could only ever draw straight decision boundaries, however deep it was. Inserting a non-linear activation after each layer breaks this collapse. With non-linear activations, a network with enough hidden units can approximate almost any continuous function, a result known as the universal approximation theorem, proved for sigmoid networks by George Cybenko in 1989.
The common choices
| Function | Output range | Typical use | Watch out |
|---|---|---|---|
| ReLU | 0 to ∞ | Hidden layers: the default | "Dead" neurons stuck at 0 if their inputs stay negative |
| Leaky ReLU, GELU | Small negatives allowed | Hidden layers; GELU in transformers | Slightly more computation |
| Sigmoid | 0 to 1 | Output for yes/no probabilities | Saturates, causing vanishing gradients in deep stacks |
| Tanh | −1 to 1 | Older networks, some recurrent networks | Also saturates |
| Softmax | 0 to 1, summing to 1 | Output for choosing one of many classes | Use with cross-entropy loss |
| Linear (none) | Any value | Output for regression | Never in hidden layers |
Why ReLU took over
For decades, sigmoid and tanh were standard. Their problem is that for large positive or negative inputs they flatten out, so their derivative is close to zero. During backpropagation, these small derivatives are multiplied together layer after layer, and the gradient reaching the early layers can shrink towards nothing: the vanishing gradient problem. ReLU's derivative is exactly 1 for any positive input, so gradients pass through undiminished. It is also extremely cheap to compute. Its widespread adoption around 2010 to 2012, including in the AlexNet network that won the ImageNet competition in 2012, was one of the changes that made deep networks trainable.
- Default recipe: ReLU (or GELU) in hidden layers; sigmoid for a single yes/no output; softmax for multi-class output; no activation for regression output.
- Softmax exaggerates differences: scores 2, 1 and 0 become probabilities of about 0.67, 0.24 and 0.09.
- In language models, the temperature setting divides the scores before softmax, making outputs more or less random.
How it works
| Function | Output range | Used in | Note |
|---|---|---|---|
| ReLU | [0, ∞) | Hidden layers (the default) | Fast, but neurons can "die" |
| Leaky ReLU | (−∞, ∞) | Hidden layers | Small slope for negatives |
| GELU | (≈−0.17, ∞) | Transformers | A smooth version of ReLU |
| Sigmoid | (0, 1) | Binary output | Saturates, avoid in deep hidden layers |
| Tanh | (−1, 1) | RNNs, some hidden layers | Centred on zero |
| Softmax | (0, 1), sums to 1 | Multi-class output | Gives a probability for each class |
How to choose
- Hidden layers: start with ReLU. Use GELU for Transformers.
- Binary classification output: sigmoid.
- Multi-class output: softmax.
- Regression output: no activation at all.
The maths
- The score for class k
Softmax of the scores [2, 1, 0]:
e² = 7.389 e¹ = 2.718 e⁰ = 1
Sum = 11.107
Probabilities: 7.389/11.107 = 0.665
2.718/11.107 = 0.245
1/11.107 = 0.090
They add up to 1.In code
import numpy as np
relu = lambda x: np.maximum(0, x)
sigmoid = lambda x: 1 / (1 + np.exp(-x))
softmax = lambda z: np.exp(z) / np.exp(z).sum()
x = np.array([-2.0, 0.0, 2.0])
print("ReLU:", relu(x))
print("Sigmoid:", sigmoid(x).round(3))
print("Tanh:", np.tanh(x).round(3))
print("Softmax of [2, 1, 0]:", softmax(np.array([2.0, 1.0, 0.0])).round(3))Output
ReLU: [0. 0. 2.] Sigmoid: [0.119 0.5 0.881] Tanh: [-0.964 0. 0.964] Softmax of [2, 1, 0]: [0.665 0.245 0.09 ]
Common mistakes
- Using sigmoid throughout a deep network. Gradients shrink towards zero as they pass back.
- Adding softmax before a loss that already applies it, such as PyTorch's CrossEntropyLoss.
- Using softmax for multi-label tasks, where several labels can be true. Use a sigmoid per label.
Check yourself
Summary
- Activations add the non-linearity networks need.
- ReLU for hidden layers. Sigmoid or softmax for outputs.
- A poor choice can stop a network from training.
Finished reading?