I6 · Neural Networks and Deep Learning Essentials

Activation Functions

An activation function is a non-linear function applied to a neuron's weighted sum. It lets a network learn curved, complex patterns. Common ones are ReLU in hidden layers, sigmoid for yes-or-no outputs, and softmax for multi-class outputs.

10 min Intermediate · Sublevel Intermediate

What you need: The earlier topics of this module.

Definition

An activation function is a non-linear function applied to a neuron's weighted sum. It lets a network learn curved, complex patterns. Common ones are ReLU in hidden layers, sigmoid for yes-or-no outputs, and softmax for multi-class outputs.

In simple words

It's a dimmer switch on each neuron that decides how strongly to pass on its signal. Without it, a network can only draw straight lines. With it, stacks of bends can build any shape, like making a curve out of many short straight pieces.

Overview

  • An activation function transforms a neuron's weighted sum into its output.
  • Non-linear activations let networks learn curved patterns; without them, many layers collapse into one linear layer.
  • Good activations keep gradients healthy so deep networks can train.
  • Hidden layers usually use ReLU or its relatives; output layers use activations matched to the task.
  • Sigmoid and tanh saturate for large inputs, which causes vanishing gradients.
  • Transformers commonly use GELU or SwiGLU activations.

Why it matters

The choice affects whether a network trains at all. Sigmoid in deep hidden layers caused vanishing gradients for years, and the switch to ReLU was a key step in the deep learning boom.

Key terms

Activation function
A non-linear function applied to a neuron's weighted sum.
ReLU
max(0, z): passes positive values, zeroes negative ones.
Sigmoid
1 / (1 + e⁻ᶻ): squashes values into 0 to 1.
Tanh
Squashes values into −1 to 1, centred at zero.
Softmax
Turns a vector of scores into probabilities that add to 1.
Vanishing gradient
When gradients become tiny in early layers, so they barely learn.

Types

There are many activation functions. These are the main types and where each is used.

1. Sigmoid (logistic)

σ(z) = 1 ÷ (1 + e⁻ᶻ), squashing any input into the range 0 to 1.

  • How it works: Its derivative is at most 0.25 and near zero for large |z|, so stacked sigmoids make gradients vanish.
  • Example: σ(2) ≈ 0.88, read as an 88% probability of the positive class.
  • When to use it: Use it in the output layer for binary classification and in LSTM gates; avoid it in deep hidden layers.

2. Tanh

The hyperbolic tangent, squashing inputs into −1 to 1.

  • How it works: It is zero-centred, which helps learning compared with sigmoid, but it still saturates for large inputs.
  • Example: tanh(1) ≈ 0.76 and tanh(−1) ≈ −0.76.
  • When to use it: Use it in recurrent networks and small networks where outputs should be centred around zero.

3. ReLU (rectified linear unit)

ReLU(z) = max(0, z).

  • How it works: It is cheap to compute and passes gradients of 1 for positive inputs, which made training deep networks practical around 2010–2012; units stuck at negative inputs can "die".
  • Example: ReLU(−3) = 0, ReLU(2.5) = 2.5.
  • When to use it: Use it as the default activation for hidden layers in CNNs and MLPs.

4. Leaky ReLU, ELU, GELU and Swish

ReLU variants that keep a small gradient for negative inputs or curve smoothly.

  • How it works: Leaky ReLU uses a small slope such as 0.01 below zero; GELU and Swish multiply the input by a smooth gate, which often trains slightly better.
  • Example: BERT and GPT models use GELU in their feed-forward layers.
  • When to use it: Use them when ReLU units die or in transformer-style models.

5. Softmax

Turning a vector of scores into probabilities that add up to 1.

  • How it works: softmax(zᵢ) = e^(zᵢ) ÷ Σ e^(zⱼ); larger scores get exponentially larger shares.
  • Example: Scores 2, 1 and 0 become probabilities of about 0.67, 0.24 and 0.09.
  • When to use it: Use it in the output layer for multi-class classification and inside attention.

6. Linear (identity) output

No activation: the output equals the weighted sum.

  • How it works: It allows any real value, which is needed when predicting unbounded numbers.
  • Example: The output neuron of a network predicting house prices or temperature.
  • When to use it: Use it in the output layer for regression tasks. Pair it with a regression loss such as MSE or Huber, and scale the target if its values are very large.

Explained in detail

Why non-linearity is essential

A layer without an activation computes a linear function of its inputs. Stack ten such layers and the result is still just one linear function, because a linear function of a linear function is linear. Such a network could only ever draw straight decision boundaries, however deep it was. Inserting a non-linear activation after each layer breaks this collapse. With non-linear activations, a network with enough hidden units can approximate almost any continuous function, a result known as the universal approximation theorem, proved for sigmoid networks by George Cybenko in 1989.

The common choices

FunctionOutput rangeTypical useWatch out
ReLU0 to ∞Hidden layers: the default"Dead" neurons stuck at 0 if their inputs stay negative
Leaky ReLU, GELUSmall negatives allowedHidden layers; GELU in transformersSlightly more computation
Sigmoid0 to 1Output for yes/no probabilitiesSaturates, causing vanishing gradients in deep stacks
Tanh−1 to 1Older networks, some recurrent networksAlso saturates
Softmax0 to 1, summing to 1Output for choosing one of many classesUse with cross-entropy loss
Linear (none)Any valueOutput for regressionNever in hidden layers

Why ReLU took over

For decades, sigmoid and tanh were standard. Their problem is that for large positive or negative inputs they flatten out, so their derivative is close to zero. During backpropagation, these small derivatives are multiplied together layer after layer, and the gradient reaching the early layers can shrink towards nothing: the vanishing gradient problem. ReLU's derivative is exactly 1 for any positive input, so gradients pass through undiminished. It is also extremely cheap to compute. Its widespread adoption around 2010 to 2012, including in the AlexNet network that won the ImageNet competition in 2012, was one of the changes that made deep networks trainable.

  • Default recipe: ReLU (or GELU) in hidden layers; sigmoid for a single yes/no output; softmax for multi-class output; no activation for regression output.
  • Softmax exaggerates differences: scores 2, 1 and 0 become probabilities of about 0.67, 0.24 and 0.09.
  • In language models, the temperature setting divides the scores before softmax, making outputs more or less random.

How it works

FunctionOutput rangeUsed inNote
ReLU[0, ∞)Hidden layers (the default)Fast, but neurons can "die"
Leaky ReLU(−∞, ∞)Hidden layersSmall slope for negatives
GELU(≈−0.17, ∞)TransformersA smooth version of ReLU
Sigmoid(0, 1)Binary outputSaturates, avoid in deep hidden layers
Tanh(−1, 1)RNNs, some hidden layersCentred on zero
Softmax(0, 1), sums to 1Multi-class outputGives a probability for each class

How to choose

  • Hidden layers: start with ReLU. Use GELU for Transformers.
  • Binary classification output: sigmoid.
  • Multi-class output: softmax.
  • Regression output: no activation at all.

The maths

ReLU
Sigmoid
Tanh
Softmax
The score for class k

Softmax of the scores [2, 1, 0]:

e² = 7.389    e¹ = 2.718    e⁰ = 1
Sum = 11.107
Probabilities: 7.389/11.107 = 0.665
               2.718/11.107 = 0.245
               1/11.107     = 0.090
They add up to 1.

In code

activations.py
import numpy as np

relu = lambda x: np.maximum(0, x)
sigmoid = lambda x: 1 / (1 + np.exp(-x))
softmax = lambda z: np.exp(z) / np.exp(z).sum()

x = np.array([-2.0, 0.0, 2.0])
print("ReLU:", relu(x))
print("Sigmoid:", sigmoid(x).round(3))
print("Tanh:", np.tanh(x).round(3))
print("Softmax of [2, 1, 0]:", softmax(np.array([2.0, 1.0, 0.0])).round(3))

Output

ReLU: [0. 0. 2.]
Sigmoid: [0.119 0.5   0.881]
Tanh: [-0.964  0.     0.964]
Softmax of [2, 1, 0]: [0.665 0.245 0.09 ]

Common mistakes

  • Using sigmoid throughout a deep network. Gradients shrink towards zero as they pass back.
  • Adding softmax before a loss that already applies it, such as PyTorch's CrossEntropyLoss.
  • Using softmax for multi-label tasks, where several labels can be true. Use a sigmoid per label.

Check yourself

Question 1

1. What's the default choice for hidden layers?

Question 2

2. You classify images into 10 classes. Which output activation?

Question 3

3. What is ReLU(−3)?

Summary

  • Activations add the non-linearity networks need.
  • ReLU for hidden layers. Sigmoid or softmax for outputs.
  • A poor choice can stop a network from training.

Finished reading?

Sources: Goodfellow, Bengio & Courville, Deep Learning