I6 · Neural Networks and Deep Learning Essentials

Layers and Network Architecture

A neural network arranges neurons in layers: an input layer that receives the features, one or more hidden layers that transform them, and an output layer that produces predictions. In a fully connected (dense) layer, every neuron connects to every neuron in the previous layer. The architecture is the choice of how many layers, how many neurons in each, which activation functions and how layers are connected.

12 min Intermediate · Sublevel Intermediate Jump to the activity

What you need: The earlier topics of this module.

Definition

One neuron can draw only a straight line between two classes. Stack a few layers of them, and the network can draw almost any boundary. How do you decide how many layers, and how wide?

A neural network arranges neurons in layers: an input layer that receives the features, one or more hidden layers that transform them, and an output layer that produces predictions. In a fully connected (dense) layer, every neuron connects to every neuron in the previous layer. The architecture is the choice of how many layers, how many neurons in each, which activation functions and how layers are connected.

In simple words

Recognising a face happens in stages: early layers notice edges, middle layers combine edges into eyes and noses, later layers combine those into a face. Each layer builds more complex ideas from the simpler ones before it.

Overview

  • A neural network is built from layers: an input layer, one or more hidden layers and an output layer.
  • Each layer transforms its input with weights, biases and activations.
  • Depth is the number of layers; width is the number of neurons per layer.
  • The architecture is the arrangement of layers, chosen to suit the data type.
  • The number of parameters of a dense layer is inputs × outputs + outputs.
  • The universal approximation theorem says a wide enough network can approximate any continuous function, but depth makes this far more efficient.

Why it matters

Architecture decides what a network can learn, how much data it needs, how fast it trains and how well it generalises. Reading and designing architectures is a core skill for using deep learning.

Key terms

Input layer
Holds the features; it does no computation.
Hidden layer
A layer between input and output that learns intermediate features.
Output layer
Produces the prediction, with an activation suited to the task.
Dense layer
A fully connected layer: every input connects to every neuron.
Depth and width
The number of layers and the number of neurons per layer.
Parameters
All the weights and biases; a dense layer from n inputs to m neurons has n × m + m.

Types

Networks are built from several types of layer, and these combine into a few main types of architecture.

1. Dense (fully connected) layers

Layers in which every neuron connects to every input.

  • How it works: Output = activation(W x + b); a layer from 784 inputs to 128 neurons has 784 × 128 + 128 = 100,480 parameters.
  • Example: The hidden layers of an MLP classifying handwritten digits.
  • When to use it: Use them for tabular data and as the final layers of most networks.

2. Convolutional and pooling layers

Layers that slide small filters over grid data and reduce its size.

  • How it works: Convolutions share weights across positions, detecting the same pattern anywhere; pooling keeps the strongest responses in small regions.
  • Example: 3 × 3 filters detecting edges in the first layer of an image classifier.
  • When to use it: Use them for images, spectrograms and other grid-like data.

3. Recurrent layers

Layers that process sequences step by step, keeping a hidden state.

  • How it works: The same weights are applied at every time step; LSTM and GRU cells add gates to remember information longer.
  • Example: An LSTM reading a sentence word by word for sentiment analysis.
  • When to use it: Use them for sequences such as text, speech and sensor data, though transformers now dominate.

4. Attention and embedding layers

Embedding layers turn tokens into vectors; attention layers let each position look at every other.

  • How it works: An embedding is a learned lookup table; self-attention weights other positions by query-key similarity.
  • Example: Every block of a GPT model combines self-attention with a feed-forward layer.
  • When to use it: Use them for language, and increasingly for images, audio and multimodal data.

5. Normalisation and regularisation layers

Batch normalisation, layer normalisation and dropout.

  • How it works: Normalisation keeps activations at a stable scale so training is faster; dropout randomly switches off neurons during training to reduce overfitting.
  • Example: Layer normalisation inside every transformer block; dropout of 0.5 in a small classifier.
  • When to use it: Use them in deep networks to make training stable and models generalise.

6. Architecture families

Standard designs: feed-forward MLPs, CNNs, RNNs, transformers, autoencoders and GANs.

  • How it works: Each family arranges layers to suit a data type; residual (skip) connections, as in ResNet (2015), let networks reach hundreds of layers.
  • Example: ResNet-50 for images; GPT for text; U-Net for medical image segmentation.
  • When to use it: Start from the family that matches your data, usually with a pre-trained model.

Explained in detail

A simple network

Consider a network that predicts whether a student will pass from four features: attendance, hours of study, past marks and assignments submitted. The input layer has 4 values. A hidden layer of 8 neurons each computes a weighted sum of all 4 inputs plus a bias and applies ReLU: that is 4 × 8 weights plus 8 biases, 40 parameters. A second hidden layer of 4 neurons adds 8 × 4 + 4 = 36. The output layer has 1 neuron with a sigmoid giving the probability of passing: 4 + 1 = 5. The whole network has 81 parameters. Image and language models use the same arithmetic with millions or billions of parameters.

Why depth helps

The universal approximation theorem says a network with a single, wide enough hidden layer can approximate almost any continuous function. In practice, deeper networks with fewer neurons per layer often learn the same function far more efficiently, because each layer reuses the features built by the one before. This compositional structure matches the real world: pixels form edges, edges form shapes, shapes form objects; letters form words, words form phrases, phrases form meaning. Training very deep networks used to be hard because gradients vanished, but ReLU, better initialisation, normalisation layers and residual connections, introduced in the ResNet architecture of 2015, made networks with hundreds of layers trainable.

Choosing the output layer

TaskOutput neuronsActivationLoss
Predict a number1None (linear)Mean squared error
Yes or no1SigmoidBinary cross-entropy
One of k classeskSoftmaxCategorical cross-entropy
Several independent labelsOne per labelSigmoid on eachBinary cross-entropy per label

Architecture families

  • Multilayer perceptrons (dense layers only) suit tabular data and small problems.
  • Convolutional networks share weights across positions and suit images.
  • Recurrent networks process sequences step by step with memory.
  • Transformers use attention to relate every position to every other, and now dominate language and increasingly vision.
  • Start small: a network that is too large for the data overfits and wastes computation; grow it only if validation results improve.

A useful habit is to write the shape of the data after every layer, for example (batch, 4) then (batch, 8) then (batch, 4) then (batch, 1). Most bugs in deep-learning code are shape mismatches, and this simple check catches them before training begins.

How it works

  1. Set the input layer to the number of features.
  2. Choose the output layer and loss for the task.
  3. Start with one or two hidden layers of moderate width, using ReLU.
  4. Count the parameters and compare with the amount of data.
  5. Adjust depth and width based on validation performance.

In code

forward_pass.py
import numpy as np

rng = np.random.default_rng(0)
layer_sizes = [4, 8, 4, 1]  # inputs, hidden, hidden, output
weights = [rng.normal(0, 0.5, (a, b)) for a, b in zip(layer_sizes, layer_sizes[1:])]
biases = [np.zeros(b) for b in layer_sizes[1:]]

relu = lambda z: np.maximum(0, z)
sigmoid = lambda z: 1 / (1 + np.exp(-z))

def forward(X):
    a = X
    for i, (W, b) in enumerate(zip(weights, biases)):
        z = a @ W + b
        a = sigmoid(z) if i == len(weights) - 1 else relu(z)
        print(f"layer {i + 1}: output shape {a.shape}")
    return a

X = np.array([[0.9, 3.0, 0.72, 1.0], [0.6, 1.0, 0.45, 0.0]])  # two students
print("P(pass):", forward(X).ravel().round(3))
print("Parameters:", sum(W.size + b.size for W, b in zip(weights, biases)))

Output

layer 1: output shape (2, 8)
layer 2: output shape (2, 4)
layer 3: output shape (2, 1)
P(pass): [0.514 0.558]
Parameters: 81

Try it yourself

From edges to faces

A face-recognition network has many layers; researchers visualise what each one responds to.

Which output layer?

Choose the output setup for each task.

  • Will this email be opened?

  • Which of 10 digits is in the image?

  • Predict tomorrow's temperature

  • Which of 5 crop diseases?

  • Predict a house price

Common mistakes

  • Choosing a network far larger than the data can support.
  • Using the wrong output activation for the task.
  • Leaving out non-linear activations in hidden layers.

Check yourself

Question 1

1. How many parameters does a dense layer from 10 inputs to 5 neurons have?

Question 2

2. Which output setup suits classifying one of 10 digits?

Question 3

3. Why do deep networks learn complex patterns efficiently?

Summary

  • Networks have input, hidden and output layers; dense layers connect everything to everything.
  • Depth lets layers build complex features from simple ones.
  • Match the output layer and loss to the task, and size the network to the data.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Dive into Deep Learning (Zhang, Lipton, Li, Smola) · TensorFlow Playground