I6 · Neural Networks and Deep Learning Essentials

The Perceptron

The perceptron, introduced by Frank Rosenblatt in 1958, is the simplest neural network: one neuron that computes a weighted sum of its inputs and outputs 1 if the total passes a threshold, otherwise 0. It can only learn patterns separable by a straight line.

10 min Intermediate · Sublevel Intermediate

What you need: The earlier topics of this module.

Definition

The perceptron, introduced by Frank Rosenblatt in 1958, is the simplest neural network: one neuron that computes a weighted sum of its inputs and outputs 1 if the total passes a threshold, otherwise 0. It can only learn patterns separable by a straight line.

In simple words

Should you go to the cricket match? You weigh a few things: is it a weekend, are friends going, is it raining? Good reasons add points, and bad ones take them away. If the total passes your limit, you go. That's a perceptron.

Overview

  • The perceptron is the simplest neural network: one neuron with a step activation.
  • It predicts 1 if w · x + b ≥ 0 and 0 otherwise.
  • After each mistake, the weights move by η × (target − prediction) × input.
  • It is guaranteed to converge if the data can be separated by a straight line.
  • Minsky and Papert showed in 1969 that it cannot learn XOR, which slowed research for years.
  • Stacking perceptron-like units in layers solves XOR and much more.

Why it matters

It's where neural networks began. Its best-known limit, the inability to solve XOR, was highlighted by Minsky and Papert in 1969 and led to the idea of hidden layers. Every neuron in a modern network is a descendant of this one.

Key terms

Perceptron
A single artificial neuron with a step activation that outputs 0 or 1.
Weight
A number multiplying each input, showing how much it matters.
Bias
A number added to the weighted sum, shifting the threshold.
Step function
Outputs 1 if the input is at least 0, otherwise 0.
Linearly separable
Classes that a single straight line or flat plane can divide.
Perceptron learning rule
After a mistake, nudge each weight by learning rate × error × input.

Types

Perceptron ideas appear in several types of model. These are the main ones and how they relate.

1. Single-layer perceptron

One layer of perceptron units connected directly to the inputs.

  • How it works: Each unit draws one straight boundary; with several units it can separate several linearly separable classes.
  • Example: Classifying points above and below the line y = x.
  • When to use it: Use it for linearly separable problems and to learn the basic training rule.

2. Perceptron for logic gates

Perceptrons with weights that compute AND, OR and NOT.

  • How it works: AND uses weights 1, 1 and bias −1.5; OR uses 1, 1 and bias −0.5; XOR cannot be done by any single perceptron.
  • Example: Inputs (1, 1) give 1 + 1 − 1.5 = 0.5 ≥ 0, so AND outputs 1.
  • When to use it: Use it to see what a single linear boundary can and cannot represent.

3. Multi-layer perceptron (MLP)

A network with one or more hidden layers between input and output.

  • How it works: Hidden layers with non-linear activations combine simple boundaries into complex ones; training uses backpropagation.
  • Example: Two hidden units (OR and NAND) feeding an AND unit compute XOR.
  • When to use it: Use it for tabular data and as the general-purpose neural network (MLPClassifier in scikit-learn).

4. Averaged and voted perceptrons

Variants that average the weights seen during training for more stable predictions.

  • How it works: Instead of keeping only the final weights, they average every weight vector, which reduces sensitivity to the order of examples.
  • Example: Averaged perceptrons were widely used for part-of-speech tagging before deep learning.
  • When to use it: Use them for fast, simple linear classifiers on large sparse data.

5. Perceptron versus logistic regression

Two linear classifiers that differ in their output and training.

  • How it works: The perceptron outputs a hard 0/1 and updates only on mistakes; logistic regression outputs a probability through the sigmoid and minimises log loss.
  • Example: Both find a straight boundary for spam versus not spam, but only logistic regression says how sure it is.
  • When to use it: Prefer logistic regression when probabilities are needed; the perceptron is mainly of historical and teaching value.

Explained in detail

Rosenblatt's machine

Frank Rosenblatt, a psychologist at the Cornell Aeronautical Laboratory, introduced the perceptron in 1958 and built it as hardware, the Mark I Perceptron, with a grid of 400 light sensors wired to adjustable resistors that acted as weights, turned by electric motors during learning. The New York Times reported that the navy expected it to "walk, talk, see, write, reproduce itself and be conscious of its existence". The hype was premature, but the core idea, a unit that computes a weighted sum and learns its weights from mistakes, is still inside every modern neural network.

How it learns

  1. Start with small or zero weights and bias.
  2. For each training example, compute z = w · x + b and output 1 if z ≥ 0, otherwise 0.
  3. If the output is right, change nothing.
  4. If it is wrong, update each weight: w ← w + η × (target − output) × x, and the bias similarly.
  5. Repeat over the data until there are no mistakes or a limit is reached.

The perceptron convergence theorem guarantees that if the classes are linearly separable, this procedure finds a separating line in a finite number of steps. Geometrically, the weights define a line (or, with more inputs, a flat plane): points on one side get output 1, points on the other side 0, and each mistake rotates and shifts the line towards a better position.

The XOR problem and the AI winter

Input AInput BANDORXOR
00000
01011
10011
11110

A perceptron can learn AND and OR, because one line separates their 1s from their 0s. It cannot learn XOR, where the 1s sit on opposite corners: no single straight line separates them. In 1969 Marvin Minsky and Seymour Papert's book Perceptrons proved such limits rigorously, and enthusiasm and funding for neural networks fell sharply for over a decade. The fix was known in principle, stacking perceptrons into layers with smooth activations, but training multi-layer networks only became practical once backpropagation was popularised in 1986.

How it works

  1. Start with zero or random weights.
  2. For each example, make a prediction.
  3. If it's wrong, move the weights towards the right answer. If it's right, change nothing.
  4. Repeat until it stops making mistakes.
x₁x₂ANDORXOR
00000
01011
10011
11110

A straight line can separate AND and OR, so a perceptron learns them. No single line can separate XOR, and so it fails. Add one hidden layer and the problem goes away.

The maths

Prediction
Learning rule
0 when right, and +1 or −1 when wrong
The learning rate

In code

perceptron.py
def train(data, epochs=10, rate=0.1):
    w = [0.0, 0.0]
    b = 0.0
    for _ in range(epochs):
        for (x1, x2), y in data:
            prediction = 1 if w[0]*x1 + w[1]*x2 + b > 0 else 0
            error = y - prediction
            w[0] += rate * error * x1
            w[1] += rate * error * x2
            b += rate * error
    return w, b

def accuracy(data, w, b):
    right = sum((1 if w[0]*x1 + w[1]*x2 + b > 0 else 0) == y for (x1, x2), y in data)
    return right / len(data)

AND = [((0,0),0), ((0,1),0), ((1,0),0), ((1,1),1)]
XOR = [((0,0),0), ((0,1),1), ((1,0),1), ((1,1),0)]

print("AND accuracy:", accuracy(AND, *train(AND)))
print("XOR accuracy:", accuracy(XOR, *train(XOR, epochs=100)))

Output

AND accuracy: 1.0
XOR accuracy: 0.5

Common mistakes

  • Expecting it to learn any pattern. One perceptron only draws a straight line.
  • Thinking XOR killed neural networks for good. It led to multi-layer networks, which solve it easily.

Check yourself

Question 1

1. Which can't one perceptron learn?

Question 2

2. When the prediction is correct, what does the rule do to the weights?

Summary

  • One neuron with a threshold.
  • It learns by correcting its mistakes.
  • It can't solve XOR, which is why hidden layers exist.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Goodfellow, Bengio & Courville, Deep Learning