Definition
Nobody gives a toddler a labelled data set on how to walk. They try, fall, get up and slowly learn which movements keep them upright. Reinforcement learning teaches computers in the same way: by acting, receiving rewards or penalties, and improving.
Reinforcement learning (RL) is a type of machine learning in which an agent learns by interacting with an environment. At each step the agent observes the state, chooses an action, and receives a reward and a new state. Its goal is a policy, a rule for choosing actions, that maximises the total discounted reward over time. Q-learning is a classic RL algorithm that learns Q(s, a), the expected future reward of taking action a in state s and acting well afterwards, using the update Q(s,a) ← Q(s,a) + α[r + γ·max Q(s′,a′) − Q(s,a)]. Agents balance exploration (trying new actions) and exploitation (using what works).
In simple words
Training a puppy: when it sits on command, it gets a treat; when it chews shoes, it gets a firm "no". Over time it learns which actions lead to treats. In RL the puppy is the agent, the treats and "no"s are rewards, and what it learns is the policy.
Overview
- In reinforcement learning (RL), an agent learns a policy by acting in an environment and receiving rewards.
- The goal is to maximise the total discounted future reward.
- Problems are modelled as Markov decision processes: states, actions, transitions, rewards and a discount γ.
- Agents balance exploration (trying new actions) and exploitation (using what works).
- Q-learning learns the value of each action in each state from experience.
- RL powered AlphaGo (2016) and is used to fine-tune chatbots with human feedback.
Why it matters
RL suits problems where decisions come in sequences and their effects appear later: game playing (AlphaGo beat a world champion at Go in 2016), robot control, traffic-signal timing, energy management in data centres, and fine-tuning large language models with human feedback.
Key terms
- Agent and environment
- The learner that acts, and the world it acts in.
- State, action, reward
- The situation, the choice made, and the numeric feedback received.
- Policy
- A rule mapping each state to an action.
- Discount factor (γ)
- A number between 0 and 1 that makes future rewards count less than immediate ones.
- Q-value
- The expected total future reward of an action in a state.
- Exploration vs exploitation
- Trying new actions to learn versus using the best-known action; ε-greedy picks a random action with probability ε.
Types
RL methods are grouped into several types by what they learn and whether they model the environment.
1. Value-based methods (Q-learning, SARSA, DQN)
Methods that learn how good each action is in each state.
- How it works: Q-learning updates Q(s, a) towards r + γ max Q(s′, a′); SARSA uses the action actually taken next; DQN (2015) replaces the table with a neural network.
- Example: A 4 × 4 grid agent whose episodes fell from 14.2 to 6.4 steps as it learned.
- When to use it: Use them for discrete actions; DQN learned dozens of Atari games from pixels.
2. Policy-based methods (policy gradients)
Methods that learn the policy directly, as probabilities of actions.
- How it works: They adjust policy parameters in the direction that increases expected reward; REINFORCE (1992) is the classic algorithm.
- Example: A robot arm learning smooth, continuous joint movements.
- When to use it: Use them for continuous actions and stochastic policies. They can be slow to learn because their updates are noisy.
3. Actor-critic methods (A2C, PPO)
Methods combining a policy (actor) with a value estimate (critic).
- How it works: The critic judges how good actions were, reducing the noise in policy updates; PPO (2017) limits how far each update can change the policy.
- Example: PPO is widely used in robotics and in reinforcement learning from human feedback (RLHF) for chatbots.
- When to use it: Use them as robust general-purpose RL algorithms.
4. Model-based RL
Methods that learn or are given a model of the environment and plan with it.
- How it works: The agent predicts next states and rewards, then plans or simulates many futures before acting, needing fewer real interactions.
- Example: AlphaZero planning with Monte Carlo tree search using known game rules; MuZero learning the rules itself.
- When to use it: Use it when real experience is expensive, as with physical robots.
5. Multi-armed bandits
A simplified RL problem with no states: choosing repeatedly among options to maximise reward.
- How it works: Strategies such as ε-greedy, upper confidence bound and Thompson sampling balance trying options with using the best so far.
- Example: Choosing which of five headlines to show to maximise clicks, adapting as results arrive.
- When to use it: Use them for online experiments, ad selection and recommendations where decisions do not change the state.
Explained in detail
How RL differs from supervised learning
In supervised learning, every example comes with the correct answer. In reinforcement learning, nobody tells the agent the correct action; it only receives a reward, often delayed. A chess move may seem fine but lose the game twenty moves later. The agent must work out which earlier actions deserve credit or blame, the credit assignment problem. It also generates its own data by acting, so a poor policy sees only part of the world. That is why exploration matters: an agent that always takes the action it currently thinks is best may never discover a much better route. The ε-greedy strategy explores with a small probability ε, such as 0.1, and exploits otherwise, often reducing ε as learning progresses.
Q-learning step by step
Q-learning keeps a table with one value for each state-action pair, starting at zero. In each step the agent picks an action (ε-greedy), observes reward r and next state s′, and moves Q(s, a) towards the target r + γ·max Q(s′, a′): the reward now plus the discounted value of the best action from the next state. The learning rate α controls how far it moves. Repeating this over many episodes spreads information backwards from rewarding states, so Q-values near the goal rise first, then those further away. Once learned, the policy is simply to take the action with the highest Q-value in each state. For problems with huge numbers of states, such as video-game screens, a neural network replaces the table: this is deep Q-learning (DQN, 2015). In the 4 × 4 grid in the code, episodes take 14.2 steps on average at first, while the agent is mostly exploring, and 6.4 steps by the end; the learned arrows lead from the start to the goal in the shortest possible 6 moves while steering around the pit.
| Learning type | Feedback | Example |
|---|---|---|
| Supervised | Correct answer for every example | Spam or not spam |
| Unsupervised | No answers; find structure | Customer segments |
| Reinforcement | Rewards after actions, often delayed | Learning to play a game or steer a robot |
Designing rewards needs care. An agent maximises exactly what you reward, not what you meant: a cleaning robot rewarded for "no visible dust" might learn to cover its camera. This is called reward hacking, and avoiding it is an important part of safe AI.
How it works
- Define states, actions and rewards for the problem.
- Initialise a Q-table with zeros.
- Run episodes: choose actions with ε-greedy, observe rewards, update Q-values.
- Reduce ε over time and watch the episode reward or length improve.
- Read the learned policy and test it without exploration.
The maths
- Learning rate
- Discount factor
A robot is in state s with Q(s, right) = 2. It moves right, receives reward r = 1 and lands in state s′ where the best Q-value is 5. With γ = 0.9 and α = 0.5, the target is 1 + 0.9 × 5 = 5.5. The error is 5.5 − 2 = 3.5, so the new value is 2 + 0.5 × 3.5 = 3.75. The estimate has moved halfway towards the target. Discounting also shows why sooner is better: a reward of 10 received in 3 steps is worth 0.9³ × 10 ≈ 7.3 now.
In code
import numpy as np
# The worked example: one Q-learning update
q_old, r, best_next, gamma, alpha = 2.0, 1.0, 5.0, 0.9, 0.5
print("Updated Q:", q_old + alpha * (r + gamma * best_next - q_old))
# A 4 x 4 grid: start top-left, goal bottom-right (+10), a pit at (1, 2) (-10), each step -1
SIZE, GOAL, PIT = 4, (3, 3), (1, 2)
MOVES = {0: (-1, 0), 1: (1, 0), 2: (0, -1), 3: (0, 1)} # up, down, left, right
ARROWS = "^v<>"
def step(state, action):
r, c = state; dr, dc = MOVES[action]
nxt = (min(max(r + dr, 0), SIZE - 1), min(max(c + dc, 0), SIZE - 1))
if nxt == GOAL: return nxt, 10, True
if nxt == PIT: return nxt, -10, True
return nxt, -1, False
rng = np.random.default_rng(0)
Q = np.zeros((SIZE, SIZE, 4))
lengths = []
for episode in range(300):
eps = max(0.05, 1 - episode / 150) # explore less over time
s, done, n = (0, 0), False, 0
while not done and n < 50:
a = rng.integers(4) if rng.random() < eps else int(np.argmax(Q[s]))
s2, reward, done = step(s, a)
target = reward if done else reward + 0.9 * Q[s2].max()
Q[s][a] += 0.5 * (target - Q[s][a])
s, n = s2, n + 1
lengths.append(n)
print("Average steps per episode: first 20 =", np.mean(lengths[:20]), "| last 20 =", np.mean(lengths[-20:]))
for r in range(SIZE):
print(" ".join("G" if (r, c) == GOAL else "X" if (r, c) == PIT else ARROWS[int(np.argmax(Q[r, c]))] for c in range(SIZE)))Output
Updated Q: 3.75 Average steps per episode: first 20 = 14.2 | last 20 = 6.4 > v < v > v X v > v > v > > > G
Try it yourself
Watch an agent learn
Watch the agent explore the maze. Notice how it wanders at first, then finds the reward, and how later runs follow a shorter path.
Which kind of learning?
Choose supervised, unsupervised or reinforcement learning.
Teaching a robot arm to stack blocks by trial and error
Predicting house prices from past sales
Finding topics in news articles
Grouping shoppers by buying habits
Timing traffic lights to reduce waiting
Common mistakes
- Never exploring, so the agent gets stuck with a mediocre policy.
- Designing a reward that can be gamed in unintended ways.
- Expecting RL to learn quickly; it often needs many episodes or a simulator.
Check yourself
Summary
- In RL an agent learns a policy from rewards by interacting with an environment.
- Q-learning updates Q(s, a) towards r + γ·max Q(s′, a′); γ discounts the future and α sets the step size.
- Balance exploration and exploitation, and design rewards carefully to avoid reward hacking.
Finished reading?
Sources: Sutton and Barto, Reinforcement Learning: An Introduction, 2nd edition (free book) · OpenAI Spinning Up: key concepts in reinforcement learning