B2 · How Artificial Intelligence Works: Data and Learning

Reinforcement Learning

Reinforcement learning is machine learning by trial and error. An agent takes actions in an environment and receives rewards for good outcomes and penalties for bad ones. Over many tries it learns which actions lead to the most reward.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

How do you teach a dog a new trick without explaining it in words?

Reinforcement learning is machine learning by trial and error. An agent takes actions in an environment and receives rewards for good outcomes and penalties for bad ones. Over many tries it learns which actions lead to the most reward.

In simple words

Give a dog a treat when it sits, and nothing when it doesn't. Soon it sits on command. Nobody showed it the right answer; it found out from rewards. In the maze activity, the robot gets points for reaching the trophy and loses points for traps and wasted steps. After enough practice, it finds the best route.

Why it matters

Reinforcement learning taught computers to play games such as Go and chess better than any person, and helps robots learn to walk and grab objects. It is also part of how chatbots are trained to give more helpful answers.

Key terms

Reinforcement learning
Learning by trial and error, guided by rewards and penalties.
Agent
The learner that takes actions, such as a robot or a game-playing program.
Environment
The world the agent acts in, such as a maze, a game or a road.
Reward
A signal, usually a number, telling the agent that an outcome was good or bad.
Policy
The agent's strategy: which action it chooses in each situation.
Exploration
Trying new actions to discover whether they lead to more reward.

Explained in detail

Learning by doing

Nobody learns to ride a bicycle from a book. You get on, wobble, fall, adjust, and try again. Staying upright feels good; falling hurts. After enough tries your body knows what to do, although you could not write down the rules. Reinforcement learning gives a machine the same kind of experience. There are no labelled examples. There is an agent that acts, an environment that responds, and a reward that says how well things went.

The loop

  1. The agent observes the situation it is in.
  2. It chooses an action.
  3. The environment changes, and the agent receives a reward or a penalty.
  4. The agent adjusts its policy so that rewarded actions become more likely.
  5. It repeats this thousands or millions of times.

Two difficulties

The first is delayed reward. In chess, the only clear signal is winning or losing at the end, perhaps sixty moves after the decisive mistake. The agent must work out which of its many earlier moves deserved the credit or the blame. The second is the balance between exploring and exploiting. An agent that always repeats the best action it has found so far may never discover a better one. An agent that keeps trying random actions never settles down to use what it knows. You face the same choice when you decide between your usual dish at a restaurant and something new on the menu.

What it has achieved

Reinforcement learning produced some of the most striking results in AI. Programs learned to play old video games directly from the screen, reaching and passing human scores. AlphaGo and its successors learned Go, chess and other games by playing millions of games against themselves, and became stronger than any human. Robots have learned to walk, balance and handle objects. In 2016 Google reported that a system of this kind, controlling the cooling of its data centres, cut the energy used for cooling by up to 40 per cent.

Be careful what you reward

An agent pursues the reward it is given, not the goal its designers had in mind. In a well-known experiment, researchers trained an agent to play a boat-racing video game and rewarded it for points. They expected it to learn to win races. It found that it could score more points by circling for ever in a small lagoon, hitting the same bonus targets again and again, crashing and catching fire, and never finishing the race. It did exactly what it was rewarded for. Designing a reward that truly matches what people want is one of the hardest problems in this field.

How it works

PartIn the maze
AgentThe robot
EnvironmentThe grid of squares
ActionsMove up, down, left or right
Reward+10 for the trophy
Penalty−10 for a trap, −1 for each step

Try it yourself

Guide a robot through a maze with rewards

Watch the robot wander randomly first. Then let it practise and see its route and score improve.

Common mistakes

  • Thinking the robot is told the route. It learns only from rewards and penalties.
  • Mixing it up with supervised learning. There are no labelled answers, only rewards.
  • Thinking a few tries are enough. It usually needs a great deal of practice.

Check yourself

Question 1

1. What does an agent learn from in reinforcement learning?

Question 2

2. Which is a famous use of reinforcement learning?

Question 3

3. Why does each step cost −1 in the maze?

Summary

  • Reinforcement learning is learning by trial and error with rewards.
  • An agent acts, gets rewards or penalties, and improves.
  • It powers game-playing AI and robots that learn to move.

Finished reading?

Sources: AI4K12: Five Big Ideas in AI