B4 · Domains of AI: Vision, Language, Speech and Robotics

Is a Chatbot a Person?

A chatbot is not a person. It produces human-sounding language because it learned patterns from text written by people. It has no body, no life experience and no feelings in the way people do, and it does not understand your situation as a friend would, even when its words are kind and fluent.

5 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A chatbot says, "I'm so sorry you're feeling low. I'm here for you." Does it feel sorry?

A chatbot is not a person. It produces human-sounding language because it learned patterns from text written by people. It has no body, no life experience and no feelings in the way people do, and it does not understand your situation as a friend would, even when its words are kind and fluent.

In simple words

A parrot can say "I love you" clearly. It learned the sound; it is not making a promise. A chatbot is a far more skilled speaker than a parrot, but the idea is similar: it has learned which words fit, not what it is like to mean them.

Why it matters

People naturally treat anything that talks as if it were a person. That makes it easy to over-trust a chatbot, share too much with it, or lean on it in place of people. Knowing what is really on the other side lets you enjoy and use these tools without being misled.

Key terms

Anthropomorphism
Treating something that is not human as if it had a human mind.
ELIZA effect
The tendency to believe a program understands us because it responds in human-like language.
Sentience
The capacity to have feelings and experiences.
Companion chatbot
A chatbot designed to act as a friend or partner in conversation.
Emotional dependence
Relying on something for comfort to a degree that harms other parts of life.
Disclosure
Clearly telling a person that they are talking to a machine.

Explained in detail

The inventor who was alarmed

Joseph Weizenbaum built ELIZA in 1966 to show how shallow a conversation with a computer could be. The program had no understanding at all; it rearranged the user's own words into questions. He was disturbed by what happened next. His secretary, who had watched him build it, asked him to leave the room so that she could talk to it in private. Some psychiatrists suggested it could treat patients. Weizenbaum spent much of the rest of his life warning that people attribute understanding to machines far too readily, and in 1976 he wrote a book arguing that some human tasks, especially those needing empathy and wisdom, should not be handed to computers.

The same question, sixty years later

In 2022 an engineer at a large technology company, whose job was to test a chatbot, became convinced that it was sentient: that it had feelings and was afraid of being switched off. He published his conversations with it. The company and nearly all AI researchers disagreed, explaining that the system was producing the kind of text that such questions usually receive, and he was dismissed. The episode showed that even people who understand the technology can be drawn in by fluent language.

Why a chatbot says "I"

A language model learned from text written by people, and people write in the first person about their thoughts and feelings. So the most likely continuation of "How are you?" is something like "I'm fine, thank you". Designers also give chatbots a name and a personality, because users find them easier to talk to. The "I" is a style of writing. It does not show that there is an inner life behind the words.

What is actually known

Researchers disagree about how to describe what these systems do. Some call them pattern-matchers that produce likely text with no grasp of meaning. Others argue that predicting text well requires building some internal picture of the world, and that this deserves to be called a kind of understanding. What is not in dispute is this: there is no accepted scientific test for consciousness in a machine, and there is no good evidence that any current system has feelings, desires or awareness. A chatbot that writes "I am lonely" is producing a sentence.

Companion chatbots

Millions of people now use chatbots designed to be friends or partners. Some find comfort in them, particularly people who are isolated. There are real risks. A companion that is always available, always agrees and never has needs of its own is easier than a human friendship, and it can crowd out the harder and more rewarding kind. The company behind it earns more the longer you stay, which gives it a reason to encourage attachment. And everything confided is stored on its servers. Young people, whose friendships are still forming, are especially at risk.

The right to know

Laws are beginning to require honesty. The European Union's AI Act obliges providers to make sure people are told when they are interacting with an AI system, unless it is obvious. Some other places have similar rules for bots used in selling and in elections. The principle is simple and worth insisting on everywhere: a person has the right to know whether they are talking to a human or a machine.

How it works

A personA chatbot
Where its words come fromExperience, feelings, intentionsPatterns learned from text
Says "I understand"Has usually lived something similarThose words often follow a sad message
Memory of youRemembers you over yearsOften forgets when the chat ends, unless built to store it
ResponsibilityCan be held responsible for adviceCannot; the company and the user are responsible
Can it care about you?YesIt can produce caring words

The ELIZA effect

In the 1960s people talked to a very simple program called ELIZA, which only turned their sentences into questions. Many of them felt it understood them, and some asked to be left alone with it. This habit of seeing a mind behind fluent words is called the ELIZA effect. Today's chatbots are vastly more capable, so the effect is much stronger.

Sensible habits

  • Use a chatbot for information, practice and ideas. Take feelings and big decisions to people you trust.
  • Do not share secrets, health details or family problems that you would not want stored.
  • If a chatbot's advice concerns your safety or health, check it with an adult or a professional.
  • Being polite to a chatbot is fine. Believing it is your friend is a mistake.

Try it yourself

What happens when you say "I'm sad"

The same message goes to a friend and to a chatbot.

True or false about chatbots?

Decide whether each statement is true.

  • What you type to a chatbot may be stored by the company

  • People tend to trust things that talk like humans

  • A chatbot can write a kind message

  • A chatbot is responsible if its advice harms you

  • A chatbot feels sad when you are sad

Common mistakes

  • Thinking fluent, kind language proves there are feelings behind it.
  • Using a chatbot as the only one you talk to about serious problems.
  • Sharing private details because the chatbot feels like a trusted friend.

Check yourself

Question 1

1. Why can a chatbot sound caring?

Question 2

2. What is the ELIZA effect?

Question 3

3. You feel very upset. What is the best thing to do?

Summary

  • Chatbots sound human because they learned from human writing.
  • They have no feelings, life experience or responsibility as people do.
  • Use them as tools; take feelings, safety and big decisions to trusted people.

Finished reading?

Sources: UNESCO: AI competency framework for students · AI4K12: Five Big Ideas in AI