Does ChatGPT Understand What It Writes?

AI text generators like ChatGPT are incredible tools. They can summarize and expand source text, write poems and essays, and even program. But as with any tool, it’s important to ask what AI text generators can and cannot do.

It’s one thing to list out tasks that ChatGPT is good at. It’s another to understand why AI text generators have the limitations they do. To answer that, we need to ask something a bit more abstract: does ChatGPT understand what it writes?

No, it doesn’t. Here’s why it matters.

How does ChatGPT Work?

ChatGPT is an extremely sophisticated text predictor. It’s “trained” on mountains of data, and uses statistics to crunch numbers and predict which word is most likely to come next in a sentence. By repeating this predictive process, ChatGPT can generate sentences, paragraphs, and even whole essays.

But isn’t that like human language learners? After all, we too learn language over time, and get progressively better at crafting ever more intricate sentences and trains of thought. We too read texts, helping us understand style and word usage. If ChatGPT and human language users are doing the same sort of thing, then complete automation of content creation could be just around the corner.

How Do Humans Work?

The thing is, humans are more than just text predictors. Just ask the father of modern linguistics, Noam Chomsky.

In a New York Times op-ed, Chomsky & his co-authors explain that humans have a kind of “innate, genetically installed ‘operating system’ that endows humans with the capacity to generate complex sentences and long trains of thought.”

This isn’t impressionistic hand-waving about human creativity. Proponents of so-called “generative linguistics” argue that humans have an evolved capacity to quickly learn and use human languages, which share certain structural features. Children pick up the structure of languages remarkably fast, even without much “training data”. By comparison, ChatGPT is trained on at least 300 billion words.

None of this is to deny the importance of recent advances in large language models (LLMs). Also note that this difference between human and computer language learning is not an argument that computers will never do what humans can. However, genuine understanding involves something qualitatively different than the super-charged statistics which constitutes machine learning.

Why ChatGPT doesn’t understand

ChatGPT predicts what word is statistically likely to come next. But that’s not how understanding works. To understand grammar, you need to know how sentences are built up in steps from smaller building blocks.

Consider the sentence

“Napoleon is short and Wellington is not green”.

To understand this, you need to see how it’s built up in stages. For example, you first combine the words “not” and “green” to form the property “not green”.

After you build up “Napoleon is short” and “Wellington is not green” you then combine them with the word “and” to make the full sentence.

AI text generators miss out on all of this. Here’s another example. By grasping language structure, humans can understand the logical consequences of sentences.

For example, from the premises

1. “Either Bill carries an umbrella or he risks getting wet.”

2. “Bill does not risk getting wet.”

I can infer the conclusion

3. “Bill carries an umbrella.”

ChatGPT might correctly output this consequence, but it will need a lot more textual data than an average human who simply understands how the words “Either”, “or”, and “not” work. In other words, there’s more to reasoning than mere prediction.

Syntax is just one facet of language ChatGPT doesn’t grasp. It also doesn’t understand context. If you ask me whether it’s a good idea to hire Bill, and I say “Well, he can walk and breathe at the same time”, you will probably pick up that I don’t think much of Bill’s abilities. ChatGPT won’t pick up on this, or if it does, it will only do so by training on text written by a human who did pick up on sarcasm of this sort.

Boiled down, the problem is simple. There are infinitely many potential sentences, which can be used in infinitely many potential contexts. AI text generators can effectively mimic human writing by training on massive amounts of text. But humans can use language more flexibly, and with far less data, due to an innate understanding of the shared structure that underlies human languages.

Why it matters

ChatGPT doesn’t understand what it writes. It has the impressive ability to seem as if it understands what it writes, but only by training on a vast corpus of material written by humans that (hopefully) know what they’re talking about.

This has some straightforward consequences.

• Since ChatGPT doesn’t understand what it writes, it has no conception of truth or falsity. This means that it needs a careful human editor to ensure it’s not “hallucinating”, i.e. generating nonsense or outright falsehoods.

• ChatGPT misses out on nuance. It can guess what sentence is likely to come next, but because it doesn’t understand context, it doesn’t know why that sentence is appropriate.

• This translates into a lack of flexibility. ChatGPT’s training set is huge, but the number of possible sentences is infinite. A writer who genuinely understands a topic will be a lot better at extending their writing to new topics and contexts not covered by training data.

AI text generators like ChatGPT are an amazing bit of technology, and their use cases will only multiply with time. But ChatGPT doesn’t understand the text it generates, and knowing why tells you something important about its limits.