ChatGPT Tackles Ancient Plato Puzzle With Human Like Flaws

In a fascinating intersection of ancient philosophy and modern technology, researchers at the University of Cambridge subjected ChatGPT-4 to a 2,400-year-old math problem, revealing unexpected behavior that mimics a human learner, complete with improvisation and plausible errors.

An Ancient Mathematical Challenge

The problem, known as "doubling the square," was first described by Plato around 385 BCE. In his dialogue, Socrates guides an uneducated boy to double a square's area. The boy's first instinct is to double the side length, a common mistake that actually quadruples the area. Through a series of guided questions, Socrates helps the boy discover the correct solution: the new square's sides must equal the length of the original square's diagonal. This puzzle has long been a key lesson in mathematics and a topic of philosophical debate about the origins of knowledge.

ChatGPT Takes the Test

Researchers Dr. Nadav Marco and Professor Andreas Stylianides put this same challenge to ChatGPT-4. They posed a series of questions in the Socratic style, progressively introducing errors and new variations of the problem. Their central question was whether the AI would simply recall the classic solution from its training data or generate a novel approach.

A Surprising Algebraic Approach

Despite its training on classical texts, ChatGPT-4 did not immediately provide the expected geometric solution. Instead, it initially opted for an algebraic method, a technique that was not known in Plato's time. The chatbot resisted the researchers' attempts to steer it toward the classical geometric answer. It was only when the researchers expressed their “disappointment” that the AI produced the alternative geometric solution, proving it had knowledge of it all along.

"If it had only been recalling from memory, it would almost certainly have referenced the classical solution...straight away," Stylianides explained. "Instead, it seemed to take its own approach."

Human-Like Errors and Improvisation

The team further tested the AI by asking it to double the area of a rectangle and a triangle. In both instances, ChatGPT again favored an algebraic method. When pushed on the rectangle problem, it made a distinctly human-like error, mistakenly claiming that no geometric solution was possible, even though several exist. The researchers believe this wasn't a failure of its knowledge base but an improvised guess based on their previous conversation.

However, after more prompting on the triangle problem, the chatbot eventually provided a correct geometric answer. This mix of improvisation, error, and eventual correction is what the researchers found so compelling.

Implications for AI and Education

The study, published in the International Journal of Mathematical Education in Science and Technology, concludes that ChatGPT’s behavior appears to be a blend of data retrieval and “on-the-fly reasoning.” The researchers compare its performance to the educational concept of the “zone of proximal development” (ZPD)—the gap between what a learner knows and what they can achieve with guidance.

This suggests that AI's limitations can be transformed into learning opportunities. The team recommends that students use collaborative prompts, such as “Let’s explore this problem together,” instead of just asking for an answer. This method encourages the development of critical thinking and reasoning skills, using the AI as a tool for exploration rather than a simple answer machine.