In order to solve problems, we humans instinctively resort to analogies: we approach a new problem by recalling similar, already known questions and try to transfer their solution to the new situation. This “analog thinking”, i.e. the ability to draw logical conclusions from similarities, is an important part of the human thinking apparatus. Researchers now want to have found out that large language models such as GPT also have this ability.
Advertisement
A team from the University of California, Los Angeles (UCLA) wanted to find out whether artificial intelligence could solve tasks and problems that it hadn’t encountered before. The study was published in the journal “Nature Human Behaviour”.
For the study, the team led by brain and AI researcher Taylor Webb had around 50 UCLA students compete against the GPT-3 language model in two test rounds (the current GPT-4 was not yet available at the start of the study). The aim was to solve three task blocks that require analogue thinking. The tasks were created specifically for the study so that they do not appear in the training data of the AI, but are based on standardized tests such as those used for entrance exams at US universities or intelligence tests.
GPT outperforms students
The first block of tasks involved solving so-called progressive matrices. The subjects and the AI had to understand a text sample presented to them and correctly select the missing part from a range of possible parts. In the second block, they had to add rows of letters, i.e. derive from the given sequence abcdabce what follows from ijkl. In the third block, the task was to add word sequences (e.g. ‘love’ is for ‘hate’ what ‘rich’ is to ‘?’) and to draw content-related analogies from short stories, i.e. the correct causal connections between several texts understand.
In the first three tasks, GPT-3 consistently performed better than the students. The biggest difference was in the matrices that the language model correctly solved with an 80 percent hit rate, while the human participants only managed just under 60 percent. In the case of the letter and word sequences, the lead of the AI was smaller overall, but still significant.
In a comparison of content, however, the students came out on top: When it came to drawing causal analogies from stories, the success rate was over 80 percent. In this case, GPT-3 only achieved around 70 percent. The more complex the story, the more trouble the AI had in relating it to others. At least if she wasn’t specifically pointed out.
Advertisement
Clever for some problems, stupid for others
The researchers write that the results suggest that GPT-3 “has developed an abstract notion of succession.” This is not surprising, because human language is full of analogies, and when the large language models are trained to reproduce human language as faithfully as possible, they are given the ability to draw logical conclusions on the fly.
At the same time, the team points out various limitations. The software didn’t always recognize what it should do with the presented problems and had to be prompted to do so.
“As impressive as our results are, it’s important to emphasize that this system has significant limitations. While it can make analogous reasoning, it fails at tasks that humans find very easy,” says study leader Taylor Webb. For the moment anyway. Because preliminary tests with GPT-4 already indicate that the new language model delivers significantly better results.
(jl)
To home page