My first post on this blog was a perspective on a very interesting paper by Webb et al., “Emergent Analogical Reasoning in Large Language Models”. Taylor Webb, the first author of that paper, responded by email to my perspective, and he gave me permission to post his response on this blog. Here is Taylor’s thoughtful response, which is well worth reading. I will make a few comments at the end.
I wonder whether such problems really test the reasoning capabilities of ChatGPT, or its capabilities to abstract. It is after all trained as a _language_ model. I tested the following and found the results remarkable - though designing analogies to the more complex abstract problems would be a fair bit of work:
Human:
Here is a riddle: Please fill in the blank. For Valentine's day, Marcy gives to Aaaron a feather, a nest, and an egg. Aaaron gives to Marcy a leaf, a flowerpot, and _________ .
ChatGPT:
Aaaron gives to Marcy a leaf, a flowerpot, and soil.
Human:
Here is a riddle: Please fill in the blank. Marcy and Aaaron like to exchange gifts. Marcy has a pawn, a rook, a peach, and a bishop. She gives to Aaaron as a gift the pawn, the rook, and the bishop. Aaaron has a spoon, a knife, a fork, and a book. He gives to Marcy as a gift _____________ .
ChatGPT:
Aaaron gives to Marcy as a gift the spoon, the knife, and the fork.
The version of ChatGPT used here was presumably GPT 3.5 based, and the "soil" answer seems merely a related word, rather than completing an analogy.
GPT-4 (Bing/Creative) gives the much better answer of "seed".
"That’s a tricky riddle. I think the answer is a seed. The items that Marcy and Aaaron give to each other are related to the life cycle of a plant and a bird. A feather, a nest, and an egg are part of a bird’s life cycle. A leaf, a flowerpot, and a seed are part of a plant’s life cycle."
I wonder whether such problems really test the reasoning capabilities of ChatGPT, or its capabilities to abstract. It is after all trained as a _language_ model. I tested the following and found the results remarkable - though designing analogies to the more complex abstract problems would be a fair bit of work:
Human:
Here is a riddle: Please fill in the blank. For Valentine's day, Marcy gives to Aaaron a feather, a nest, and an egg. Aaaron gives to Marcy a leaf, a flowerpot, and _________ .
ChatGPT:
Aaaron gives to Marcy a leaf, a flowerpot, and soil.
Human:
Here is a riddle: Please fill in the blank. Marcy and Aaaron like to exchange gifts. Marcy has a pawn, a rook, a peach, and a bishop. She gives to Aaaron as a gift the pawn, the rook, and the bishop. Aaaron has a spoon, a knife, a fork, and a book. He gives to Marcy as a gift _____________ .
ChatGPT:
Aaaron gives to Marcy as a gift the spoon, the knife, and the fork.
Nice.
The version of ChatGPT used here was presumably GPT 3.5 based, and the "soil" answer seems merely a related word, rather than completing an analogy.
GPT-4 (Bing/Creative) gives the much better answer of "seed".
"That’s a tricky riddle. I think the answer is a seed. The items that Marcy and Aaaron give to each other are related to the life cycle of a plant and a bird. A feather, a nest, and an egg are part of a bird’s life cycle. A leaf, a flowerpot, and a seed are part of a plant’s life cycle."