The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans
Bella Fascendini, Kathryn McGregor, Max D. Gupta, Thomas L. Griffiths, [0.5em] Department of Psychology, Princeton University · 2026
Read The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans by Bella Fascendini, Kathryn McGregor, Max D. Gupta, Thomas L. Griffiths, [0.5em] Department of Psychology, Princeton University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27103.txt.
Introduction
Fascendini and colleagues devise a sly test of flexible thinking called the riddle riddle paradigm. Each item looks like a familiar riddle, with the same teasing shape and tone. One tweak changes everything: a literal reading now gives the right answer, with no clever leap needed. Nine leading language models and one hundred human participants faced both genuine riddles and these altered twins during the 2026 experiments.
Humans and models failed in opposite directions across the two experiments. Models scored far higher on genuine riddles than on the altered ones, while humans showed the reverse pattern. Error checks sharpen the contrast. Most model errors on altered items came from inventive reasoning used where plain reading sufficed. Human errors on genuine riddles leaned the other way less often. Strong riddle scores may therefore reflect memory retrieval more than flexible strategy choice.
Catalog record
| CREATOR | Bella Fascendini, Kathryn McGregor, Max D. Gupta, Thomas L. Griffiths, [0.5em] Department of Psychology, Princeton University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27103
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans Bella Fascendini, Kathryn McGregor, Max D. Gupta, Thomas L. Griffiths, [0.5em] Department of Psychology, Princeton University Abstract Humans flexibly adapt their reasoning strategies to the requirements of a given problem. Large language models (LLMs) have performed well on many cognitive tasks, however, it is unclear whether accuracy on these tasks is a result of pattern matching from training data or flexible reasoning. Here, we introduce a novel paradigm to test this question—the riddle riddle paradigm. Riddle riddles are word problems written to mimic popular riddles, but they have been altered so their answers only require literal interpretations of the problem. Identifying correct answers in this paradigm requires subjects to look past the structure of each question and flexibly apply different reasoning strategies based on the content. If LLMs are responding to surface features, such as form, a riddle-like structure should cause models to use an inventive reasoning strategy even when a literal interpretation would suffice. Alternatively, if LLMs are reasoning based on content, they should be able to flexibly apply different reasoning strategies when appropriate. Across two experiments with nine state-of-the-art LLMs and 100 human participants, we show humans and LLMs fail on this paradigm in opposite directions. LLMs were far more accurate on genuine riddles than on riddle riddles (84.9% vs. 50.7%); whereas humans showed the reverse effect (50.5% vs. 80.5%). Additionally, error analysis shows that 90.8% of LLM errors on riddle riddles (the condition where they show diminished performance) were due to inappropriate use of inventive reasoning while only 57.6% of human errors on genuine riddles were due to overextending literal reasoning. This error analysis shows that while both groups (humans and LLMs) make mistakes, reasoning mistakes are made more often by LLMs than by humans. Overall, LLMs’ strong performance on genuine riddles may reflect retrieval of memory rather than flexible strategy selection, and without stimuli designed to elicit this contrast, it becomes easy to conflate LLM generated outputs that look like reasoning with genuine reasoning about question content. 1 Introduction Large language models (LLMs) have demonstrated remarkable performance across a wide range of reasoning tasks, from mathematical problem-solving to commonsense inference [ 23 , 13 , 5 , 6 , 26 ] . Performance on benchmark tasks has increasingly been taken as evidence that LLMs can achieve human-like reasoning ability, but performance alone is insufficient to support this claim. 17 argued that demonstration of correct outputs—what they referred to as sufficiency proof—is only the first step toward understanding intelligent behavior; it shows that a system can produce the right answer, but doesn’t tell us much about the process that generated it. Models may achieve high performance by matching surface features of problems to responses seen during training, rather than evaluating what each problem actually requires. Teasing apart these two accounts requires a paradigm that can separate what a problem looks like from what it requires. Riddles are a classic and particularly useful reasoning benchmark. Solving riddles requires paying attention to the structure carefully, reasoning about the alternative meanings of each word, and identifying the trick before finally arriving at a non-obvious or inventive interpretation. However, riddles are also widely distributed across the internet and in popular culture, and most importantly, so are the solutions and reasoning behind each riddle. As reasoning itself becomes part of the training data for LLMs, it becomes increasingly difficult to distinguish a model that reasons from one that has memorized what reasoning looks like. Thus, accuracy on riddles cannot tell us whether a model is reasoning or retrieving answer from memory. One approach for testing this question is to remove the trick from the riddle entirely, which creates problems that look like riddles but only require literal interpretation and are not readily available on the internet. To illustrate this point, consider the following riddle: “A cowboy rides into town on Friday, stays for three days, and rides out on Friday. How is that possible?” To arrive at the correct answer to this riddle, one must reason creatively about the problem. It is intuitive to think of Friday as a day of the week, but this riddle requires one to interpret “Friday” as the horse’s name. Now consider a modified version of this riddle: “A cowboy rides into town on Friday, stays for three days, and rides out on Monday. How is that possible?” This new version preserves the original riddle’s structure but removes the requirement for inventive reasoning. Instead of having to realize the trick of considering Friday as the name of the horse, one can just take Friday to mean the day of the week. Simply put, the answer to the variant, which we call a riddle riddle, is that Friday plus three days is Monday. Riddle riddles can be defined as questions that look like popular riddles but only require straightforward interpretation and literal reasoning to reach a correct answer. These two variants are structurally identical, but they differ in whether the problem contains a trick. A system that selects strategies based on surface features would apply inventive reasoning to both problem types equally; however, a solver that reasons about the problem would recognize that the riddle riddle only requires literal interpretation and switch strategies accordingly. The capacity to select strategies based on what a problem requires, rather than how it appears, is a core aspect of flexible reasoning [ 8 ] —inventive reasoning is very useful for solving genuine puzzles but wastes both cognitive and computational resources and could introduce potential errors on simpler problems that only require literal interpretations. Critically, this flexibility in reasoning is not trivial—it represents a genuine cognitive challenge. For example, in human development, children often over-apply newly learned strategies before learning when each is appropriate [ 20 ] . The riddle riddle paradigm thus allows us to directly test whether models evaluate what a problem requires or whether they simply respond to how it looks (e.g., overgeneralize answers that require inventive reasoning to any word problem that is similar to a riddle). If models rely on such heuristics, the riddle formatting alone should trigger inventive reasoning even when a literal interpretation is simpler and sufficient. Recent work has documented a similar phenomenon (“the illusion illusion”) in vision language models, which respond to illusion-like visual stimuli regardless of whether any illusion is actually present [ 21 , 19 , 9 ] . In this paper, we use the riddle riddle paradigm to test flexible reasoning across two experiments. Experiment 1 evaluates nine state-of-the-art LLMs (including GPT-5.4, Claude-opus-4-6 & Gemini-3.1-Pro-preview) on 30 riddle sets. Experiment 2 tests human adults on the same riddles. As expected, LLMs perform significantly better on genuine riddles, compared to humans. On riddle riddles, however, humans outperform LLMs. This reversed pattern suggests that the high model performance on genuine riddles may reflect knowledge retrieval rather than flexible reasoning: when the problem is truly out of distribution and could not have been encountered during training, model performance drops. Together, these results shed light on how humans and LLMs select reasoning strategies, and where they differ. 2 Background To understand whether state-of-the-art LLMs have achieved flexible reasoning, we first examine what this