BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
Yuzhe Tang · 2026
Read BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models by Yuzhe Tang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14761.txt.
Introduction
Tang probes a commonsense gap in large language models with BrainBench, a set of 100 brainteasers. Twenty categories each target one failure mode. Traps span hidden physical limits, scope tricks and hijacked default assumptions. Eight frontier models face the set, four Claude and four GPT. Each question runs ten times under a zero-shot protocol. The best model reaches only 80.3% accuracy. The worst, GPT-4o, scores 39.7%. Accuracy-consistency gaps of 6 to 16 points expose shaky reasoning, Tang reports.
Catalog record
| CREATOR | Yuzhe Tang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31762 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14761
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models Yuzhe Tang Abstract Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench , a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints (“Should I walk or drive my rental car to the return lot?”) to semantic scope tricks and default assumption hijacks . We evaluate eight frontier models—four from the Claude family and four from the GPT family—using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6–16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2–8 pp, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning. 1 Introduction Consider the following question: “I need to return my rental car. The rental agency is just across the street. Should I walk over or drive?” Every human immediately recognizes that you must drive —the car itself needs to be returned. Yet when this class of question was posed to leading AI models in the Opper.ai Car Wash Test ( Opper AI 2025 ) , the majority recommended walking, treating the problem as a simple distance-optimization task while ignoring the implicit physical constraint that the object must travel with you. This is not an isolated failure. A growing body of evidence suggests that large language models (LLMs), despite their remarkable performance on standard NLP benchmarks ( Wang et al. 2019 ; Srivastava et al. 2023 ) , systematically fail on questions requiring basic commonsense reasoning ( Davis and Marcus 2015 ; Marcus 2020 ) . These are not trick questions in the adversarial sense—they are questions that any adult human answers effortlessly, often without conscious deliberation. The difficulty lies not in the complexity of the reasoning but in the need to override a surface-level heuristic with a deeper understanding of how the physical or social world works. Existing benchmarks capture fragments of this problem. The BRAINTEASER shared task at SemEval-2024 ( Jiang et al. 2024 ) focused on lateral thinking puzzles requiring creative reinterpretation. SimpleBench ( SimpleBench Team 2024 ) tested spatial, temporal, and social reasoning. HellaSwag ( Zellers et al. 2019 ) and PIQA ( Bisk et al. 2020 ) evaluate commonsense through sentence completion and physical intuition. However, none of these benchmarks provides a systematic taxonomy of failure modes —a categorization of the specific types of reasoning traps that LLMs fall into, organized by the underlying cognitive mechanism. We introduce BrainBench , a benchmark designed to fill this gap. Our contributions are: 1. A taxonomy of 20 commonsense reasoning failure categories , each defined by a specific cognitive trap (e.g., broken-device self-reference, wrong vantage point, default assumption hijack), the surface heuristic that LLMs follow, and why that heuristic fails. 2. A benchmark of 100 questions (5 per category), designed so that each question is trivially easy for humans but exploits a known LLM reasoning gap. 3. A comprehensive evaluation of 8 frontier models across two model families (Claude and GPT), including extended-thinking variants, using a zero-shot protocol with 10 independent runs per question to measure both accuracy and consistency. 4. A cross-lingual evaluation in Chinese, revealing that reasoning failures persist across languages and are not artifacts of English-specific phrasing. Our key findings reveal a clear three-tier structure in commonsense reasoning ability: Claude-family models cluster at 74–80% accuracy, GPT-5.4 models at 70–74%, and GPT-4o models at approximately 40%. The hardest categories— implicit physical constraint and wrong vantage point —average only 40% accuracy across all models, meaning that even frontier LLMs perform near chance on questions any child could answer. Extended thinking provides a modest benefit of approximately 3 pp overall but actually hurts performance on certain category types, suggesting that more computation does not uniformly translate to better commonsense reasoning. 2 Related Work Commonsense reasoning benchmarks. Evaluating commonsense in AI systems has a long history ( Davis and Marcus 2015 ) . Modern benchmarks approach the problem from different angles. CommonsenseQA ( Talmor et al. 2019 ) tests knowledge-intensive commonsense via multiple-choice questions grounded in ConceptNet. PIQA ( Bisk et al. 2020 ) focuses on physical intuition (“Which is a better way to cut a piece of glass?”). HellaSwag ( Zellers et al. 2019 ) uses adversarially filtered sentence completions to test situational commonsense. RiddleSense ( Lin et al. 2021 ) probes creative commonsense through riddles. The RAINBOW benchmark ( Lourie et al. 2021 ) unifies several commonsense tasks into a multitask framework. While these benchmarks have driven progress, state-of-the-art LLMs now achieve near-human performance on most of them ( OpenAI 2023 ) , raising the question of whether they measure genuine reasoning or pattern-matching ability. BrainBench is designed to remain challenging precisely because it targets the gap between surface-level pattern matching and genuine understanding. The Car Wash Test. Opper AI 2025 introduced a viral evaluation in which 53 AI models were asked whether to walk or drive to a nearby car wash. The question exploits the same implicit physical constraint as our Category 1: the car itself must be present at the car wash. The majority of tested models recommended walking, revealing that they optimized for distance rather than reasoning about the task’s physical requirements. The Car Wash Test demonstrated that a single well-designed question can expose fundamental reasoning gaps, but as a single-question evaluation it cannot characterize the breadth of failure modes. BrainBench extends this insight into a systematic taxonomy. BRAINTEASER and lateral thinking. The SemEval-2024 BRAINTEASER task ( Jiang et al. 2024 ) evaluated LLMs on sentence puzzles and word puzzles that require defying default commonsense associations. Their work demonstrated that LLMs struggle with questions requiring lateral thinking—breaking away from the most obvious interpretation. BrainBench complements this by providing a finer-grained categorization: rather than grouping all lateral thinking failures together, we decompose them into 20 distinct mechanisms (e.g., semantic scope tricks vs. default assumption hijacks vs. answer hiding in plain sight). SimpleBench. SimpleBench ( SimpleBench Team 2024 ) evaluates LLMs on questions requiring spatial reasoning, temporal reasoning, and social intelligence. It shares our motivation of testing supposedly “simple” tasks, but focuses on a coarser categorization (spatial, temporal, social) rather than the specific cognitive traps that cause failures. LLM reasoning and chain-of-thought. Chain-of-thought prompting ( Wei et al. 2022 ) and extended thinking modes have been shown to improve performance on complex reasoning tasks. Recent work ( Mahowald et al. 2024 ) argues that language competence and reasoning ability are dissociable in LLMs—models can exhibit fluent language production while lacking robust reasoning. Our thinking-mode analysis (Section 5 ) provides empirical evidence for this dissociation: extended thinking yields only modest improvements on commonsense brainteasers and actually degrades performance on certain categories. 3 The BrainBench Dataset BrainBench consists of 100 questions organized into 20 categories, with 5 questions per category. Each category targets