BIG-Bench Extra Hard
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, Orhan Firat · 2025
Read BIG-Bench Extra Hard by Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, Orhan Firat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19187.txt.
Introduction
The standard hard test for language model reasoning has gone soft: top models now ace nearly all of it. Mehran Kazemi and co-authors replace it with a sterner exam in this 2025 paper. BIG-Bench Extra Hard swaps every old task for a new one probing the same skill at far greater difficulty, keeping the breadth that made the original useful.
The results are humbling. The best general model averaged under ten percent accuracy across the suite. Tasks span many styles of reasoning beyond the math and code that dominate other benchmarks, which is the point: a single framework for general reasoning. The authors present the suite as headroom for the next years of model progress, and as evidence that fluent answers still hide shallow thought on tasks humans find merely tricky.
Catalog record
| CREATOR | Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, Orhan Firat |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-19187
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BIG-Bench Extra Hard Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, Orhan Firat Abstract Large language models (LLMs) are increasingly deployed in everyday applications, demanding robust general reasoning capabilities and diverse reasoning skillset. However, current LLM reasoning benchmarks predominantly focus on mathematical and coding abilities, leaving a gap in evaluating broader reasoning proficiencies. One particular exception is the BIG-Bench dataset, which has served as a crucial benchmark for evaluating the general reasoning capabilities of LLMs, thanks to its diverse set of challenging tasks that allowed for a comprehensive assessment of general reasoning across various skills within a unified framework. However, recent advances in LLMs have led to saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH). State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus diminishing its utility. To address this limitation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty. We evaluate various models on BBEH and observe a (harmonic) average accuracy of 9.8% for the best general-purpose model and 44.8% for the best reasoning-specialized model, indicating substantial room for improvement and highlighting the ongoing challenge of achieving robust general reasoning in LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh . 1 Introduction Figure 1 : Model performances on BBEH (harmonic mean over individual task performances). Recent research has made significant strides in evaluating the reasoning capabilities of large language models (LLMs), but the focus has been disproportionately skewed towards math/science and coding. This emphasis is likely driven by the availability of challenging benchmarks in these domains and the relative ease of evaluating quantitative solutions. However, reasoning encompasses a far broader spectrum of cognitive skills, including logical deduction, temporal and spatial understanding, commonsense reasoning, and even the ability to comprehend humor. † † Llama and Qwen experiments in this paper were conducted only by parties outside of Google. To assess these diverse facets of reasoning, the community has relied on the BIG-Bench benchmark ( Srivastava et al., 2022 ) , specifically its more challenging subset, BIG-Bench Hard (BBH) ( Suzgun et al., 2022 ) . BBH has served as the de facto standard for evaluating general reasoning in LLMs due to its versatility and the wide array of reasoning skills it probes. However, the rapid advancements in LLM development has led to a saturation of BBH, with state-of-the-art models achieving over 90% accuracy. This performance ceiling renders BBH less effective in discriminating between the reasoning abilities of the latest generation of LLMs. To address this challenge and push the boundaries of LLM evaluation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to assess advanced reasoning capabilities. BBEH builds upon BBH by replacing each of the 23 tasks from BBH with a novel counterpart that probes similar reasoning capabilities, but exhibits significantly increased difficulty. Solving the tasks in BBEH requires even further reasoning skills than the problems in BBH. These skills include, but are not limited to, many-hop reasoning, learning on the fly, finding errors in reasoning traces, processing long-context inputs and finding (multi-)needles in a haystack, going against strong prior, dealing with long-range dependencies, dealing with distractors and inducing patterns from examples. By presenting LLMs with problems that demand a wider array of reasoning capabilities, BBEH aims to provide a more accurate measure of their general reasoning abilities. We provide a comprehensive evaluation of several state-of-the-art LLMs on BBEH. Our results demonstrate that BBEH presents a significant challenge even for the most advanced models. Specifically, we observe a ceiling accuracy of 23.9% for the best general-purpose model and 54.2% for the model reasoning-specialized model, highlighting the substantial headroom for improvement in robust general reasoning for LLM. Furthermore, we conduct a detailed failure analysis, revealing intriguing failure modes for both general-purpose LLMs and models specifically designed for reasoning and thinking. These findings offer valuable insights into the current limitations of LLMs and provide guidance for future research aimed at enhancing their reasoning capabilities. 2 Related Work There has been significant emphasis on using LLMs for mathematical and scientific reasoning. This has led to the popularity and proliferation of math- and STEM-based evaluations, such as ( Hendrycks et al., 2020 ; Cobbe et al., 2021 ) , and more recently, ( Glazer et al., 2024 ; Phan et al., 2025 ) . However, the generalizability of mathematical reasoning skills to broader domains remains unclear. Indeed, attempts to make existing benchmarks more robust—for example, Mirzadeh et al. (2024) —have highlighted an overall lack of robustness and logical reasoning capabilities. Several benchmarks have also been developed to address specific areas of reasoning, including temporal reasoning ( Xiong et al., 2024 ; Beniwal et al., 2024 ; Dhingra et al., 2022 ) , spatial understanding ( Bohnet et al., 2024 ; Yamada et al., 2023 ; Mirzaee et al., 2021 ; Shi et al., 2022 ) , commonsense reasoning ( Zellers et al., 2019 ; Talmor et al., 2018 ; Sakaguchi et al., 2021 ) , and logical reasoning ( Saparov and He, 2022 ; Tafjord et al., 2020 ; Saparov et al., 2023 ; Parmar et al., 2024 ) . However, these benchmarks tend to focus narrowly on specific domains, leading to potential evaluation biases if a more holistic view of model capabilities is not considered. To address this limitation, several benchmarks have been developed to integrate multiple tasks into a single evaluation framework, including ( Wang, 2018 ; Wang et al., 2019 ; Weston et al., 2015 ; Lu et al., 2023 ; Kazemi et al., 2024 ; Hendrycks et al., 2020 ; Wang et al., 2024 ; Parmar et al., 2024 ; Srivastava et al., 2022 ) . Our work builds on this line of research, introducing a new set of challenging tasks for future model evaluation and performance improvement. The multi-task nature of our benchmark with fine-grained tasks each focused on some reasoning skills enables model developers to discover and analyze failure modes in further depth. Note that while private initiatives such as ChatBot Arena ( lmarena, ) and the SEAL leaderboard ( ScaleAI, ) conduct model evaluations across various aspects, they may suffer from several potential issues as pointed out in Bansal and Maini (2024) . Our benchmark provides an open evaluation framework with an automatic and deterministic scoring mechanism, ensuring full transparency and reproducibility for the broader research community. 3 Going Beyond BIG-Bench Hard BIG-Bench, and its later subset BIG-Bench Hard (BBH), have served the research community for several years as de facto benchmarks for measuring general reasoning capabilities. Here, we mainly focus on BBH which has been favored in the recent years. We believe three key factors in the success of BBH were the following: 1- it was challenging even for the frontier models at the time, thus helping reveal model shortcomings and areas for improvement, 2- despite being composed of various tasks, all the questions in BBH were