ErrorMap and ErrorAtlas : Charting the Failure Landscape of Large Language Models
Shir Ashury-Tahan, Yifan Mai, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen · 2026
Read ErrorMap and ErrorAtlas : Charting the Failure Landscape of Large Language Models by Shir Ashury-Tahan, Yifan Mai, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15812.txt.
Introduction
Shir Ashury-Tahan and four co-authors map why language models fail in a 2026 arXiv paper. Benchmarks tell us when models fail, but not why. A wrong answer on a reasoning set can come from many causes. Formatting issues, calculation errors, or dataset noise may be to blame. Weak reasoning is only one suspect. Without separating causes, benchmarks cannot guide improvement well.
Their tools are ErrorMap and ErrorAtlas, per the title. They chart the failure landscape of large models. A chart of causes points builders to the right fix. Fixing format is cheap; fixing reasoning is not. Misdiagnosed failures waste effort on the wrong cure. For evaluation science, this paper pushes past scores toward diagnosis.
Catalog record
| CREATOR | Shir Ashury-Tahan, Yifan Mai, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15812
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ErrorMap and ErrorAtlas : Charting the Failure Landscape of Large Language Models Shir Ashury-Tahan, Yifan Mai, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen Abstract Large Language Models (LLM) benchmarks tell us when models fail, but not why they fail. A wrong answer on a reasoning dataset may stem from formatting issues, calculation errors, or dataset noise rather than weak reasoning. Without disentangling such causes, benchmarks remain incomplete and cannot reliably guide model improvement. We introduce ErrorMap , the first method to chart the sources of LLM failure. It extracts a model’s unique “failure signature”, clarifies what benchmarks measure, and broadens error identification to reduce blind spots. This helps developers debug models, aligns benchmark goals with outcomes, and supports informed model selection. ErrorMap works on any model or dataset with the same logic. Applying our method to 35 35 datasets and 83 83 models we generate ErrorAtlas , a taxonomy of model errors, revealing recurring failure patterns. ErrorAtlas highlights error types that are currently underexplored in LLM research, such as omissions of required details in the output and question misinterpretation. By shifting focus from where models succeed to why they fail, ErrorMap and ErrorAtlas enable advanced evaluation --- one that exposes hidden weaknesses and directs progress. Unlike success, typically measured by task-level metrics, our approach introduces a deeper evaluation layer that can be applied globally across models and tasks, offering richer insights into model behavior and limitations. We make the taxonomy and code publicly available 1 1 1 https://github.com/IBM/ErrorMap , with plans to periodically update ErrorAtlas as new benchmarks and models emerge. † † affiliationnotice: * Equal supervision. “ It is possible to fail in many ways… while to succeed is possible only in one way. ” Aristotle, Nicomachean Ethics , Book II, \sim 320BC Figure 1 : ErrorMap Schema: Two Method Stages. The first stage performs a one-time analysis of each incorrect prediction at the instance level. The second stage constructs a layered taxonomy through recursive iterations, based on either the maximum grouping depth or a defined layer threshold. 1 Introduction Benchmarking plays a central role in advancing large language models (LLMs), offering a standard bottom-line score to assert progress ( Biderman et al. 2024 ) . This abstraction eases proving a model’s success or its overall superiority, but it also obscures the nature and origin of model errors, complicating skill comparisons and hindering efforts to diagnose limitations or guide improvements. In response to these limitations, there is growing interest in developing more interpretable and diagnostic evaluation frameworks ( Maimon et al. 2025 ; Zeng et al. 2025 ; Tjuatja & Neubig 2025 ) or in highlighting specific errors models make ( Mukherjee et al. 2025 ; Pan et al. 2025 ; Li et al. 2024 ; Honovich et al. 2022 ; Kryscinski et al. 2019 ) . While current diagnostic methods offer valuable insights, their analysis primarily relies on the challenges posed by the input (e.g., counting failures on differential equations questions). However, where success on a challenge necessarily proves competence, a failure can have many causes (e.g., misunderstanding the question, miscalculation or applying a wrong axiom). Moreover, benchmark examples themselves may introduce ambiguity or error, further complicating evaluation, especially when the model’s answer is not considered. Ultimately, pinpointing the cause of failure requires analyzing both the input, including the question and the instruction, and the resulting answer. We introduce ErrorMap (§ 2 ) to address this gap. ErrorMap offers a model-oriented error analysis, highlighting why models fail. The method transforms raw language model failures into a structured, interpretable taxonomy in natural language. To do so, ErrorMap follows a pipeline, first profiling the issue underlying each failure, then recursively generates taxonomy categories through an iterative refinement process, and finally applies the constructed taxonomy to all failures (see Figure 1 ). As our analysis requires simple unstructured text, it applies seamlessly to any language model and domain. Overall, ErrorMap provides a flexible way to analyze a practitioner’s specific setting or to compare the failure fingerprint of several models or datasets. While a dynamic taxonomy that would fit an ad hoc analysis is best for many needs, a stable taxonomy simplifies comparisons across time and replicability. Applying ErrorMap to 83 models and 35 datasets, we release ErrorAtlas – a taxonomy of current LLM failures (see § 3 ). ErrorAtlas is a comprehensive and static taxonomy of model errors designed to facilitate cross-field comparisons, enhance efficiency, and ensure replicability. We present several findings on common model failures, which both stand on their own and highlight the effectiveness of our methods. Applying the ErrorAtlas taxonomy to current models and datasets we find issues that are prevalent but understudied (§ 4 ). For example, models often misinterpret the question’s intent or provide incomplete answers. Beyond ErrorAtlas , ErrorMap can support practitioners throughout the LLM lifecycle, from development and fine-tuning to evaluation and benchmarking. We provide in § 5 two test cases, a model developer diagnosing the differences between two versions of Gemini ( Team et al. 2024 ) and a benchmark curator testing MMLU-pro ( Wang et al. 2024a ) . Furthermore, in § 6 , we validate that the stages involved in extracting the error analysis are accurate, robust and cover well the errors models make. This work advances our ability to understand and analyze model behavior at a deeper level. Our contributions are as follows: 1. We introduce ErrorMap , an LLM-based technique to generate a dedicated taxonomy of LLM errors. It enables analysis across a diverse set of domains, input formats and model comparisons. 2. We present ErrorAtlas , a static taxonomy of model errors, generated using ErrorMap . It captures common failure modes across benchmarks and models. These errors reflect underlying limitations in model behavior, supporting meaningful and interpretable comparisons of model weaknesses. 3. We provide analysis across a large number of models and datasets. Finding common errors that are understudied. Moreover, we find model versions, types, and families exhibit distinct error patterns, allowing for nuanced behavioral profiling and more targeted evaluation. 4. We demonstrate the applicability of both ErrorMap and ErrorAtlas for nuanced model comparison, benchmark analysis, and model debugging. 5. We publicly release the code, taxonomy and associated data. 2 ErrorMap Our technique targets a common scenario: evaluating multiple models on the same dataset examples, as typically done in benchmark runs, or even reusing a benchmark run for deeper analysis. It leverages all available data in the benchmark, including inputs, reference answers, and model outputs, and produces comparative insights across models. The process is unsupervised and consists of two stages. First, incorrect predictions are analyzed on a per-example basis to form the leaf nodes of the taxonomy, creating the initial layer. Second, the taxonomy is constructed top-down through recursive process. The method schema overview is shown in Figure 1 , and additional details, including the specific prompts used, are provided in Appendix A . Stage 1: Per-Instance Error Analysis Our goal at this stage is to create a structured summary of the resulted errors and provide interpretable analysis of it. To achieve this, we task an analyst LLM with performing a detailed, structured analysis for each incorrect prediction. This includes evaluating a list of criteria with associated features, providing