Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
Rongyuan Tan 1 Jue Zhang 2 Zhuozhao Li 1† Qingwei Lin 2 Saravan Rajmohan 2 Dongmei Zhang 2 · 2026
Read Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks by Rongyuan Tan 1 Jue Zhang 2 Zhuozhao Li 1† Qingwei Lin 2 Saravan Rajmohan 2 Dongmei Zhang 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17761.txt.
Introduction
Tan and five colleagues study contrastive, LRP-based attribution for analyzing LLM failures on realistic benchmarks, where prior work uses short prompts. Failure analysis becomes contrastive attribution: the gap between a wrong output token's logit and the right alternative's is traced to input tokens and internal states. An efficient extension builds cross-layer graphs for long-context inputs. Comparing patterns across datasets, model sizes, and checkpoints, token-level contrastive attribution yields informative signals in some failure cases. It is not universally applicable, showing its utility and limits.
Catalog record
| CREATOR | Rongyuan Tan 1 Jue Zhang 2 Zhuozhao Li 1† Qingwei Lin 2 Saravan Rajmohan 2 Dongmei Zhang 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96089 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17761
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks Rongyuan Tan 1 Jue Zhang 2 Zhuozhao Li 1† Qingwei Lin 2 Saravan Rajmohan 2 Dongmei Zhang 2 Abstract Interpretability tools are increasingly used to analyze failures of Large Language Models (LLMs), yet prior work largely focuses on short prompts or toy settings, leaving their behavior on commonly used benchmarks underexplored. To address this gap, we study contrastive, LRP-based attribution as a practical tool for analyzing LLM failures in realistic settings. We formulate failure analysis as contrastive attribution , attributing the logit difference between an incorrect output token and a correct alternative to input tokens and internal model states, and introduce an efficient extension that enables construction of cross-layer attribution graphs for long-context inputs. Using this framework, we conduct a systematic empirical study across benchmarks, comparing attribution patterns across datasets, model sizes, and training checkpoints. Our results show that this token-level contrastive attribution can yield informative signals in some failure cases, but is not universally applicable, highlighting both its utility and its limitations for realistic LLM failure analysis. Our code is available at: https://aka.ms/Debug-XAI . 1 Introduction While the capabilities of Large Language Models (LLMs) have improved substantially in recent years, their deployment in real-world scenarios still reveals a wide range of failures [ 12 ] . Systematically debugging these failures is therefore crucial, both for understanding their underlying causes and for identifying actionable directions for further model improvement. Existing analyses of LLM failures predominantly rely on behavioral analysis , which examines model inputs, outputs, and responses to perturbations. Such analyses are effective at identifying failures, categorizing error types, and measuring sensitivity to prompt variations, but they primarily characterize what fails rather than why failures occur. A key limitation of this paradigm is its inherent ambiguity: similar behaviors can arise from distinct internal causes, such as underweighting relevant context tokens, overweighting irrelevant tokens, or biases embedded in internal representations. This motivates interpretability-based analysis [ 42 ] , which moves beyond behavior to reveal how models form preferences, attribute decisions, and propagate influence across layers. On the interpretability side, substantial progress has been made for transformer-based models [ 20 ] . However, applications to LLMs have largely been limited to toy settings, short inputs, or highly constrained prompt templates [ 60 , 45 , 37 ] . Interpretability analyses of realistic failures, such as those arising from standard benchmarks or real-world deployments, remain scarce. This limitation is partly attributable to the open-ended nature of such failures and their long input contexts, which can span thousands of tokens (e.g., in agentic tasks) and exceed the computational limits of many existing interpretability tools. 1 1 1 For example, the TransformerLens package [ 47 ] sets the default context length to 2048 tokens for the Qwen3 model series [ 63 ] . Although recent work has begun to address scalability challenges in mechanistic interpretability [ 53 ] , direct applications of interpretability methods to analyze realistic LLM failure cases remain limited, leaving their practical utility insufficiently understood. Motivated by both the need for deeper failure debugging and the desire to establish the practical relevance of model interpretability, we investigate the use of interpretability methods for analyzing failures on standard benchmarks (e.g., GAIA2 [ 4 ] ) as a first step toward more realistic settings. We focus on the central research question: Can interpretability analysis provide practical value for LLM failure case analysis? In particular, we study scenarios that are difficult to resolve through behavioral analysis alone: (1) when a model produces an incorrect token, can interpretability reveal clues about the underlying decision process; (2) when a stronger model outperforms a weaker one, can interpretability expose evidence that the stronger model genuinely corrects the weaker model’s failure modes; and (3) across training checkpoints, can interpretability reflect the process by which models learn to make fewer mistakes. To explore these questions, we formulate failure analysis as a problem of explaining why a model prefers an incorrect output token over a more correct alternative. We cast this as Contrastive Attribution [ 66 ] , attributing the logit difference between an incorrect token and a correct candidate to input tokens and internal model states. We build on AttnLRP [ 1 ] , a state-of-the-art variant of the Layer-wise Relevance Propagation (LRP) framework [ 7 ] , and introduce an efficient extension that enables the construction of cross-layer hidden-state attribution graphs for long-context inputs. Using this framework, we conduct a systematic empirical study on hundreds of failure cases collected over multiple benchmarks, comparing attribution patterns across datasets, model sizes, and training checkpoints. Our results demonstrate that token-level contrastive attribution can yield informative signals for certain classes of failures, but is not universally effective, thereby clarifying both the promise and the limitations of interpretability methods for realistic LLM failure analysis. Our contributions are summarized as follows: • We formulate LLM failure analysis as a token-level contrastive attribution problem, enabling the application of model interpretability methods to analyze failures in a more realistic setting than prior work. • We introduce an efficient extension of LRP-based attribution that enables scalable construction of cross-layer hidden-state attribution graphs for long-context inputs. • We conduct a systematic interpretability-based failure attribution analysis across common benchmarks, characterizing attribution patterns across datasets, model sizes, and training checkpoints in realistic failure scenarios. 2 Related Work Analyzing LLM Failures. LLM failures are often encountered and studied after benchmark evaluation. Such analyses have revealed systematic weaknesses in areas including natural language inference, semantic and abstract reasoning, robustness, and bias [ 12 ] . However, these studies largely treat LLMs as black boxes, relying on output-level metrics without examining the internal mechanisms that lead to failures. Another prominent line of research studies LLM failures under the umbrella of hallucination . Empirical work in this area characterizes hallucination behaviors [ 44 ] as covered by recent surveys [ 31 , 30 , 70 ] . To facilitate controlled analysis, several hallucination-focused benchmarks have been proposed [ 36 , 34 , 9 ] . While these datasets are valuable for isolating hallucination phenomena, the resulting failures are tightly coupled to dataset design and differ from errors that arise organically in standard benchmark evaluations. Interpretability methods have also been applied to study hallucination-related behaviors, but predominantly in synthetic or simplified settings. Prior work [ 45 , 14 , 25 , 46 , 65 , 37 ] often relies on short prompts or highly structured formats, such as subject–relation–object templates [ 45 ] . These settings abstract away much of the complexity present in realistic, open-ended benchmark failures. Research on LLM reasoning failures has also gained renewed attention with the emergence of large reasoning models [ 35 ] . Recent studies categorize reasoning errors [ 56 ] , monitor reasoning processes [ 8 ] , steer or interpret reasoning via sparse features [ 24 ] , and debug reasoning failures [ 68 ] . However, most of these works focus primarily on reasoning tokens, rather than connecting failures to broader benchmark-defined errors. Lastly, a growing body of research work examines failures in LLM-powered agent systems, including failure taxonomies [ 11 ] , failure localization [ 69 ] , and agent debugging methods