Where’s the Bug? Attention Probing for Scalable Fault Localization

Adam Stein, Arthur Wayne, Aaditya Naik, Mayur Naik, Eric Wong · 2025

Read Where’s the Bug? Attention Probing for Scalable Fault Localization by Adam Stein, Arthur Wayne, Aaditya Naik, Mayur Naik, Eric Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13966.txt.

Introduction

Adam Stein and co-authors find bugs by reading a model's attention in a 2025 paper. Repair tools can propose fixes, but only if they first locate the fault. Existing locators need runnable tests, costly line labels, or giant models.

Their probe, BAP, learns localization with no direct labels at all. Across eight data sets spanning languages and bug types, it beat the strongest baseline by about thirty-five percent on top-one accuracy. Against zero-shot prompting of GPT-4o, the gain was over ninety percent. The probe also runs at a small fraction of the prompting cost. Attention patterns, it turns out, already whisper where the bug lives; the probe just learns to listen.

Catalog record

CREATORAdam Stein, Arthur Wayne, Aaditya Naik, Mayur Naik, Eric Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13966

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Where’s the Bug? Attention Probing for Scalable Fault Localization Adam Stein, Arthur Wayne, Aaditya Naik, Mayur Naik, Eric Wong Abstract Ensuring code correctness remains a challenging problem even as large language models (LLMs) become increasingly capable at code-related tasks. While LLM-based program repair systems can propose bug fixes using only a user’s bug report, their effectiveness is fundamentally limited by their ability to perform fault localization (FL), a challenging problem for both humans and LLMs. Existing FL approaches rely on executable test cases, require training on costly and often noisy line-level annotations, or demand resource-intensive LLMs. In this paper, we present Bug Attention Probe ( BAP ), a method which learns state-of-the-art fault localization without any direct localization labels, outperforming traditional FL baselines and prompting of large-scale LLMs. We evaluate our approach across a variety of code settings, including real-world Java bugs from the standard Defects4J dataset as well as seven other datasets which span a diverse set of bug types and languages. Averaged across all eight datasets, BAP improves by 34.6% top-1 accuracy compared to the strongest baseline and 93.4% over zero-shot prompting GPT-4o. BAP is also significantly more efficient than prompting, outperforming large open-weight models at a small fraction of the computational cost. 1 1 1 BAP is open-sourced here: https://github.com/adaminsky/BAP † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Comparison of our approach Bug Attention Probe ( BAP ) with baselines DeepFL and LLM prompting on a Java program snippet. The program has two bugs: the age condition on line 3 is reversed and line 6 throws a null pointer exception. BAP correctly localizes both bugs. Here, our method is trained on Llama-3.2-1B, a “small” language model (SLM), with only weak supervision i.e. binary bug presence labels. Obtaining comparable accuracy via prompting demands a significantly more resource-intensive LLM, such as Llama-3.2-90B, or even larger. Previous approaches to fault localization like DeepFL require executable test cases before they can attempt to provide useful information. Correctness is a fundamental desirable property of code. Both human-written and LLM-generated code are prone to bugs ( Karampatsis & Sutton, 2020 ; Jesse et al., 2023 ) including syntax errors that prevent the execution of code, semantic mistakes that cause incorrect or unintended behaviors, and vulnerabilities that compromise security in otherwise correct code. While there are various methods for detecting bugs (e.g. failed tests, via human bug report, program crash, etc.), identifying its root cause, or localizing the bug, is still costly ( Vessey, 1985 ; Wong et al., 2016a ) . Automated software fault localization (FL) aims to help a programmer answer the question, “Where’s the bug?”, ideally pointing to specific lines of buggy code. The traditional FL approaches rely on executable tests to determine the buggy lines ( Johnson et al., 2013 ) . Without relying on tests, FL is even more challenging since such a system must reason about what is buggy without external feedback. But recently, supervised training of models on large, labeled datasets ( Yang et al., 2024a ; Meng et al., 2022 ) , and prompting of the largest LLMs such as GPT-4o has shown promise at FL without tests ( Wu et al., 2023 ) . On a single method context, large-scale supervised training and prompting approaches can significantly surpass the traditional techniques which need tests ( Yang et al., 2024a ; Wu et al., 2023 ) . State-of-the-art FL methods, however, are still limited in scalability , or the ability to leverage cheaply available supervision to reach strong performance, even with small models. This lack of scalability leads to the impracticality of many FL techniques. For example, we show a simple buggy Java code snippet in Figure 1 which we want to run FL on, but we encounter several issues: traditional methods require executable tests and LLM prompting methods are only effective for the largest of models. On large codebases, LLMs must be called at least once per function, which quickly becomes expensive. Further, training-based approaches assume extensive amounts of strong supervision for FL, which is rarely available in practice ( Zhang et al., 2024 ) . This leads to our central question: How do we achieve strong fault localization performance without relying on executable tests, costly large-scale LLMs, or strong supervision? We answer this question by proposing the Bug Attention Probe ( BAP ), a scalable LLM probing technique for FL, scaling to use available bug related data without strong FL supervision and scaling with base model size while still achieving strong performance with small models. BAP exhibits three desirable properties: (1) lightweight , (2) code-level localization , and (3) localization of multi-line bugs . First, lightweight refers to the limited requirement for supervision (we use an attention mechanism to learn from binary bug presence rather than fine-grained location supervision), the test-free nature, and the model size ( BAP can probe small language models (SLMs) to elicit performance significantly stronger than the underlying SLM). Second, BAP localizes bugs in a human-interpretable manner to expressions, statements, or lines in code, even though LLMs operate on the token level. Third, BAP localizes multi-line bugs, or multiple bugs in one method, better than existing approaches. Multi-line bugs are practically relevant since the majority of real bugs are multi-line ( Pearson et al., 2016 ) . To evaluate BAP , we use eight diverse and widely-used fault localization benchmarks, including syntax errors, semantic mistakes, and weaknesses. Our evaluation suite covers over 50,000 code samples across three languages: Python, Java, and C. This notably includes Defects4J ( Just et al., 2014 ) , the most commonly used FL dataset. We evaluate BAP on top of the Llama-3 family of models and compare it to state-of-the-art FL methods, including traditional test-based FL and prompting of various proprietary and open-weights LLMs. Averaged across eight datasets, BAP improves by 23.4% over the strongest baselines for top-1 FL accuracy which includes a 24.2% improvement for Defects4J, and a 50.5% improvement on DeepFix ( Gupta et al., 2017 ) . In addition, BAP achieves these performance improvements at over ten times greater efficiency in terms of model size and FLOPs for inference. BAP also localizes multi-line bugs better than existing methods and continues to have stronger performance than prompting for longer code sequences. While BAP significantly outperforms competitive baselines, it is able to achieve 35% top-1 FL accuracy on average over our datasets, and our evaluation highlights avenues for further advances in scalable FL. In summary, our work makes the following contributions: • We propose Bug Attention Probe ( BAP ) as a general method for scalable fault localization, requiring only coarse-grained detection supervision and eschewing the need for localization labels (Section 3 ). • BAP significantly improves over state-of-the-art fault localization methods by 34.6% top-1 accuracy on average over eight fault localization benchmarks while utilizing ten times less memory resources and FLOPs. • BAP exhibits better length generalization and can predict multi-line bugs more effectively than prompting. We also identify areas for further advances. Figure 2 : Illustration of BAP as a method to elicit line-level fault localization from a frozen LLM through weak supervision. In step one, the probe is trained as a binary classifier to distinguish buggy from non-buggy code.

Read the complete text