Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs
Maria Camporese, Fabio Massacci · 2025
Read Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs by Maria Camporese, Fabio Massacci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20977.txt.
Introduction
Maria Camporese and Fabio Massacci test in 2025 whether language models truly fix code flaws or just recall fixes they saw in training. High scores in past studies may hide gifts such as leaked answers or the exact flaw spot being handed to the model.
To probe this, the pair shift the stated flaw spot by set lines before asking for a fix. If recall drives the scores, a shift should not change the count of good fixes. A second model reviews each patch, and tests check that the flaw is gone and old skills stay. Hand checks on a sample gauge how often the pipeline errs.
Catalog record
| CREATOR | Maria Camporese, Fabio Massacci |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-20977
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs Maria Camporese, Fabio Massacci Abstract Automated Vulnerability Repair (AVR) is a rapidly emerging subfield of program repair. Large Language Models (LLMs) have recently shown promise in this area, delivering compelling results beyond traditional code generation and fault detection tasks. [ Hypothesis :] These results, however, may be influenced by “invisible hands”— hidden factors such as code leakage or perfect fault localization, which allow the LLM to reproduce fixes previously authored by humans for the same code fragments. [ Objective :] We aim to replicate prior AVR studies using LLMs for patch generation under controlled conditions, where we deliberately introduce errors into the vulnerability localization presented in the prompts. If LLMs are merely reproducing memorized fixes, both small and large localization errors should result in a statistically equivalent number of correct patches, as each type of error should steer the model away from the original fix. [ Method :] We introduce a pipeline for repairing vulnerabilities in the Vul4J and VJTrans benchmarks. The pipeline intentionally offsets the fault localization by n lines from the ground truth. An initial LLM generates a patch, which is then reviewed by a second LLM. The resulting patch is evaluated using regression testing and vulnerability proof tests. Finally, we conduct a manual audit of a sample of patches to assess correctness and compute the statistical error rate using the Agresti-Coull-Wilson method. Vrije Universiteit Amsterdam , NL [email protected] I Introduction Large language models (LLMs) are increasingly proposed for automated vulnerability repairs (AVR) [ 1 ] . At the time of submission, Google Scholar reports 483 works on the topic from 2024, excluding ArXiv papers, SoKs and systematic literature reviews. LLMs results (e.g., [ 2 ] ) seem more promising than traditional repair approaches based on testing (e.g., [ 3 ] ). APR4Vul tested traditional program repair methods on the Vul4J dataset [ 4 ] and obtained at best 5 correct patches. LLM methods obtained much better results: Codex was able to fix 10 vulnerabilities without fine-tuning [ 5 ] , later works provided 12 perfect fixes [ 6 ] , and up to 14 patches [ 7 ] . Yet, replicating LLMs’ success is challenging [ 1 ] , mirroring issues previously observed with deep learning models [ 8 ] . We argue that three “ invisible hands ” could be at play: code leakage, completion vs patching, and perfect localization . These factors could help LLMs achieve “exceptional” results that testing-based program repair methods did not achieve. The first “hand” involves code leakage: as a benchmark for vulnerability repair is eventually published, the new LLMs are likely to be trained on that very dataset, and the LLMs used for the original study might no longer be available. For instance, replicating the studies in [ 5 , 9 ] require access to Codex, which is now deprecated. Further, suppose we wanted to use PrimeVul [ 10 ] , a high-quality dataset with robust, albeit automatically verified, vulnerable and fixed functions. The corresponding paper was published at ICSE in May 2025, and therefore, GPT-4o-mini, deployed in May 2024, would seem to be a good candidate (it is used in 34 of the mentioned LLM papers). Unfortunately, the dataset was published on GitHub in March 2024 and includes commits pushed even earlier, so the possibility of data leakage cannot be ruled out. The second and third invisible hands stem from perfect fault localization and prompt design. Providing the exact location of the vulnerability is often the gold standard in program repair, e.g., in [ 3 ] (non-LLM) and [ 11 ] (LLM-based). By itself, this is a reasonable approach to distinguish success in vulnerability localization from success in patch generation. The key problem happens when it is combined with prompt designs that frame repairs as code completion: the fragment to be patched is replaced with a placeholder, and the LLM is asked to complete the code (e.g. [ 9 , 7 , 5 ] ). Given the likely leakage of code (see first invisible hand), LLMs may simply reproduce the correct fix via completion, a task where LLMs excel [ 12 ] . Overarching Hypothesis. A substantial portion of the LLM’s vulnerability repair success may be attributable to training data memorization, facilitated by precise fault localization. Our is a skeptical replication study , aimed at disentangling the potential influence of “invisible hands” on the success of LLM-based AVR. How can we remove the influence of these hands in order to test the hypothesis? We need to measure some events that are very unlikely to happen if the LLM is generalizing (“ideal” LLM) vs the LLM is just relying on memorization (“cheater” LLM). Bayes theorem would then allows us to distinguish the posterior probability. Some preliminary evidence in this respect has been already explored for normal bug fixing [ 13 ] . Our idea is to systematically introduce errors in the localization of vulnerabilities. If the LLM is ideal and genuinely generate repairs based on the localized vulnerability, then it is very unlikely that fixing a line that is completely off will generate a correct fix. And even less the fix of the developer… Conversely, if the LLM only memorizes fixes during training, then it is very likely that it will ignore the localization suggested by the prompt and propose the already known fix (if at all). Either way, we expect the LLM to reach the same results irrespective of the magnitude of the displacement offset (it’s cheating). To strengthen the result, we add a second step: we ask a second LLM to review the result of the first LLM before testing . This is considered a good solution to scale [ 14 ] albeit a recent theoretical analysis [ 15 ] has shown that MLs patch-reviewer model can only be useful if precision and recall are above a certain threshold. Worse, if our overarching hypothsis is true the second LL M is likely to suffer from the same memorization issues that the first LLM is being tested. If LLMs merely reproduce and recognize fixes they have already seen, we expect cheater LLMs as reviewers to recognize (memorized) plausible patches better than other plausible but wrong patches. As soon as we perturb the pipeline so that recognition is hampered, such difference will disappear. Localization rrors will not make a difference as they are ignored by cheaters. Operationally, we build on the pipelines from [ 5 ] and [ 7 ] : we assess the impact of different prompt designs and include in our input vulnerabilities obtained through code transformations so that they have no corresponding published fix. This contribute to isolate model generalization from memorization. We will validate generated patches using both regression tests and Proof of Vulnerability (PoV) tests 1 1 1 A proof of vulnerability in test-based vulnerability repair [ 16 , 3 , 17 ] is a test which flips between the vulnerable and non-vulnerable code fragments. II Terminology and background Automated Program Repair (APR) aims to support developers by automatically fixing bugs in software. End-to-end APR pipelines cover the full repair process: locating the faulty code (fault localization), generating a patch, and validating it to