An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems
Md Hasan Saju, Maher Muhtadi, Akramul Azim · 2026
Read An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems by Md Hasan Saju, Maher Muhtadi, Akramul Azim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00254.txt.
Introduction
Md Hasan Saju, Maher Muhtadi, and Akramul Azim compare three ways to aim language models at buggy, unsafe code. Finding software flaws early is vital for security. The team tests retrieval augmented generation first. It pulls facts from the web and a weakness database. Supervised fine tuning with small adapters comes second. A dual agent design comes third.
In that design, a second agent audits the first one. A plain baseline model sets the floor. Test data mix a public flaw set with real GitHub code. Five grave weakness classes anchor the labels. Retrieval wins on accuracy and F1 score overall. Fine tuning also performs strongly in the tests. The dual agent adds clearer reasoning and fewer errors. It does so with lower resource overhead. Domain knowledge, in some form, proves decisive.
Catalog record
| CREATOR | Md Hasan Saju, Maher Muhtadi, Akramul Azim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-00254
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems Md Hasan Saju, Maher Muhtadi, Akramul Azim Abstract The rapid advancement of Large Language Models (LLMs) presents new opportunities for automated software vulnerability detection, a crucial task in securing modern codebases. This paper presents a comparative study on the effectiveness of LLM-based techniques for detecting software vulnerabilities. The study evaluates three approaches, Retrieval-Augmented Generation (RAG), Supervised Fine-Tuning (SFT), and a Dual-Agent LLM framework, against a baseline LLM model. A curated dataset was compiled from Big-Vul [ 1 ] and real-world code repositories from GitHub, focusing on five critical Common Weakness Enumeration (CWE) categories: CWE-119, CWE-399, CWE-264, CWE-20, and CWE-200. Our RAG approach, which integrated external domain knowledge from the internet and the MITRE CWE database, achieved the highest overall accuracy (0.86) and F1 score (0.85), highlighting the value of contextual augmentation. Our SFT approach, implemented using parameter-efficient QLoRA adapters, also demonstrated strong performance. Our Dual-Agent system, an architecture in which a secondary agent audits and refines the output of the first, showed promise in improving reasoning transparency and error mitigation, with reduced resource overhead. These results emphasize that incorporating a domain expertise mechanism significantly strengthens the practical applicability of LLMs in real-world vulnerability detection tasks. I Introduction A software vulnerability is a flaw, caused by weaknesses such as buffer overflows, authentication errors, code injection, or design deficiencies, in the source code that can be exploited by hackers to breach security measures and gain unauthorized access to a system or network [ 2 ] . This can lead to severe consequences, including data theft, system manipulation, service disruption, and financial loss [ 3 ] . For example, according to the Cost of a Data Breach Report 2024 by IBM, the average cost of a data breach is USD 4.88 million which includes the costs of detecting and addressing the breach, disruption and losses, and the damage to the business reputation [ 3 ] . Vulnerabilities are especially significant in safety-critical systems, where the consequences of exploitation can be catastrophic. As highlighted in [ 4 ] , real-time systems like automotive control systems (e.g., anti-lock braking, cruise control) depend on both logical and temporal correctness for faultless operation. A breach in such systems can disrupt timing constraints, leading to missed deadlines and potentially life-threatening failures. Therefore, it is important to detect and mitigate vulnerabilities in a timely manner. Vulnerability detection involves identifying security weaknesses in software code that attackers could exploit. Conventional detection approaches like rule-based methods and signature-based techniques rely on predefined patterns to spot known vulnerabilities but often fail to detect new or sophisticated threats [ 5 ] . Recent advances in machine learning, especially deep learning, have transformed this field by enabling systems to automatically learn complex patterns from code [ 5 ] . Moreover, the rise of large language models (LLMs) has further enhanced detection capabilities, as these models can analyze code syntax and context to identify vulnerabilities more effectively. LLM models are further customized to improve vulnerability detection through Retrieval-Augmented Generation (RAG) and fine-tuning approaches. However, X. Du et al. , [ 6 ] and A. Z. H. Yang et al. [ 7 ] highlighted challenges such as false positives and computational costs persist, motivating the exploration of hybrid approaches like Dual-Agent systems. In this paper, we evaluate and compare the effectiveness of different LLM-based approaches for detecting source code vulnerabilities. The approaches investigated in this paper are RAG, Supervised Fine-Tuning (SFT), and a Dual-Agent system. These approaches are then compared to the performance of the base LLM model. The Dual-Agent system comprises a detector model for identifying vulnerabilities and a validation model for reviewing the first agent’s findings. There are some relevant motivations behind this study. Firstly, this study aims to provide a holistic comparison of the three different LLM techniques like RAG, fine-tuning, and Dual-Agent LLMs for vulnerability detection and help researchers or developers decide on the best approach for their needs. Secondly, this paper is the first study to implement and apply a Dual-Agent system in the domain of code vulnerability detection, to the best of our knowledge [ 8 ] . Finally, this paper addresses many problems and gaps seen in most vulnerability datasets such as class imbalance, vulnerability coverage limitations and generalizability gaps [ 5 ] , by curating data from multiple sources. The remainder of this paper is organized as follows: Section II reviews related work while identifying the gaps that our study addresses; Section III outlines the methodology used to set up the experiment and execute the study; Section IV discusses the results and limitations of the experiment; and Section V concludes the paper with a summary of the discussion. II Background Code vulnerability detection has evolved significantly from traditional rule-based methods to modern LLM-based approaches, particularly for safety-critical systems where vulnerabilities can have catastrophic consequences. This section provides context for our research by examining existing LLM-based solutions for vulnerability detection and highlighting how our work addresses dataset gaps in the field. II-A Retrieval-Augmented Generation Generally, when an LLM is prompted with a question, it generates a response and reasoning based on the data it was trained on. However, in a RAG [ 9 ] setup, the LLM’s reasoning process is enhanced by providing access to a more up-to-date knowledge base. This means the LLM’s output is informed not only by its inherent reasoning skills but also by the latest information available. When a user submits a query, the LLM first retrieves relevant knowledge from the repository containing the most current data, then refines and ranks that knowledge, synthesizes context, and finally produces the answer. This approach allows the LLM to generate more accurate and timely responses. For example, Vul-RAG [ 6 ] uses a knowledge-level framework to detect subtle vulnerabilities, while VulScribeR [ 10 ] employs RAG for dataset augmentation, improving detection performance. II-B Supervised Fine-Tuning: LLMs are trained on vast amounts of data, which enables them to excel at reasoning and performing general tasks. However, we often require the LLM to handle specialized work. As such, training the model on a specialized dataset can help improve its performance in a specific domain. Parameter Efficient Fine-Tuning: There are two ways to provide new knowledge to an LLM through training. One approach is to fully re-train the model, which requires substantial resources and is prohibitively expensive. It also runs the risk of catastrophic forgetting, where the LLM loses previously acquired knowledge and reasoning. A more efficient option is parameter-efficient fine-tuning [ 11 ] , where instead of altering the weights of the pretrained model, we introduce a small set of additional parameters and fine-tune them for our specific use case. This method greatly reduces computational overhead while preserving the original reasoning and knowledge of the model. QLoRA: LoRA (Low-Rank Adaptation) [ 12 ] addresses the challenge of fully training an LLM by introducing adapters. Instead of training all the model’s weights, it trains only the adapter weights to adapt the model to the target domain using a relevant dataset. This approach provides better reasoning and performance with limited resources. QLoRA is an extension of LoRA that further optimizes memory usage through quantization, storing model parameters