Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback
Vidyut Sriram, Sawan Pandita, Achintya Lakshmanan, Aneesh Shamraj, Suman Saha · 2026
Read Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback by Vidyut Sriram, Sawan Pandita, Achintya Lakshmanan, Aneesh Shamraj, Suman Saha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00509.txt.
Introduction
Code written by AI often has security holes, logic slips, and lines that will not run. Vidyut Sriram and four co-authors tackle that flaw in a 2026 paper. Their workflow lets one code-writing model fix its own work in rounds. Each round feeds it notes from three tools. One is the compiler, which flags lines that break. One is a security scanner, and one tests code paths with symbols, not fixed inputs.
The workflow also looks up past fixes that worked on like flaws. The team tested the setup on more than three thousand programs. The programs came from two open code models, one small and one larger. For the small model, security flaws fell by about ninety-six in each hundred. For the larger model, the share of code with grave flaws fell a lot too. Tool feedback, the study shows, can make AI code much safer.
Catalog record
| CREATOR | Vidyut Sriram, Sawan Pandita, Achintya Lakshmanan, Aneesh Shamraj, Suman Saha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-00509
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback Vidyut Sriram, Sawan Pandita, Achintya Lakshmanan, Aneesh Shamraj, Suman Saha Abstract Large Language Models (LLMs) can generate code but often introduce security vulnerabilities, logical inconsistencies, and compilation errors. Prior work demonstrates that LLMs benefit substantially from structured feedback, static analysis, retrieval augmentation, and execution-based refinement. We propose a retrieval-augmented, multi-tool repair workflow in which a single code-generating LLM iteratively refines its outputs using compiler diagnostics, CodeQL security scanning, and KLEE symbolic execution. A lightweight embedding model is used for semantic retrieval of previously successful repairs, providing security-focused examples that guide generation. Evaluated on a combined dataset of 3,242 programs generated by DeepSeek-Coder-1.3B and CodeLlama-7B , the system demonstrates significant improvements in robustness. For DeepSeek, security vulnerabilities were reduced by 96%. For the larger CodeLlama model, the critical security defect rate was reduced from 58.55% to 22.19%, highlighting the efficacy of tool-assisted self-repair even on “stubborn” models. 1 Introduction Large Language Models (LLMs) have rapidly advanced the state of automated code generation, yet their outputs remain highly unreliable without structured external guidance. Numerous studies show that LLM-generated programs frequently suffer from compilation failures, logical inconsistencies, and severe security vulnerabilities. For example, 1 demonstrate that GitHub Copilot produces insecure code in over 40% of tested cases, often introducing buffer overflows, unsafe input handling, or cryptographic misuse. These findings indicate that LLMs tend to replicate insecure patterns from training data unless augmented with explicit safety mechanisms. A broader perspective is offered by 4 , who provide the most comprehensive taxonomy of LLM-based Automated Program Repair (APR) to date. Their survey of sixty-three APR systems concludes that single-pass LLM output is rarely sufficient for correctness or robustness. Instead, the most successful systems incorporate external tools—such as static analyzers, execution engines, and retrieval modules—into procedural repair workflows. This observation motivates our approach: rather than treating the LLM as a standalone generator, we embed it within a multi-step verification and self-repair loop. Execution-based refinement has also proven effective in improving code reliability. 2 introduce Self-Debug, a system where compiler and runtime error messages are fed back into the model to iteratively repair incorrect programs. Their results reveal dramatic improvements in correctness through structured feedback, supporting the hypothesis that LLMs can self-correct when provided actionable diagnostic signals. Complementary work on iterative reasoning, such as STaR by 3 , further demonstrates that multi-step refinement can substantially enhance model performance across domains. Motivated by these findings, we design a retrieval-augmented, tool-guided self-repair workflow for secure code generation across two contrasting LLMs: DeepSeek-Coder-1.3B and CodeLlama-7B. Our approach is distinguished by the tight integration of three complementary mechanisms: retrieval-augmented context, multi-tool verification, and iterative self-repair within a single inference-time loop. Retrieval provides prior examples of successful secure repairs, reducing the tendency to repeat unsafe idioms; tool-based analysis supplies concrete diagnostics that expose syntactic, semantic, and security flaws; and iterative self-repair enables the model to adapt its output in response to these signals without requiring fine-tuning or additional generative agents. By combining these signals, the workflow addresses distinct failure modes that are not adequately handled by any component in isolation, offering a practical alternative to agent-heavy or training-intensive approaches. 2 Literature Review The literature consistently emphasizes two key failures of large language models in code generation: (1) the tendency to generate syntactically incorrect or uncompilable code, and (2) the frequent introduction of subtle or severe security vulnerabilities. 1 conducted one of the first systematic evaluations of LLM security, showing that Copilot-generated code commonly contains memory-safety violations, unchecked input flows, integer overflows, and other critical defects. Their vulnerability taxonomy motivated our integration of CodeQL, which detects precisely these classes of issues. Beyond security defects, several works show that LLMs struggle with deeper program semantics. 2 demonstrated that symbolic execution and compiler feedback dramatically improve correctness when used iteratively. Their “Self-Debug” findings guided our integration of KLEE, which exposes defects unreachable by simple test-case evaluation. 3 introduced STaR, a method showing that LLMs substantially improve when placed in iterative refinement loops rather than one-shot inference. This supported our design philosophy that secure code generation requires multi-step reasoning rather than single-pass outputs. Finally, the comprehensive taxonomy provided by 4 categorizes 63 APR systems and concludes that the most successful ones are tool-augmented rather than purely prompt-driven. They identify retrieval-augmented repair and static-analysis-guided workflows as particularly promising directions. Their findings directly support our integration of RAG with CodeQL and KLEE, highlighting that reliable code generation requires multi-tool orchestration rather than pure LLM sampling. 3 Methodology Our system integrates compiler diagnostics, symbolic execution, static analysis, and retrieval-based guidance into a unified self-repair framework centered around a single code-generating LLM. A lightweight sentence embedding model ( all-MiniLM-L6-v2 ) performs semantic retrieval over a database of previously successful repairs, returning security-focused examples that are injected into the prompt. The retrieval model does not generate or modify code; it serves solely to select relevant contextual examples. The active LLM (DeepSeek-Coder-1.3B or CodeLlama-7B, depending on the experiment) generates candidate code conditioned on the task description and retrieved examples. The generated program is analyzed by compiler diagnostics (GCC), symbolic execution (KLEE), and static analysis (CodeQL). If any errors are detected—including compilation failures, failing KLEE assertions, or CodeQL security alerts—the diagnostic messages are appended to the prompt and fed back to the same LLM in an iterative self-repair loop. This process repeats for up to three attempts per prompt. When a program passes all checks, the final code and its associated context are embedded and stored back into the RAG database. During future generations, semantically similar successful repairs are retrieved and used to guide the LLM toward more secure and robust patterns from the outset. Figure 1 summarizes this workflow, showing how retrieval, compilation, symbolic execution, and security analysis interact in a closed feedback loop around a single LLM. Figure 1: Block diagram of the proposed multi-tool feedback system. A lightweight embedding model retrieves security-focused examples from a RAG memory to augment the prompt. A single code-generating LLM produces candidate code, which is analyzed by compiler diagnostics (GCC), symbolic execution (KLEE), and static analysis (CodeQL). Diagnostic feedback is iteratively fed back to the same LLM, and successful repairs are stored back into the RAG database. 4 Error Taxonomy and Dataset We evaluated the workflow on a total of 3,242 generated programs (1,522 from DeepSeek and 1,720 from CodeLlama). Each sample is labeled along three error dimensions: • Compilation Errors: The program fails to compile due to syntax or type issues. • Security Errors: CodeQL reports at least one critical security issue (such as cpp/unbounded-write , unchecked input flows, or integer overflows). • Semantic Errors: The program compiles but fails KLEE-based semantic checks in the provided test harness. 5 Results 5.1 Performance on DeepSeek-Coder-1.3B With the full workflow enabled for the DeepSeek model (1,522 programs), we observed dramatic improvements. The compilation success rate improved by 19.4 percentage points. Most notably, the system achieved a 98.6% security-clean rate (up from a baseline of roughly 64%), effectively eliminating almost all vulnerabilities. Metric Baseline Repair Loop Improvement Compilation 39.79% 20.43% 19.36% Red. Security 36.35% 1.45% 34.90% Red. Semantic 60.09% 5.72% 54.37% Red. Table 1: DeepSeek-1.3B performance comparison. Compilation Security Semantic 0 20 40 60 Error Rate (%)