Training Language Models to Generate Quality Code with Program Analysis Feedback
Feng Yao Zilong Wang Liyuan Liu Junxia Cui Li Zhong Xiaohan Fu, Haohui Mai Vish Krishnan Jianfeng Gao Jingbo Shang · 2025
Read Training Language Models to Generate Quality Code with Program Analysis Feedback by Feng Yao Zilong Wang Liyuan Liu Junxia Cui Li Zhong Xiaohan Fu, Haohui Mai Vish Krishnan Jianfeng Gao Jingbo Shang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22704.txt.
Introduction
Feng Yao and co-authors train language models to write code that is not just working but clean and safe. Their 2025 paper names the gap in today's code generation: models happily produce code with injection flaws or missing type notes, and hand-labeling quality does not scale. Their framework, REAL, replaces human judgment with automated program analysis as the reward signal.
A checker flags security and upkeep defects in each draft, while unit tests confirm the code still runs. Reinforcement learning then pays the model for drafts that pass both gates. The loop aims at production-grade habits, security and maintainability, learned from tools rather than from costly annotation.
Catalog record
| CREATOR | Feng Yao Zilong Wang Liyuan Liu Junxia Cui Li Zhong Xiaohan Fu, Haohui Mai Vish Krishnan Jianfeng Gao Jingbo Shang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-22704
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training Language Models to Generate Quality Code with Program Analysis Feedback Feng Yao Zilong Wang Liyuan Liu Junxia Cui Li Zhong Xiaohan Fu, Haohui Mai Vish Krishnan Jianfeng Gao Jingbo Shang Abstract Code generation with large language models (LLMs), often termed vibe coding , is increasingly adopted in production but fails to ensure code quality , particularly in security (e.g., SQL injection vulnerabilities) and maintainability (e.g., missing type annotations). Existing methods, such as supervised fine-tuning and rule-based post-processing, rely on labor-intensive annotations or brittle heuristics, limiting their scalability and effectiveness. We propose ReaL , a reinforcement learning framework that incentivizes LLMs to generate production-quality code using program analysis-guided feedback. Specifically, ReaL integrates two automated signals: (1) program analysis detecting security or maintainability defects and (2) unit tests ensuring functional correctness. Unlike prior work, our framework is prompt-agnostic and reference-free, enabling scalable supervision without manual intervention. Experiments across multiple datasets and model scales demonstrate that ReaL outperforms state-of-the-art methods in simultaneous assessments of functionality and code quality. Our work bridges the gap between rapid prototyping and production-ready code, enabling LLMs to deliver both speed and quality. 1 Introduction Large language models (LLMs) have revolutionized code generation, enabling rapid workflows colloquially termed vibe coding . Coding assistants like Copilot ( GitHub, 2021 ) , Cursor, and Windsurf ( Xu et al., 2022 ) ) exemplify this shift, with developers increasingly relying on LLMs to automate tasks from prototyping to production, highlighting the critical need to ensure code quality . In production settings, code quality extends beyond functional correctness to encompass: security (e.g., resistance to injection attacks or misuse of unsafe functions) and maintainability (e.g., proper type annotations, consistent style, and modular structure). These properties are crucial for long-term reliability and team collaboration. However, LLMs are known to generate code that is syntactically plausible but flawed in subtle or dangerous ways ( Yang et al., 2024 ; Wan et al., 2024 ) . Existing methods improve code quality either through supervised fine-tuning on large corpora of manually curated, vulnerability-free code ( He et al., 2024 ) or by applying rule-based post-processing at inference to enforce security constraints ( Fu et al., 2024 ; Nazzal et al., 2024 ) . The former incurs high annotation costs, while the latter depends on hand-crafted constraints specific to each coding task. Both strategies exhibit limited scalability and effectiveness in real-world production scenarios. Figure 1: Overview of the ReaL framework. Given a coding task, the LLM policy generates a candidate program, which is then evaluated along two automated axes: (1) Vulnerability Detector applies program analysis to flag security and maintainability defects, (2) Functionality Verifier runs unit tests to assess correctness. The two reward signals are averaged and fed into a policy-gradient update, steering the LLM toward high-quality, functionally correct code with minimal human effort. We introduce ReaL (Reinforcement rEwards from Automated program anaLysis), a reinforcement learning framework that trains LLMs to generate quality code through program analysis-guided feedback. Unlike prior methods that either teach LLMs to mimic human-verified code examples or correct their outputs post hoc with brittle heuristics, ReaL employs verifiable and reference-free reward signals to incentivize quality code generation with minimal human efforts. As illustrated in Figure 1 , ReaL ’s compound reward combines: (1) program analysis–based detection of vulnerabilities in security or maintainability, and (2) unit-test–based verification of functional correctness. To demonstrate ReaL ’s efficacy, we evaluate it across multiple benchmarks spanning diverse production scenarios, assessing code quality along two key dimensions. (1) For security evaluation, we augment SecCodePLT dataset ( Yang et al., 2024 ) with a program analysis-based detector built by us that effectively identifies 17 Common Weakness Enumerations (CWEs) § A.1 , resulting in an enhanced benchmark we term SecCodePLT+. To enable fine-grained evaluation of high-impact vulnerabilities, we additionally introduce SafeSQL, a targeted dataset featuring realistic database query tasks susceptible to SQL injection attacks. (2) For maintainability assessment, we augment APPS dataset ( Hendrycks et al., 2021 ) to APPS+ with comprehensive static analysis, including type checking, unreachable code detection, and function signature verification for Python code. In addition to code quality evaluation, we assess functional correctness using unit tests. To provide a holistic evaluation, we introduce a composite metric that jointly measures both functionality and quality. This resolves a critical gap: structurally sound code that fails functionally should not be prioritized. By integrating both objectives, our evaluation reflects the real-world needs, where code quality and functionality are inseparable. Extensive experiments across diverse benchmarks and model sizes demonstrate that ReaL consistently outperforms state-of-the-art baselines, confirming its scalability and effectiveness in delivering reliable and production-quality code generation. To summarize, our contributions 1 1 1 Our code and datasets are released at https://github.com/yaof20/ReaL.git are three-fold: • We propose ReaL , a novel reinforcement learning framework that integrates program analysis as automated feedback, enabling LLMs to generate quality code with minimal manual intervention. • We contribute three datasets for quality code generation: (1) SecCodePLT+, enhancing ( Yang et al., 2024 ) with detectors for 17 CWEs, (2) APPS+, augmenting ( Hendrycks et al., 2021 ) with static analysis for maintainability, and (3) SafeSQL, a targeted dataset for SQL injection vulnerabilities. • We design a holistic evaluation protocol that jointly prioritizes functionality and code quality, resolving the oversight of prior work that treats these objectives independently. 2 Problem Formulation In this section, we first formalize the concept of quality code we investigate in this paper ( § 2.1 ), then outline the existing paradigms and their limitations that motivate our work ( § 2.2 ), and finally introduce our task formulation and holistic evaluation protocol for quality code generation ( § 2.3 ). 2.1 Quality Code: Beyond Functional Correctness In real-world production, quality code should be both functionally correct and vulnerability resistant. In this work, we concentrate on two classes of vulnerabilities that critically impact production code: • Security Vulnerabilities . These include exploits such as SQL injection and Cross-Site Request Forgery (CSRF), corresponding to entries in the Common Weakness Enumeration (CWE) ( MITRE, 2024 ) . These vulnerabilities pose significant risks and critical threats in production environments. • Maintainability Vulnerabilities . In dynamically typed languages (e.g., Python), the absence of explicit type information, unreachable code paths, or inconsistent function signatures often leads to latent bugs, runtime errors, and degrade long-term code reliability and maintainability. 2.2 Limitations of Existing Approaches Existing methods for quality code generation have primarily targeted reducing security vulnerabilities, with limited attention to maintainability aspects. We further identify some other critical limitations in both their evaluation protocols and methodological frameworks as follows. First, prior work suffers from incomplete evaluation protocols, manifesting in three key shortcomings: • Quality-Functionality Isolation . Code quality and functionality are evaluated separately using disjoint datasets ( He et al., 2024 ) or omitting functionality entirely ( Bhatt et al., 2023 ) , failing to capture real-world production requirements of satisfying both criteria at the same time. • Single-Vulnerability Assumption . Evaluations by default assume each coding problem contains only one vulnerability type and rely on this false premise for methodology design ( Fu et al., 2024 ) , ignoring production scenarios where the code can involve multiple defects simultaneously. • Limited Detection Paradigms . Existing evaluations rely on two