SecureCodeRL: Security-Aware Reinforcement Learning for Code Generation with Partial-Credit Rewards

Suryansh Singh Sijwali, Suman Saha · 2026

Read SecureCodeRL: Security-Aware Reinforcement Learning for Code Generation with Partial-Credit Rewards by Suryansh Singh Sijwali, Suman Saha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01184.txt.

Introduction

Suryansh Singh Sijwali and Suman Saha train language models to write code that is both correct and safe. Models often produce code that runs but fails the required tests, or that slips in risky patterns. Their pipeline, SecureCodeRL, uses reinforcement learning with a reward that mixes function and security, and it gives partial credit along the way.

Partial credit rewards code for being valid, for running, and for printing output, even before it passes a full test. That steady signal keeps learning from stalling on hard tasks. In a small pilot study on held-out problems, the tuned model raised valid syntax from forty-five to sixty percent and was the only variant to pass any test. Its output also stayed clean under a standard security scanner. The authors stress the pilot scale, but the reward design is a useful template for safer code generation.

Catalog record

CREATORSuryansh Singh Sijwali, Suman Saha
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01184

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SecureCodeRL: Security-Aware Reinforcement Learning for Code Generation with Partial-Credit Rewards Suryansh Singh Sijwali, Suman Saha Abstract Large Language Models (LLMs) can generate plausible code, but in settings that require exact stdin/stdout behavior they frequently produce programs that compile yet fail tests, and in some cases they introduce security-sensitive patterns. This paper presents SecureCodeRL, a reinforcement learning (RL) pipeline for security-aware code generation that optimizes a combined reward R = α ​ R func + β ​ R sec R=\alpha R_{\text{func}}+\beta R_{\text{sec}} . The key idea is a partial-credit functional reward that assigns intermediate scores for syntactic validity, successful execution, and producing output, reducing reward sparsity that otherwise stalls learning on competitive-programming style tasks. I evaluate supervised fine-tuning (SFT) and PPO variants on a small held-out prompt set from APPS+ and observe that PPO with partial credit (using a continued-training variant) improves syntax validity from 45% (SFT) to 60% and achieves the only non-zero test success signal in this pilot evaluation (5% at-least-one-test-pass), while remaining 100% clean under Bandit static analysis. Although Bandit findings were absent in this small evaluation, the security term is integrated into training to discourage insecure shortcuts when they appear. I Introduction Code-capable LLMs are now widely used as programming assistants, but “looks correct” is not the same as is correct . In competitive-programming style problems, the program must parse input exactly, implement the right algorithm, and print output in the expected format. In practice, many generations fail due to wrong output, timeouts, crashes, or even producing no output at all. In parallel, there is a security angle: prior work has found that model suggestions can include insecure patterns and risky APIs [ 9 ] . This project grew out of a simple question I kept running into while benchmarking code models: If the reward is “all tests pass,” how does an RL system learn anything when almost nothing passes? In my APPS+ runs, the most common failures were not dramatic crashes; they were “nearly-right” programs that were off by formatting, missed a print statement, or computed a value but never emitted it. A binary pass/fail reward treats these failures the same as syntax errors, which makes reward sparse and learning brittle. SecureCodeRL is my attempt to make that learning signal less fragile. This approach optimizes two objectives simultaneously: • I optimize for functional correctness using test execution feedback. • I incorporate security awareness using static analysis (Bandit) as a penalty term. To bridge the gap between “fails everything” and “passes everything,” I introduce a partial-credit R_{\text{func}} that explicitly rewards intermediate milestones that matter in stdin/stdout judges. The full codebase for this project is publicly available. 1 1 1 Repository: https://github.com/SuryanshSS1011/basic-rl-feedback-workflow I-A Contributions I make the following contributions: • Evidence-driven motivation via benchmarking. I evaluate multiple open code models on APPS+ and quantify how often they produce syntactically valid code, pass tests, and remain security-clean. • A joint objective for correctness and security. I train with R=\alpha R_{\text{func}}+\beta R_{\text{sec}} , where R_{\text{sec}} is derived from Bandit findings. • Partial-credit reward shaping. I design an execution-stage reward that distinguishes syntax errors from runnable-but-silent programs (the “missing print” pattern) and from partial test matches. • Pilot RL results. On a small held-out prompt set, PPO with partial credit improves syntax validity and is the only evaluated variant to achieve non-zero test success while remaining Bandit-clean. II Background and Related Work II-A Code generation and execution feedback Recent code LLMs (e.g., DeepSeek-Coder, Code Llama, StarCoder2) are trained primarily with next-token prediction on large code corpora and can produce fluent solutions, but correctness under strict unit tests remains challenging [ 3 , 4 , 5 ] . A natural direction is to close the loop with compiler/test feedback. Prior work has explored reinforcement learning for code generation using execution signals, including CodeRL [ 6 ] and reinforcement learning from unit test feedback [ 7 ] . These ideas motivate SecureCodeRL: define reward using tools that directly reflect what a judge or developer cares about. II-B Security analysis as a training signal Static analyzers provide an operational way to discourage insecure patterns. In Python, Bandit flags a range of risky constructs (e.g., use of eval , unsafe subprocess usage, weak cryptography) [ 8 ] . In SecureCodeRL, I treat static-analysis findings as a penalty shaping term: code that passes tests but triggers serious findings should receive a lower reward. III Dataset, Benchmark, and Failure Modes III-A Dataset I use the APPS family of competitive programming problems [ 1 ] , and specifically the APPS+ distribution used in my implementation [ 2 ] (the APPS_Plus release). 2 2 2 Direct data file used in my pipeline: https://raw.githubusercontent.com/Ablustrund/APPS_Plus/refs/heads/main/data/v1/data.json . Each problem includes a natural language statement and multiple stdin/stdout test cases. This stdin-style format is exactly where small I/O mistakes (missing prints, extra labels, whitespace) can cause total failure. III-B Multi-model benchmark: the gap between “valid” and “correct” Before training SecureCodeRL, I benchmarked several models to quantify baseline difficulty. Table I reports the summary table I used to motivate this project: even when models generate syntactically valid code, full test success remains low. This gap between parses and passes is the core reason I wanted an RL loop driven by execution. (Later, in my pilot RL evaluation, I also report an “at least one test passes” indicator to detect the first signs of functional learning at small N .) TABLE I : Benchmark summary on stdin-style APPS+ subset (my runs). Test Pass denotes the percentage of prompts where the generated program passes all provided tests. Security Clean is computed using Bandit findings in my pipeline (with an additional lightweight regex heuristic used during benchmarking for debugging); Bandit is the primary metric reported. Model Syntax Valid Test Pass Security Clean DeepSeek-6.7B 83.4% 14.3% 96.3% CodeLlama-7B 48.9% 7.6% 99.5% StarCoder2-7B 20.2% 1.3% 99.7% The headline is not that “models are bad”—it is that the learning signal is sparse . If only a small fraction of generations ever fully pass, then binary reward ( 1 if all tests pass, else 0 ) provides almost no gradient. That is exactly the regime where reward shaping matters. III-C Failure taxonomy: what actually goes wrong To design a reward that helps PPO learn in this regime, I looked at how generations fail under the test harness. The dominant outcomes fall into a small set of categories shown in Table II . The key point is that many failures are “near misses” (especially wrong output and no output), which a binary reward collapses into the same zero as a syntax error. These proportions come from my benchmark logs over the stdin-style APPS+ subset (Section III-B ). TABLE II : Compact failure taxonomy observed during my benchmark analysis (approximate shares). Failure type Share of failures Wrong output \sim 60% No output \sim 25% Timeout/hang \sim 8% Crash \sim 5% Memory error \sim 2% This taxonomy is the bridge from benchmark to reward design: partial credit is explicitly shaped to separate syntax , runtime stability , and I/O behavior (the no-output bucket) before demanding full correctness. III-D Qualitative failure modes (representative examples) Numbers are helpful, but for code generation the patterns are often clearer in short snippets. III-D 1 Failure mode A: the

Read the complete text