Execution-Grounded Credit Assignment for GRPO in Code Generation
Abhijit Kumar, Natalya Kumar, Shikhar Gupta · 2026
Read Execution-Grounded Credit Assignment for GRPO in Code Generation by Abhijit Kumar, Natalya Kumar, Shikhar Gupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16158.txt.
Introduction
Kumar, Kumar, and Gupta pin GRPO credit to the exact tokens where code goes wrong. One pass-fail signal smears blame across whole programs in GRPO. Their EGCA method runs candidate and reference code side by side. Matching traces reveal the earliest semantic split point. Advantage then lands only on the matching token span. Later tokens stay masked out of that update. No critic, extra loss, or learned verifier joins the loop. HumanEval pass-at-1 hits 82.1 percent, up 3.1 on GRPO. MBPP gains 1.5 points at 18 percent more time.
Catalog record
| CREATOR | Abhijit Kumar, Natalya Kumar, Shikhar Gupta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27930 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16158
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Execution-Grounded Credit Assignment for GRPO in Code Generation Thanks: Accepted to the ICLR 2026 Workshop on Scaling Post-Training for LLMs (SPOT). Abhijit Kumar, Natalya Kumar, Shikhar Gupta Abstract Critic-free reinforcement learning with verifiable rewards (RLVR) improves code generation by optimizing unit-test pass rates, but GRPO-style updates suffer from coarse credit assignment: a single outcome signal is spread uniformly across long programs even when failure stems from a localized semantic error. We propose Execution-Grounded Credit Assignment (EGCA) , which localizes GRPO updates using execution traces. For programs that satisfy algorithmic constraints but fail tests, EGCA executes the candidate and a canonical reference solution (curated once offline; used for analysis, not supervision) under identical instrumentation, identifies the earliest semantic divergence, and assigns advantage only to the corresponding token span while masking downstream tokens. EGCA is a drop-in modification requiring no critic, auxiliary loss, or learned verifier, yielding 82.1% pass@1 on HumanEval (+3.1 over GRPO) and 68.9% on MBPP (+1.5) with 18% wall-clock overhead. 1 Introduction Reinforcement learning with verifiable rewards (RLVR), where generated programs are evaluated by unit tests, has become a standard post-training approach for improving code generation models. Critic-free objectives such as GRPO ( 14 ) are appealing: they avoid a value function and directly optimize functional correctness. As base models improve, however, the nature of failures shifts. Modern models increasingly produce code that is syntactically valid, structurally plausible, and fully executable, yet still fails unit tests due to subtle semantic mistakes—an incorrect condition, a misplaced update, or a misinterpreted invariant. Unit tests provide a reliable correctness signal, but it is temporally coarse: it applies to the entire program rather than to the specific decisions that caused failure. Group-based policy gradients distribute this signal uniformly, so near-correct solutions receive gradients too diffuse to correct localized reasoning errors. This paper addresses the problem of semantic credit assignment in critic-free RLVR for code generation and targets the near-correct regime where further gains depend on precise attribution rather than coarse feedback. The following are our key contributions: 1. We show that credit assignment—not reward sparsity—is the main bottleneck in critic-free RL for code generation once models already produce syntactically valid and structurally reasonable programs. 2. We introduce EGCA, which routes each sample through deterministic failure-mode gates (syntax/constraint/logic) and, for near-correct candidates, localizes the earliest execution divergence against a reference trace to concentrate the GRPO advantage on the causal token span. 3. We show the approach is agnostic to the debugger’s code-generation ability: the trained student surpasses a 1.5B-parameter debugger by +8.2 points, ruling out knowledge distillation as the mechanism. Figure 1: Motivation: credit smear vs. localized updates. Sequence-level RLVR objectives apply unit-test outcomes uniformly across long programs, penalizing large spans of correct code for a localized semantic bug. EGCA concentrates gradient mass on the earliest semantically divergent span (identified via execution) while masking downstream tokens, improving credit assignment in the near-correct regime. 2 Related Work We situate EGCA against five lines of work that densify credit in RLVR for code. The shared limitation is that none reliably localizes failure to semantically causal regions of fully executing programs. Richer outcome signals. Methods such as RLTF ( 13 ) enrich outcome feedback from execution beyond binary pass/fail, but the signal remains outcome-anchored and does not identify where within a program failure originated. Execution-aware masking. StepCoder ( 3 ) masks unexecuted tokens during updates, reducing spurious blame. However, when programs execute to completion, all tokens are executed, and masking provides no disambiguation among them. Group-structure credit. Prefix-tree methods such as TEMPO/P2T ( 16 ) derive token-level updates from textual branching points within sample groups. This provides cleaner credit than sequence-level baselines, but textual divergence does not necessarily coincide with the causal location of semantic failure in code. Learned evaluators. Process reward models ( 11 ; 12 ) train step-level scorers for RL shaping. These meaningfully densify supervision but inherit challenges around label noise and distribution shift from the learned evaluator. Execution semantics and actor–critic. CodeRL+ ( 9 ) adds auxiliary execution-alignment objectives, and actor–critic methods ( 10 ; 15 ) provide dense shaping via learned value functions. Both depart from the critic-free regime. Concurrent directions. Concurrent work explores complementary directions: RLEF ( 5 ) grounds code LLMs in execution feedback via RL; multi-turn rewards ( 8 ) study single-step reward shaping across turns; and MURPHY ( 4 ) applies multi-turn GRPO for self-correction. EGCA differs in localizing credit to the earliest divergence within a single generation. Summary and gap. Existing methods densify feedback or reduce noise, but do not reliably attribute failure to semantically causal regions of otherwise well-formed, constraint-following programs. EGCA targets this gap: for near-miss programs that execute but fail due to a localized reasoning error, it identifies the causal span via reference-trace comparison and concentrates the GRPO advantage there. 3 Method Figure 2: EGCA pipeline. We extract constraints from a canonical reference, sample and execute a group of programs, route each into syntax / constraint / logic / correct via deterministic gates, and apply token-level GRPO by localizing advantage (compiler span for syntax , earliest reference-trace divergence for logic ) while masking downstream tokens. 3.1 Problem Setting Let x denote a programming problem and let \pi_{\theta} be a code generation policy that samples programs y\sim\pi_{\theta}(\cdot\mid x) . Programs are evaluated using unit tests, producing a base verifiable score \hat{R}(y)\in[0,1] (fraction of tests passed). We incorporate algorithmic constraints by defining the reward used for optimization as R(y)=\hat{R}(y)\,\mathbb{I}_{\mathcal{C}}(y), (1) so that a program receives full reward only when it both passes all tests and satisfies the extracted constraints. We adopt Group Relative Policy Optimization (GRPO) ( 14 ) . For each problem x , we sample a group of G programs \{y_{i}\}_{i=1}^{G} and compute group-relative advantages A_{i}=R(y_{i})-\frac{1}{G}\sum_{j=1}^{G}R(y_{j}). (2) 3.2 Canonical Solutions as a Structural Pivot For each problem x , we assume access to a canonical reference solution y^{\mathrm{ref}} , curated once offline. The reference is not used as a target for imitation. Instead, it serves as a non-parametric pivot for (i) extracting algorithmic constraints, (ii) defining a reference execution behavior, and (iii) anchoring semantic comparisons. This requirement limits applicability to settings where at least one correct solution exists (e.g., competitive programming and function synthesis with tests); extending EGCA to open-ended generation without references remains future work. 3.3 Constraint-Guided Sampling Constraint extraction. A debugging-oriented teacher model extracts from (x,y^{\mathrm{ref}}) a set of algorithmic constraints \mathcal{C}=\{c_{1},\dots,c_{M}\}, (3) where each constraint specifies a permitted or forbidden structural property (e.g., control-flow form, permitted data structures, or complexity targets). Constraints are non-executable, non-token-level, and solution-agnostic. Sampling with constraints. Constraints are injected as soft guidance via a prompt suffix: y_{i}\sim\pi_{\theta}(\cdot\mid x\;\|\;\mathcal{C}). (4) This biases sampling toward programs that are structurally comparable to y^{\mathrm{ref}} , increasing the density of samples for which semantic divergence is meaningful. Constraint satisfaction. We define a deterministic indicator \mathbb{I}_{\mathcal{C}}(y)=1 if y satisfies all constraints in \mathcal{C} , and 0 otherwise. In addition, semantic divergence is only well-defined when a candidate is comparable to the reference at a coarse structural level. We therefore compute a comparability indicator \mathbb{I}_{\mathrm{cmp}}(y)\in\{0,1\} via normalized AST/CFG validation (Section 3.4 ). Candidates that fail this gate are treated as constraint violations in m(\cdot) . 3.4 Structural Validation (Comparability Gate) We parse y and