Murphy: Multi-Turn GRPO for Self Correcting Code Generation

Chanakya Ekbote AWS AI Labs [email protected], Vijay Lingam AWS AI Labs [email protected], Behrooz Omidvar-TehraniAWS AI Labs, Jun HuanAWS AI Labs, Sujay SanghaviAmazon, Anoop DeorasAWS AI Labs, Stefano SoattoAWS AI Labs · 2025

Read Murphy: Multi-Turn GRPO for Self Correcting Code Generation by Chanakya Ekbote AWS AI Labs [email protected], Vijay Lingam AWS AI Labs [email protected], Behrooz Omidvar-TehraniAWS AI Labs, Jun HuanAWS AI Labs, Sujay SanghaviAmazon, Anoop DeorasAWS AI Labs, Stefano SoattoAWS AI Labs (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07833.txt.

Introduction

Chanakya Ekbote, Vijay Lingam, and Sujay Sanghavi extend group-based reinforcement learning to code that improves over several rounds. Standard GRPO learns from one prompt and one answer, but real coding agents read error feedback and revise. Single-turn training wastes that loop.

MURPHY builds rollout trees where failed attempts branch into feedback-conditioned retries. Retrospective credit assignment then decides which earlier choices deserve blame or praise for later success. Self-correcting generation becomes the training target itself. Code assistants that debug their own work need exactly this multi-turn credit logic, since final success often redeems a flawed first draft.

Catalog record

CREATORChanakya Ekbote AWS AI Labs [email protected], Vijay Lingam AWS AI Labs [email protected], Behrooz Omidvar-TehraniAWS AI Labs, Jun HuanAWS AI Labs, Sujay SanghaviAmazon, Anoop DeorasAWS AI Labs, Stefano SoattoAWS AI Labs
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07833

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Murphy: Multi-Turn GRPO for Self Correcting Code Generation Chanakya Ekbote AWS AI Labs [email protected], Vijay Lingam AWS AI Labs [email protected], Behrooz Omidvar-TehraniAWS AI Labs, Jun HuanAWS AI Labs, Sujay SanghaviAmazon, Anoop DeorasAWS AI Labs, Stefano SoattoAWS AI Labs Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful framework for enhancing the reasoning capabilities of large language models (LLMs). However, existing approaches such as Group Relative Policy Optimization (GRPO) and its variants, while effective on reasoning benchmarks, struggle with agentic tasks that require iterative decision-making. We introduce Murphy , a multi-turn reflective optimization framework that extends GRPO by incorporating iterative self-correction during training. By leveraging both quantitative and qualitative execution feedback, Murphy enables models to progressively refine their reasoning across multiple turns. Evaluations on code generation benchmarks with model families such as Qwen and OLMo show that Murphy consistently improves performance, achieving up to a 8 % 8\% relative gain in pass@1 over GRPO, on similar compute budgets. 1 Introduction “The road to wisdom? Well, it’s plain and simple to express: err and err and err again, but less and less and less.” —Piet Hein Qwen3-1.7B Qwen3-4B OLMo-2-7B-Ins 0 2 4 6 8 10 2.8 2.1 3.2 6.6 2.9 7.4 Models Percentage (\%) GRPO Murphy Figure 1: Percentage change in coding problems solved by models trained with Murphy and GRPO over the base model across three models and datasets. Murphy -trained models solve up to 4.2\% more problems than GRPO. See Tab. 1 for details. A growing body of work explores large language models (LLMs) as software engineering agents that interact with their environment through code execution and feedback ( Shinn et al. 2023 ; Lingam et al. 2025 ; Miserendino et al. 2025 ) . Rather than producing a single static response, these systems operate within agentic scaffolds ( Yao et al. 2023 ; Zhou et al. 2024 ) that guide iterative reasoning and allow LLMs to act, observe, and improve over multiple rounds of interaction. For example, in a typical coding agnetic scaffold ( Shinn et al. 2023 ; Lingam et al. 2025 ) , the agent generates a solution by performing a single or series of actions, and executes it for evaluation, often through unit tests or other automated checks. When execution fails, the agent receives feedback such as error messages, stack traces, or failing test cases, and is re-prompted with the original task, its previous attempt, and the new feedback. This process continues for multiple turns until the model succeeds or reaches a fixed iteration limit. These systems highlight the growing ability of LLMs to reason and adapt through environmental feedback at inference time. In code generation tasks ( Jiang et al. 2025 ) , such feedback naturally arises from execution logs, compiler errors, or test results. However, these methods ( Shinn et al. 2023 ; Lingam et al. 2025 ) remain fundamentally training-free : they improve model behavior through structured inference via feedback rather than through parameter updates. On the other hand, training methodologies such as Reinforcement Learning with Verifiable Rewards (RLVR) have enabled a new generation of language models ( OpenAI et al. 2024 ; DeepSeek-AI et al. 2025 ; Team et al. 2025 ; Yang et al. 2025 ) to exhibit strong reasoning capabilities across mathematics, coding, and general problem-solving. Recent RLVR algorithms, including Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) and its extensions ( Yue et al. 2025 ; Yu et al. 2025 ; Xu et al. 2025a ) , have become dominant approaches for post-training LLMs on verifiable reasoning tasks. However, GRPO and its extensions are fundamentally designed for a single-turn setting: they optimize model behavior using an isolated prompt–response–reward tuple, with no notion of multi-turn interaction defined in their objective. Here, a turn denotes one cycle in which the model receives a prompt (which includes feedback from its previous response), reasons based on the prompt, and generates a revised output. Taken together, these limitations underscore a methodological gap: while inference-time agentic frameworks exploit iterative feedback to refine reasoning, GRPO optimizes solely from terminal rewards on a given prompt, without leveraging intermediate environmental feedback . This motivates the following key research question. Key Research Question: How can RLVR algorithms be extended to integrate iterative feedback, and to what extent does this improve reasoning and performance within agentic frameworks? To address this question, we propose Murphy , a novel RLVR algorithm that extends GRPO to a multi-turn setting by conditioning optimization on intermediate environmental feedback. Extending GRPO beyond a single turn is non-trivial: it requires defining how rewards obtained in later turns should be propagated backward to earlier attempts, so that intermediate reasoning and output turns that initially failed but ultimately led to success through feedback receive appropriate credit. At the first turn, Murphy generates G_{1} generations per prompt and computes group-based rewards as in GRPO. In subsequent turns, generations that fail to achieve the maximum reward, e.g., those failing unit tests or producing incorrect outputs, are refined using signals from the environment such as executor logs or test results. This feedback is appended to the original prompt, and the model is re-prompted to generate a new batch of G_{s} generations (where s denotes the turn) conditioned on the combined context (prompt, previous output, and feedback), repeating this process for a fixed number of turns. Rewards from successful final-turn rollouts are then propagated backward to earlier turns using Murphy ’s credit-assignment criterion, allowing partially correct but improving attempts to receive credit. To manage the computational cost of multi-turn updates, Murphy employs pruning strategies that retain only promising trajectories while bounding total gradient updates per turn. In summary, our main contributions are: Main Contributions. 1. We introduce Murphy , a multi-turn RLVR algorithm that extends GRPO to leverage execution feedback for grounded reasoning and self-correction in code generation tasks. ( Sec. 4 ) 2. We design pruning mechanisms that enable scalable multi-turn RLVR training ( Murphy ) while preserving performance. ( Subsec. 4.1 ) 3. We evaluate Murphy on three code generation benchmarks across two model families (OLMo, Qwen) and sizes (1.7B–7B), achieving up to +5\% pass@1 over GRPO baselines. ( Sec. 5 ) 2 Related Work LLM Agents for Software Development. Recent studies ( Jiang et al. 2025 ; Zhong et al. 2024 ) investigate LLM agents for code generation, bug fixing, and code migration. A central factor behind their progress is inference-time iterative frameworks ( Shinn et al. 2023 ; Lingam et al. 2025 ) , which leverage execution feedback and self-reflection to refine candidate programs ( Yang et al. 2024 ; Xia et al. 2025 ) . While such methods enhance inference pipelines, they leave the base model unchanged. In contrast, our work improves the model itself through training-time optimization, strengthening the reasoning and self-correction abilities that agentic frameworks depend on. RVLR for LLM Reasoning. RL has emerged as a powerful paradigm for aligning LLMs with verifiable objectives. GRPO ( Shao et al. 2024 ) renewed interest in RL as an efficient alternative to PPO ( Schulman et al. 2017 ) , achieving comparable reasoning performance with lower computational cost. Follow-up variants ( Yue et al. 2025 ; Yu et al. 2025 ; Yuan

Read the complete text