Prompt Optimization for LLM Code Generation via Reinforcement Learning

Ali Mohammadi Esfahani (🖂), Nafiseh Kahani, Samuel A.Ajila · 2026

Read Prompt Optimization for LLM Code Generation via Reinforcement Learning by Ali Mohammadi Esfahani (🖂), Nafiseh Kahani, Samuel A.Ajila (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19102.txt.

Introduction

Ali Mohammadi Esfahani and two colleagues treat prompt improvement for code-generating language models as a sequential decision problem solved with reinforcement learning. A proximal policy optimisation agent refines prompts using direct generation, genetic word mutation, and meaning-based rewriting, guided by rewards shaped from unit-test feedback. Tested on three code benchmarks with three code generators, the agent beats methods including EPiC, Reflexion, and Random-Hybrid. On the 500-task test set it reaches pass rates of 57.58, 64.80, and 85.50 percent across the three generators.

Catalog record

CREATORAli Mohammadi Esfahani (🖂), Nafiseh Kahani, Samuel A.Ajila
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34914
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prompt Optimization for LLM Code Generation via Reinforcement Learning Ali Mohammadi Esfahani (🖂), Nafiseh Kahani, Samuel A.Ajila Abstract Large Language Models (LLMs) can generate code from natural language, but their performance is highly sensitive to prompt formulation. We propose a reinforcement-learning-based framework that models prompt refinement as a sequential decision-making problem. A Proximal Policy Optimization (PPO) agent iteratively improves prompts using a hybrid action space that combines direct generation, genetic lexical mutation and semantic rewriting, guided by shaped rewards derived from unit-test feedback. We evaluate the framework on MBPP+, HumanEval+, and APPS using CodeT5+, CodeLLaMA, and DeepSeek-Coder as frozen code generators. On the 500-task MBPP+ test set, the PPO agent achieves strict Pass@1 scores of 57.58%, 64.80%, and 85.50%, respectively, outperforming EPiC, Reflexion, and Random-Hybrid. SoftPass@1 reaches 67.90%, 73.10%, and 88.20%, respectively. Similar improvements are observed on HumanEval+ and APPS across all backbone models. The results demonstrate that reinforcement learning with shaped test-driven rewards improves functional correctness in LLM-based code generation. 1 Introduction Large Language Models (LLMs) have improved many automated software engineering tasks, particularly code generation from natural language descriptions [ 1 , 2 , 9 , 6 , 11 , 13 , 14 , 36 , 17 ] . However, generated code still frequently contains functional and logical defects, which limits its practical use. Recent studies have highlighted that even advanced LLMs exhibit relatively low exact-match accuracy, with critical defects frequently introduced into the generated code, requiring manual debugging and correction [ 10 , 31 ] . In a recent study, defects in LLM-generated code were investigated and categorized, with prevalent issues such as functionality, algorithmic, and logic defects being identified, emphasizing the necessity for targeted improvements [ 10 ] . The authors also demonstrated that standard prompt engineering techniques, such as Chain-of-Thought and Structured Chain-of-Thought prompting, can mitigate common defects by providing clearer instructions and context to the model [ 10 ] . However, these static prompting methods lack adaptability and often fall short when iterative refinement based on intermediate feedback is required. To address these limitations, several adaptive approaches have recently been proposed, leveraging evolutionary algorithms or reinforcement learning (RL) frameworks for prompt optimization. Techniques such as EPiC [ 30 ] employ lightweight evolutionary search algorithms to iteratively refine prompts based on the performance of generated code evaluated against test cases. Similarly, RL is used in Reflexion [ 29 ] , where language agents iteratively reflect on task feedback, incorporating self-generated linguistic insights into future prompt formulations. In addition, frameworks such as the Large Language Model Debugger (LDB) [ 38 ] enhance LLM-generated code by integrating runtime execution insights to iteratively pinpoint and correct defects. Despite the effectiveness of these approaches, lexical or semantic prompt modifications are typically treated in isolation, and multiple dimensions of prompt transformations are not cohesively integrated. Furthermore, current strategies mainly rely on binary correctness signals, often overlooking the rich information contained in partial correctness or detailed feedback. The contribution lies in coordinating reinforcement learning, genetic prompt mutation, and semantic rewriting through a learned sequential policy for executable code generation. To fill these gaps, we propose an RL-driven prompt optimization approach specifically tailored for functional code generation tasks. This paper investigates whether RL can improve prompt optimization for code generation by adaptively combining lexical mutation and semantic rewriting. The proposed framework learns to select between lexical mutation and semantic rewriting based on intermediate test feedback. Prompt optimization is modeled as a sequential decision-making problem, where shaped rewards and SoftPass@1 is used as an auxiliary learning-sensitive metric to leverage partial correctness signals. The main contributions of this work are as follows. • We formulate prompt optimization for code generation as a sequential decision-making problem and implement it as a PPO-based RL environment. • We define a hybrid action space that combines direct generation, lexical mutation, and semantic rewriting for adaptive multi-step prompt refinement. • We introduce a shaped reward function that uses partial test case correctness to provide denser learning signals than binary success alone. • We evaluate the framework on MBPP+, HumanEval+, and APPS using CodeT5+, CodeLLaMA, and DeepSeek-Coder, and compare it against direct generation, EPiC, Reflexion, and a Random-Hybrid baseline that uses the same action space without policy learning. The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 presents the proposed framework. Section 4 describes the experimental setup and results. Finally, Section 5 concludes the paper. 2 Related work Recent advances in LLMs have stimulated extensive research on prompt engineering and optimization for tasks such as code generation, instruction following, and program synthesis. Early approaches relied on manual prompt design, where human-crafted instructions were iteratively refined to elicit accurate model responses [ 3 ] , [ 6 ] . While effective in constrained settings, handcrafted prompts often lack generalization, motivating automated prompt optimization methods [ 37 ] , [ 35 ] , [ 12 ] . These approaches are broadly categorized into evolutionary search and RL frameworks. Evolutionary methods treat prompt optimization as population-based search. EPiC [ 30 ] applies lexical mutations (insertion, deletion, replacement) guided by test-case correctness as a fitness signal. EvoPrompt [ 15 ] incorporates LLMs into mutation and crossover operations to produce semantically coherent variants, while PromptBreeder [ 12 ] introduces self-referential co-evolution of prompts and mutation operators. Although effective for exploring lexical diversity, evolutionary approaches generally rely on predefined operators and lack adaptive learning for multi-step refinement. RL-based approaches formulate prompt optimization as sequential decision-making. RLPrompt [ 8 ] applies token-level edits using correctness-based rewards. PRewrite [ 19 ] extends the action space to include semantic rewriting via LLM-based paraphrasing. StablePrompt [ 20 ] improves training stability through Adaptive Proximal Policy Optimization, and PRL [ 4 ] explores RL-based prompt generation for few-shot tasks. However, most RL methods focus on narrow edit operations or fixed rewriting strategies and are typically evaluated on general language tasks rather than executable code generation with strict functional correctness constraints. Existing methods either rely on lexical search, fixed semantic rewriting, or sparse correctness signals. Few combine heterogeneous prompt transformations within a learned sequential policy, and even fewer evaluate them on executable code generation with unit-test-driven rewards. This work addresses those gaps by integrating lexical mutation and semantic rewriting in a unified PPO framework with shaped rewards based on partial correctness under unit-test evaluation. 3 Prompt Optimization via Reinforcement Learning Figure 1 : Workflow of the RL–based prompt optimization framework Prompt optimization is modeled as a sequential decision-making problem within an RL framework, where an agent iteratively refines natural language prompts in order to improve the functional correctness of code generated by an LLM. In this setting, each programming task corresponds to an episode of interaction with the environment. The state represents the current prompt, encoded as a semantic embedding. The action space consists of prompt transformation strategies that modify or reuse the current prompt. After a transformation is applied, the updated prompt is passed to a frozen code generation model to produce executable code. The generated program is evaluated against test cases, and the resulting correctness signal defines the reward . The objective of the agent is to learn a policy that selects prompt transformations maximizing functional correctness across refinement steps. The agent is

Read the complete text