ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models CCS: Software and its engineering Automatic programming CCS: Software and its engineering Open source model
Chengran Yang, Hong Jin Kang, Jieke Shi, David Lo · 2024
Read ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models CCS: Software and its engineering Automatic programming CCS: Software and its engineering Open source model by Chengran Yang, Hong Jin Kang, Jieke Shi, David Lo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17264.txt.
Introduction
Chengran Yang, Hong Jin Kang, Jieke Shi, and David Lo present ACECode, a training framework for code-generating language models, in a paper published in two thousand twenty-four. These models often write code that works but runs slowly. Slow code wastes energy and hurts software in resource-limited settings.
Earlier fixes traded correctness for speed or needed ready-made test environments. ACECode instead uses reinforcement learning to pursue both goals together. An actor model generates code samples. A reward signal comes from actually executing that code, with no predefined test cases required. The signal scores efficiency and correctness jointly. Fine-tuning against it teaches the model to produce fast code that still passes.
Catalog record
| CREATOR | Chengran Yang, Hong Jin Kang, Jieke Shi, David Lo |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17264
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models CCS: Software and its engineering Automatic programming CCS: Software and its engineering Open source model Chengran Yang, Hong Jin Kang, Jieke Shi, David Lo Abstract . CodeLLMs have demonstrated remarkable advancements in software engineering tasks. However, while these models can generate functionally correct code, they often produce code that is inefficient in terms of runtime. This inefficiency is particularly problematic in resource-constrained environments, impacting software performance and sustainability. Existing approaches for optimizing code efficiency for CodeLLMs like SOAP and PIE exhibit certain limitations. SOAP requires a compatible execution environment and predefined test cases for iterative code modification, while PIE focuses on instruction tuning, improving efficiency but compromising correctness. These shortcomings highlight the need for a fine-tuning framework that optimizes both efficiency and correctness without relying on predefined test cases or specific execution environments. To bridge this gap, we introduce ACECode, a reinforcement learning-based fine-tuning framework that aligns CodeLLMs with dual objectives of efficiency and correctness. ACECode combines three key steps: (1) generating code with an actor CodeLLM, (2) calculating a training-free reward signal derived from code execution feedback for each generated code, and (3) optimizing the CodeLLM via Proximal Policy Optimization (PPO) algorithm. This reward signal enables joint assessment of efficiency and correctness without manual labeling. We evaluate ACECode by fine-tuning four SOTA (state-of-the-art) CodeLLMs and comparing their code with three baselines: original, instruction-tuned, and PIE-tuned CodeLLMs. Extensive experiment results suggest that ACECode significantly improves both the efficiency and correctness of generated code against all baselines for all CodeLLMs. Specifically, CodeLLMs fine-tuned with ACECode improve pass@1 by 1.84% to 14.51% and reduce runtime in 65% to 72% of cases compared to original CodeLLMs. 1. Introduction Large Language Models (LLMs), such as GPT ( Achiam et al. 2023 ) and Qwen ( Bai et al. 2023 ) , have demonstrated remarkable advancements across various domains including software engineering ( Wang et al. 2024 ; Fan et al. 2023 ; Hou et al. 2023 ) . Specialized LLMs for code like Code Alpaca ( Chaudhary 2023 ) , WizardCoder ( Luo et al. 2023 ) , Magicoder ( Wei et al. 2023 ) , and WaveCoder ( Yu et al. 2023 ) , known as CodeLLMs, are built on general-purpose LLMs and fine-tuned with code datasets spanning multiple programming languages. These CodeLLMs demonstrate exceptional performance across a range of code generation tasks, such as vulnerability patch generation ( Pearce et al. 2023 ) and code completion ( Guo et al. 2023 ; Izadi et al. 2024 ) . While current CodeLLMs are capable of generating functionally correct code, this often comes at the expense of code efficiency . CodeLLM-generated code tends to be suboptimal in terms of runtime (i.e., the execution time for generated code), with prior studies ( Huang et al. 2024b ; Huang et al. 2024a ; Shi et al. 2024 ; Niu et al. 2024 ) showing that it can require 3-13 \times more runtime than human-written code. The inefficiency of CodeLLM-generated code poses a significant challenge as CodeLLMs are increasingly integrated into real-world software development workflows. For instance, Google reports that CodeLLMs now generate over a quarter of new code in their products ( Kelly 2024 ) . As such, inefficient code generated by CodeLLMs can significantly hinders the performance of software products and reduces their competitiveness, particularly in resource-constrained scenarios like mobile devices, cloud servers, and IoT systems. On the other hand, optimizing the efficiency of CodeLLM-generated code can contribute to environmental sustainability by enabling software products to consume less energy and resources, ultimately lowering their carbon footprint ( Shi et al. 2024 ) . To date, few approaches have attempted to address this issue, including SOAP ( Huang et al. 2024a ) and PIE ( Shypula et al. 2023 ) . SOAP ( Huang et al. 2024a ) adopts a two-stage inference process, where CodeLLMs first generate code that is executed with test cases to collect runtime feedback. This feedback is then used to guide a second round of code generation aimed at reducing execution overhead. However, SOAP requires a compatible execution environment for LLM-generated code and pre-defined test cases for each task, which significantly restricts its out-of-the-box usability. Moreover, the two-stage inference process doubles the inference time and introduces additional overhead during code execution On the other hand, PIE ( Shypula et al. 2023 ) crafts a dataset with efficient code snippets and applies instruction tuning for CodeLLMs to optimize code efficiency. While PIE optimizes code efficiency without relying on execution feedback, it applies instruction tuning with next-token prediction as its training objective. This approach primarily optimizes CodeLLMs as general language models, rather than addressing task-specific, multi-objective goals including code efficiency and correctness. Consequently, it leaves a gap in tailored fine-tuning methods specifically designed to optimize code efficiency. Moreover, a recent empirical study ( Waghjale et al. 2024 ) highlights a crucial drawback of existing approaches including PIE: while code efficiency improves with these methods, they often do so at the expense of functional correctness (referred to as correctness). This drawback underscores the need for a fine-tuning framework specifically designed to optimize both code efficiency and correctness in CodeLLMs, eliminating the dependency on pre-defined test cases and a compatible execution environment during inference stage. To address these limitations of existing approaches, we draw on techniques used for LLM alignment ( Ouyang et al. 2022 ; Wang et al. 2023b ; Shen et al. 2023 ) —a methodology originally developed to tailor general LLM outputs to human preferences. LLM alignment often applies reinforcement learning from human feedback (RLHF) as a fine-tuning approach ( Ouyang et al. 2022 ; Wang et al. 2023b ; Shen et al. 2023 ) . In RLHF, the LLM generates responses evaluated by a reward model (referred to as a "rewarder"), which estimates how much a human might favor the responses. The LLM’s parameters are then iteratively updated to maximize the reward signals with reinforcement learning algorithms, progressively aligning the model with human preferences. In this work, we extend RLHF principles by tailoring specific LLM output, i.e., generated code , to meet fine-grained code quality requirements, i.e., code efficiency and correctness . Applying RLHF to CodeLLMs to align the CodeLLM-generated code with correctness and efficiency requirements presents two primary challenges. First, RLHF typically requires a manually labeled dataset where LLM outputs are annotated with human preferences ( Ouyang et al. 2022 ; Dai et al. 2023 ; Wang et al. 2023b ; Shen et al. 2023 ) to train the reward model. Creating such a dataset is time-consuming and labor-intensive, requiring deep expertise in code algorithms and carrying the risk of subjective biases. The second challenge lies in representing both code efficiency and correctness within a rewarder, enabling the model to optimize for the dual objectives. To address both challenges, we propose ACECode ( A ligning Code C orrectness and E fficiency for Code LLMs), a reinforcement learning-based fine-tuning framework incorporating a training-free rewarder derived from code execution. ACECode eliminating the need for manually labeled datasets to train a reward model. Instead, it introduces a training-free, dual-objective rewarder that jointly evaluates code efficiency and correctness. Specifically, our