EffiPair : Improving the Efficiency of LLM-generated Code with Relative Contrastive Feedback
Samira Hajizadeh, Suman Jana · 2026
Read EffiPair : Improving the Efficiency of LLM-generated Code with Relative Contrastive Feedback by Samira Hajizadeh, Suman Jana (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05137.txt.
Introduction
Hajizadeh and Jana introduce differential execution feedback for raising code speed. Large language models can generate correct programs that differ in speed. Current tuning methods usually refine each entry using pointwise runtime or timing feedback. That feedback shows how costly it is or where cost arises, but gives limited guidance on how it should change. It compares implementations in program space but separated in speed. It turns their differences into evidence. They show the method in EffiPair, a tool pairing similar programs with speed gaps.
Catalog record
| CREATOR | Samira Hajizadeh, Suman Jana |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42418 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EffiPair : Improving the Efficiency of LLM-generated Code with Relative Contrastive Feedback Samira Hajizadeh, Suman Jana Abstract Large language models (LLMs) often generate code that is functionally correct but inefficient in runtime and memory. Prior approaches to improving code efficiency typically rely on absolute execution feedback, such as profiling a single program’s runtime or memory usage, which is costly and provides weak guidance for refinement. We propose Relative Contrastive Feedback (RCF), an inference-time feedback mechanism that requires no model fine-tuning or parameter updates. RCF compares two structurally similar programs for the same task and highlights the differences associated with better efficiency. Building on this idea, we introduce EffiPair , an inference-time iterative refinement framework that operates entirely at test time by generating multiple candidate solutions, identifying informative program pairs with large efficiency gaps, summarizing their execution differences into lightweight feedback, and using this signal to produce more efficient solutions. By replacing isolated scalar feedback with pairwise contrastive comparisons, EffiPair provides more direct guidance while reducing profiling and prompting overhead. Experiments on code-efficiency benchmarks show that EffiPair consistently improves efficiency while preserving correctness. For instance, with DeepSeek-Chat V3.2, EffiPair achieves up to 1.5x speedup over generation without performance feedback, while reducing token usage by more than 90% compared to prior work. 1 Introduction Large language models (LLMs) are increasingly used for code generation in settings ranging from interactive “vibe coding” to production software pipelines. In many of these workflows, code is generated and refined at inference time with minimal or no additional training, placing the burden of optimization directly on the generation process. As a result, models often produce programs that are unnecessarily verbose, inefficient, or suboptimal in their use of memory and compute. When such code is reused across services or deployed at scale, even modest inefficiencies can compound into significant execution cost. Improving the efficiency of model-generated programs has therefore emerged as an important problem alongside ensuring functional correctness. Recent work has begun treating program efficiency as an explicit optimization objective rather than a fixed outcome of generation. Benchmarks such as EffiBench ( 8 ) measure execution time and memory usage of LLM-generated programs and show that model outputs are often substantially less efficient than canonical human solutions. To address this, several approaches focus on inference-time refinement , where the model iteratively generates candidate programs, executes them, and uses feedback (e.g., runtime, memory usage, or execution traces) to guide subsequent revisions. These methods treat efficiency optimization as a feedback-driven process, analogous to learning from reward signals or human preferences, but operate entirely at inference time. However, existing approaches rely primarily on absolute execution feedback , where each candidate program is evaluated independently and assigned scalar measurements such as runtime or memory usage. While informative, this form of feedback is fundamentally limited in its information efficiency: scalar measurements provide only absolute performance values without revealing the relative structure among candidate programs. As a result, models must infer optimization directions indirectly, often requiring multiple iterations, extensive profiling, and high token overhead to converge toward efficient solutions. Figure 1: Overview of EffiPair . Given a coding task, the LLM samples N candidate programs. Candidates are checked for correctness and profiled for efficiency, then stored in a candidate pool. EffiPair selects a pair consisting of an efficient reference program p^{+} and a similar but less efficient candidate p^{-} , summarizes their relative execution differences into compact Relative Contrastive Feedback (RCF), and uses this signal to refine the program. The refined program is added back to the pool for subsequent rounds. In contrast, we argue that relative performance comparisons between candidate programs provide a richer and more actionable signal for efficiency optimization. When two programs solve the same task, their relative efficiency can directly reveal which structural choices lead to better performance. Rather than requiring the model to interpret absolute profiling values in isolation, such comparisons expose the optimization-relevant differences between candidates and provide clearer guidance on how the program should be revised. Motivated by this insight, we introduce Relative Contrastive Feedback (RCF) , a new optimization signal for LLM-based program synthesis. RCF is defined as structured feedback derived from pairwise comparisons between candidate programs based on their execution behavior. Rather than treating candidate programs as isolated trajectories, RCF leverages relational information between programs to guide refinement. This transforms efficiency optimization from scalar reward maximization into a contrastive learning problem over program executions. RCF offers several fundamental advantages over absolute profiling feedback. First, it provides stronger directional guidance by explicitly identifying which program variants are more efficient. Second, it is more information-efficient, as each comparison conveys optimization-relevant structure without requiring precise scalar measurements. Third, it enables cooperative optimization across multiple candidate generations, allowing models to learn from the relationships between programs rather than from individual executions alone. These properties make RCF particularly well-suited for iterative self-refinement settings. To operationalize RCF, we introduce EffiPair , a framework that integrates RCF into the program generation and refinement loop. EffiPair executes multiple candidate programs in parallel, compares their efficiency using lightweight summarized profiling signals, and uses these comparisons to guide subsequent refinements. This pairing mechanism enables efficient information sharing across candidate programs while minimizing token and execution overhead. Across multiple code-efficiency benchmarks and strong language models, EffiPair consistently improves runtime efficiency while preserving, and in some cases improving correctness. Relative to EffiLearner ( 7 ) , EffiPair achieves comparable or better efficiency gains, higher correctness, and much lower token overhead, reducing prompt tokens from EffiLearner’s 23.6M to 200k and completion tokens from 350k to 77k. In summary, our work makes the following contributions: (1) We introduce Relative Contrastive Feedback (RCF) , a new supervision signal for LLM program synthesis that leverages pairwise execution comparisons to guide optimization. (2) We present EffiPair , a refinement framework that operationalizes RCF and integrates contrastive execution feedback into the program generation loop. (3) We show that RCF enables more efficient optimization of generated programs compared to methods based solely on absolute profiling feedback. (4) We demonstrate that summarized execution statistics are sufficient to provide effective contrastive feedback, reducing profiling overhead while preserving optimization effectiveness. Together, these results establish RCF as a principled and effective signal for improving the efficiency of LLM-generated programs and suggest a broader paradigm for execution-grounded learning based on relational feedback between candidate solutions. 2 Related Work Large language models (LLMs) are increasingly used for code generation, spanning interactive prototyping to production pipelines ( 10 ; 5 ; 11 ) . However, generated programs are often inefficient in runtime and memory ( 8 ) , leading to nontrivial cost and sustainability concerns at scale. Recent work therefore treats efficiency as an explicit optimization objective, considered alongside ( 15 ) , or even prior to, correctness ( 19 ) . A growing set of benchmarks has been proposed to evaluate this problem ( 18 ; 8 ; 4 ; 13 ; 16 ) , consistently showing that LLM-generated code lags behind expert implementations in efficiency. To address this gap, prior work explores both training-time and inference-time strategies. Training-time approaches curate performance-improving edits or incorporate efficiency-aware fine-tuning ( 17 ; 9 ) . Inference-time methods instead rely on iterative refinement using execution feedback. EffiLearner ( 7 ) feeds runtime and