Evaluating LLMs on Real-World Software Performance Optimization

Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen · 2026

Read Evaluating LLMs on Real-World Software Performance Optimization by Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25530.txt.

Introduction

Making existing software faster is expert work, and judging whether a language model can do it needs honest tests. Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim, and Chunyang Chen build such a test in 2026 from one hundred two real optimisations written by experts in open source projects. Each task is scored with parameterised tests measuring runtime, peak memory, and time weighted memory use, under noisy conditions and varied inputs, because real performance is all of those at once.

Current models struggled badly. Their runtime gains were negligible and memory improvements nearly absent. The expert versions, in contrast, sped tasks up by a factor of about fifteen on aggregate and cut peak memory enormously on many tasks. Experts improved runtime in over ninety percent of tasks. The authors call the gap a measure of how far models sit from expert level engineering judgement.

Catalog record

CREATOREzgi Sarıkayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25530

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating LLMs on Real-World Software Performance Optimization Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen Abstract Software performance optimization is a notoriously complex and manual task. Despite the growing use of Large Language Models (LLMs) for code refinement, we still lack benchmarks that capture how optimization actually happens in real-world codebases. Existing frameworks often oversimplify the problem by focusing on isolated functions or a single performance metric, missing the critical trade-offs between execution time and memory footprint, the inherent noise of the measurement environment, and the variability introduced by different input data and execution conditions. We address this by introducing SWE-Pro, a repository-level benchmark derived from 102 expert-written optimizations from open-source projects. Unlike previous benchmarks, SWE-Pro pairs each task with parameterized tests to evaluate runtime, peak memory, and Time-Weighted Memory Usage (TWMU) across varying input data and execution conditions under noise-aware measurement conditions. Our evaluation shows that current LLMs struggle significantly: runtime gains are negligible, and memory optimizations are nearly non-existent. This stands in sharp contrast to expert implementations, which achieve an aggregate speedup of 15.5 × 15.5\times and peak memory reduction of 171.3 × 171.3\times over benchmark tasks. Expert-written improvements are observed in 91.2 % 91.2\% of tasks for runtime and 65.7 % 65.7\% for peak memory. Our findings expose a substantial gap between current LLM capabilities and the demands of expert-level engineering. 1 Introduction In recent years, LLMs have fundamentally reshaped software engineering, demonstrating remarkable proficiency in domains such as code generation 13 and automated bug fixing 42 . As these models increasingly automate large-scale software production, the evaluative focus is pivoting from basic functional correctness toward long-term sustainability and execution efficiency. In this AI-driven development era, sub-optimal code can rapidly aggregate into "performance debt," significantly inflating computational overhead 15 . Consequently, this technological shift has catalyzed interest in LLM-based code performance optimization—the critical task of refining implementations to enhance runtime and memory efficiency without altering functional semantics. Despite its importance, performance optimization remains notoriously challenging 25 . Unlike binary correctness, which provides deterministic pass/fail signals, optimization is inherently multidimensional and context-sensitive. A single optimization may involve conflicting trade-offs between runtime and memory usage, and its efficacy often fluctuates across different execution environments 33 . Moreover, performance is inextricably linked to workload characteristics; measurements are dynamic, context-dependent observations rather than static attributes 41 . Given the increasing deployment of LLMs for such tasks, a rigorous benchmark is essential to evaluate their ability to navigate the complexities of real-world software performance. Current evaluation paradigms have laid a foundational groundwork but face scalability and precision issues when transitioning from function-level to repository-level analysis. Early benchmarks like EffiBench 10 and COFFE 28 assess individual functions in isolation, often neglecting the complex dependencies inherent in real-world repositories. While recent repository-level benchmarks such as SWE-Perf 1 , SWE-FFICIENCY 20 , and GSO 35 operate at scale, they often lack sufficient evaluative rigor. Common pitfalls include using unit tests as a crude proxy for performance 1 or aggregating heterogeneous execution variants into coarse-grained scores 35 , which can obscure input-specific performance fluctuations. Such approaches offer a restricted view of how optimizations generalize across varying workloads (e.g., different input sizes, data properties, functional options) 23 and typically lack the statistical rigor needed to distinguish genuine performance gains from environmental noise. These limitations underscore a fundamental gap: existing benchmarks focus on task success under fixed conditions, failing to capture the input-dependent, statistically grounded, and multi-metric nature of modern performance optimization. This deficit motivates our investigation into whether LLMs can truly achieve robust repository-level optimization when subjected to rigorous, multi-workload scrutiny. To address this, we present SWE-Pro, a comprehensive repository-level benchmark comprising 102 expert-written optimization cases curated from real-world open-source projects. SWE-Pro is designed to shift the evaluation paradigm toward robust, workload-aware analysis. Unlike prior benchmarks, SWE-Pro evaluates tasks across diverse workloads via parameterized performance tests that reflect realistic execution scenarios. By conducting systematic parameter sweeps over input scales, data characteristics and function properties, our framework exposes performance behaviors that remain invisible to single-input benchmarks. To ensure reliability, we introduce an adaptive measurement mechanism: rather than relying on fixed-count profiling, SWE-Pro dynamically determines the necessary iterations based on the statistical convergence of the 90% Relative Confidence Interval Width (RCIW) . This effectively filters out system variance, enabling the reproduction of 87% of expert-level improvements. Furthermore, we employ a multi-metric framework—incorporating peak memory and TWMU alongside runtime—to uncover critical time–memory trade-offs, providing a nuanced perspective on LLM-generated optimizations in resource-constrained contexts. Through extensive experiments on state-of-the-art LLMs, we provide the first systematic study of repository-level code optimization that jointly evaluates runtime and memory performance across varying workloads under noise-aware conditions. Our findings reveal that while modern models excel at local logic refinements, they struggle significantly to deliver measurable efficiency gains; their runtime improvements remain negligible, and memory optimizations are nearly non-existent. This stands in sharp contrast to expert implementations within our benchmark, which achieve 15.48\times runtime speedup, 171.31\times peak memory reduction, and 619.22\times TWMU reduction, with 91.2% of tasks showing reproducible runtime improvements, 65.7% showing reliable peak memory gains, and 52.0% showing TWMU improvements. Such a substantial performance gap suggests clear trajectories for future research in developing LLMs that can navigate the demands of expert-level engineering. To support the community, We open-sourced SWE-Pro and our automated evaluation pipeline at https://github.com/probench-swe/SWE-Pro . 2 PRO-Bench Table 1: Comparison of repository-level performance optimization benchmarks. Performance Scope : how performance is evaluated per task. Structured Parameter Sweep : whether test cases are generated via structured sweeps over input dimensions. Runtime and Memory : performance metrics considered. Adaptive Iterations : whether measurement iterations are dynamically determined based on statistical convergence. Benchmark Performance Scope Structured Parameter Sweep Runtime Memory Adaptive Iterations GSO Multiple Scripts ✗ ✓ ✗ ✗ SWE-Perf Multiple Unit Tests ✗ ✓ ✗ ✗ SWE-FFICIENCY Single Workload ✗ ✓ ✗ ✗ PRO-Bench Multiple Workloads ✓ ✓ ✓ ✓ 2.1 Preliminaries Formulation. Repository-level performance optimization benchmarks task an agent with producing a patch to a given codebase that improves runtime efficiency while preserving functional correctness. SWE-Perf 1 grounds tasks in performance-improving Pull Requests (PRs) , pairing each instance with the full repository, a set of performance-related unit tests, and an expert-authored patch; evaluation reports patch applicability, functional correctness, and runtime improvement. Building on a similar construction, SWE-fficiency 20 provides each agent with a complete codebase and a single fixed workload, requiring the agent to localize bottlenecks, edit the repository, and pass existing correctness tests; performance is quantified via the speedup ratio against a gold expert run on that workload configuration. In contrast, GSO 35 constructs multiple performance tests per task from real commit histories, where each test is an experiment function that exercises the target code path across a range of input patterns and execution variants and is timed as a unit; performance is then evaluated via the OPT@K metric, measuring whether at least one of the top-K generated patches matches or exceeds the expert runtime. Limitations. Current benchmarks are predominantly limited to runtime evaluation, entirely neglecting memory usage in their performance formulations. Furthermore, assessment under fixed or coarsely aggregated conditions significantly constrains the depth of analysis: SWE-Efficiency 20 relies on a single workload per task, precluding any assessment of patch robustness across

Read the complete text