GSO : Challenging Software Optimization Tasks for Evaluating SWE-Agents

Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, Ion Stoica · 2025

Read GSO : Challenging Software Optimization Tasks for Evaluating SWE-Agents by Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, Ion Stoica (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23671.txt.

Introduction

Manish Shetty and five co-authors build a test set from 2025 for judging how well AI agents write fast code. Speeding up real software takes rare skill. Their pipeline mines the change history of ten code bases and pulls out 102 hard optimization tasks, each with a test that measures run time against an expert's fix.

Top software agents solved fewer than five percent of the tasks. Giving agents more compute at test time helped only a little. The authors sort the failures: weak skill in low-level languages, lazy shortcuts, and poor pinpointing of slow spots. They release the tasks and agent logs to spur better tools.

Catalog record

CREATORManish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, Ion Stoica
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23671

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GSO : Challenging Software Optimization Tasks for Evaluating SWE-Agents Manish Shetty, Naman Jain, Jinjian Liu, Vijay Kethanaboyina, Koushik Sen, Ion Stoica Abstract Developing high-performance software is a complex task that requires specialized expertise. We introduce GSO , a benchmark for evaluating language models’ capabilities in developing high-performance software. We develop an automated pipeline that generates and executes performance tests to analyze repository commit histories to identify 102 102 challenging optimization tasks across 10 10 codebases, spanning diverse domains and programming languages. An agent is provided with a codebase and performance test as a precise specification , and tasked to improve the runtime efficiency, which is measured against the expert developer optimization. Our quantitative evaluation reveals that leading SWE-Agents struggle significantly, achieving less than 5% success rate, with limited improvements even with inference-time scaling. Our qualitative analysis identifies key failure modes, including difficulties with low-level languages, practicing lazy optimization strategies, and challenges in accurately localizing bottlenecks. We release the code and artifacts of our benchmark along with agent trajectories to enable future research. Website: https://gso-bench.github.io/ 1 Introduction Figure 1 : An example GSO task. We develop an automated pipeline that generates performance tests and analyzes repository commit history to identify real-world code optimization tasks. Each task consists of a codebase, performance tests, and the expert developer commit that serves as the performance target for the optimization problem. LLM -based SWE-Agents are then tasked with generating optimization patches using the performance test as a precise specification for the optimization problem. We evaluate the patches for both correctness and runtime efficiency, measuring whether they match or exceed the human expert optimization performance while ensuring equivalence. High-performance software is critical for modern computing systems, from data analytics frameworks to machine learning infrastructure. Developing such systems demands specialized expertise in algorithmic optimization, hardware-aware programming, performance analysis, and reasoning across multiple layers of the software stack. The complexity of these tasks is evident in production-critical systems like VLLM ( Kwon et al., 2023 ) , HPC ( Bradski, 2000 ) , and VERL ( Sheng et al., 2024 ) , where teams dedicate substantial efforts to iterative and continuous maintenance over long development cycles. Simultaneously, SWE-Agents are gaining rapid traction in software development, demonstrating remarkable results on simple bug-fixing tasks ( Jimenez et al., 2024 ) . This has also spurred excitement in adapting LLMs to aid in automating research tasks themselves, for example improving deep learning kernels ( Ouyang et al., 2024 ) . In this work, we study the question – “Can LLM agents aid in the development of high-performance software?”. To answer this, we introduce GSO , a benchmark for evaluating SWE-Agents on challenging software optimization tasks. To create GSO , we develop an automated pipeline that generates performance tests and runs them across a repository’s commit history to identify substantial optimizations discovered by expert developers. After careful manual curation, we extract 102 challenging tasks across 10 codebases, spanning diverse domains and languages including \mathtt{Python} , \mathtt{C} , and \mathtt{SIMD} . Each task consists of a codebase , performance tests exercising real-world workloads, and a target optimization from expert developer commits. SWE-Agents receive a performance test as task specification and must produce an optimization patch that improves runtime efficiency while maintaining correctness. We evaluate these patches using our \textsc{\small Opt}\textsc{\small@}K metric, providing reliable assessment in a machine-agnostic manner. Rather than naively measuring machine-dependent speedups, we assess whether model-generated patches can consistently match or exceed the performance of human expert optimizations. Our benchmark evaluates the capabilities needed for high-impact optimization work, tracking usefulness for real-world high-performance software development. Particularly, problems in GSO evaluate challenging systems engineering tasks, including optimizing Pandas operations, Pillow image or video processing operations (like GIF animation), and LLaMA-CPP model inference runtimes. Code optimization uniquely bridges algorithmic reasoning and systems engineering, providing a challenging yet well-specified evaluation domain for LLM -based programming agents. Unlike bug-fixing SWE benchmarks that rely on potentially ambiguous natural language specifications ( Aleithan et al., 2024 ) , performance tests natively provide precise specifications for correctness and efficiency. Our tasks require substantial code changes, with gold-patches containing 4 - 15\times more lines edited than previous benchmarks (Figure 2 -middle). We evaluate leading LLMs on GSO using the state-of-the-art OpenHands agent framework ( Wang et al., 2024b ) (Section 3 ). Our evaluation reveals that most agents struggle with the benchmark, achieving less than 5% success rate measured by \textsc{\small Opt}\textsc{\small@}1 , with test-time compute also providing only modest improvements ( \textsc{\small Opt}\textsc{\small@}10 remaining around 15%). Benchmark Repo Level Evaluate Runtime Multi- Lingual Precise Specification HumanEval ✗ ✗ ✗ ✓ EvalPerf ✗ ✓ ✗ ✓ ECCO ✗ ✓ ✗ ✓ LiveCodeBench ✗ ✗ ✗ ✓ KernelBench ✗ ✓ ✗ ✓ SWEB-Verified ✓ ✗ ✗ ✗ SWEB-Multi ✓ ✗ ✓ ✗ MultiSWE-Mini ✓ ✗ ✓ ✗ GSO (Ours) ✓ ✓ ✓ ✓ Figure 2 : Benchmark Feature Comparison and Performance Gap. Left: Depicting how GSO improves over existing benchmarks across key dimensions. Middle: Distribution of oracle LoC changes across benchmarks, showing GSO solutions require over 4-15x larger edits than existing benchmarks. Right: Performance comparison of O4-Mini across LCB (algorithmic puzzles), SWEBench-Verified (repository-level bug-fixes), and GSO depicting the performance gap on optimization tasks. To understand why SWE-Agents struggle with GSO , we perform a qualitative analysis of agent behavior and failure modes (Section 5 ). First, agents struggle with low-level languages, often avoiding them entirely or introducing fatal errors.Second, agents resort to superficial optimizations (“lazy optimizations”) like compiler flag manipulation or input-specific fast-paths insertion, often making bizarre non-idiomatic code changes. Third, localization remains challenging - agents frequently misdiagnose the root cause of performance issues, leading to ineffective optimization attempts. The key contributions of this paper are: 1) An automated pipeline leveraging test-generation and execution information for generating software optimization tasks from real-world codebases, resulting in the GSO benchmark. 2) Evaluation of leading SWE-Agents on GSO , revealing a substantial performance gap in systems engineering tasks. 3) Qualitative analysis of agent behavior and failure modes with directions for future research. Given the substantial performance gap, we believe considerable progress in reasoning capabilities and SWE-Agents will be required to close the gap and hope GSO serves as a valuable resource for future LLM -based programming agent research. Tasks # Total 102 # Codebases 10 # Languages 5 # Domains 8 Commit # Lines (avg) 250 # Lines (median) 110 # Lines (max) 2278 # Files (avg) 3.9 % Non- \mathtt{Py} 58.8% Tests # Per-task (avg) 12.45 # Per-task (max) 20 Figure 3 : Left. Popular optimization concepts and examples of algorithms used in ground-truth human commits for GSO tasks highlighting the algorithmic complexity of the tasks. Right. Summary statistics for GSO tasks, the groundtruth human commits, and the performance tests highlighting the repository-level nature of the tasks spanning diverse domains and languages. 2 GSO Global Software Optimization ( GSO ) is a benchmark for evaluating SWE-Agent capabilities for aiding in high-performance software development. Each task consists of an initial codebase snapshot, performance tests measuring runtime and correctness, a build script for environment setup, and a reference human commit establishing the target performance threshold. The goal is to generate a patch that improves the performance of the codebase while maintaining functional correctness. 2.1

Read the complete text