ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
Ayush Nangia, Shikhar Mishra, Aman Gokrani, Paras Chopra · 2026
Read ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads? by Ayush Nangia, Shikhar Mishra, Aman Gokrani, Paras Chopra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19594.txt.
Introduction
Ayush Nangia and three co-authors introduce ISO-Bench, a benchmark testing whether coding agents can optimize real-world inference workloads. Tasks come from vLLM and SGLang, two popular language model serving frameworks. Each task gives an agent a codebase and bottleneck description, requiring an optimization patch judged against expert human solutions. Fifty-four tasks were curated from merged pull requests with measurable improvements.
Runtime-based metrics alone can be gamed, passing tests without capturing the code change's actual intent. ISO-Bench therefore combines hard execution-based metrics with soft language-model-based metrics, showing both are necessary for complete evaluation. No single agent dominates across codebases, and agents often spot correct bottlenecks but fail to follow through.
Catalog record
| CREATOR | Ayush Nangia, Shikhar Mishra, Aman Gokrani, Paras Chopra |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-19594
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads? Ayush Nangia, Shikhar Mishra, Aman Gokrani, Paras Chopra Abstract We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the most popular LLM serving frameworks. Each task provides an agent with a codebase and bottleneck description, whereby the agent must produce an optimization patch evaluated against expert human solutions. We curated 54 tasks from merged pull requests with measurable performance improvements. While existing benchmarks heavily use runtime-based metrics, such approaches can be gamed to pass tests without capturing the actual intent of the code changes. Therefore, we combine both hard (execution-based) and soft (LLM-based) metrics to show that both are necessary for complete evaluation. While evaluating both closed and open-source coding agents, we find no single agent dominates across codebases. Surprisingly, agents often identify correct bottlenecks but fail to execute working solutions. We also show that agents with identical underlying models differ substantially, suggesting scaffolding is as important as the model. 1 Introduction LLM inference engines have become essential for deploying large language models at scale. Systems like vLLM [ 9 ] and SGLang [ 21 ] handle production workloads in industry and research, achieving high throughput through systems-level optimizations. Methods such as PagedAttention [ 9 ] and FlashAttention [ 4 ] required extensive work and in-depth knowledge in memory management, kernel development, and scheduling. The need for optimization is growing as more models are released and model architectures continuously evolve. LLM-based coding agents have become powerful tools for software engineering, capable of finding bugs and generating patches across codebases. Systems like SWE-Agent [ 19 ] and OpenHands [ 18 ] perform well on benchmarks like SWE-bench [ 8 ] . However, recent benchmarks suggest these agents still struggle with optimization tasks. KernelBench [ 14 ] finds that frontier models match GPU kernel baselines in under 20% of cases, GSO [ 15 ] reports success rates below 5% on repository-level tasks, and SWE-Perf [ 7 ] observes large gaps between agent and expert solutions. These benchmarks measure whether agents succeed, but not why they fail. GSO takes a step further by combining execution metrics with LLM-as-a-Judge, but a key question remains: when agents fail, do they misunderstand the problem, or do they understand it but struggle to implement the solution? In this work, we present ISO-Bench, a benchmark of 54 optimization tasks from vLLM and SGLang. Beyond measuring throughput gains, we evaluate whether agents target the correct bottleneck and use appropriate strategies. Our contributions are mentioned as follows: 1. Benchmarking Tasks : 54 optimization tasks extracted from real commits in vLLM and SGLang. Each task includes a repository snapshot, throughput and latency benchmarks, and correctness tests. 2. Dual evaluation framework : Hard and soft metrics are introduced that distinguish true successes from lucky wins, showing that traditional metrics might overestimate agent capabilities. 3. Behavioral insights : Identification of an understanding-execution gap as a primary failure mode and showing that agent performance varies substantially across codebases. 2 Related Work Figure 1 : ISO-Bench evaluation pipeline. Given a codebase and task description, a coding agent produces an optimization patch. We compare this patch against the human commit using hard metrics (TTFT, throughput) and soft metrics (bottleneck targeting, implementation approach). Hard metrics measure performance improvement; soft metrics assess whether the agent targeted the correct code. We review earlier works on evaluating and building LLM systems for code generation, focusing on: (i) benchmarks evolving from correctness to efficiency, (ii) coding agent architectures, and (iii) LLM-based evaluation methods. Correctness-driven benchmarks: Early benchmarks for code generation focused on measuring functional correctness for standalone functions. HumanEval [ 2 ] introduced 164 hand-crafted Python problems with unit tests and popularized the pass@ k metric, which measures whether at least one of k sampled solutions passes all tests. However, such function-level benchmarks evaluate code generation separate from the complexities of real software development. Repository-scale benchmarks are more realistic as agents must navigate the full codebase, find the relevant code, and edit multiple files. SWE-bench [ 8 ] constructs tasks from real GitHub issues in popular Python repositories, requiring agents to pass executable tests. SWE-bench Verified [ 3 ] filters for reproducible evaluation and has become a standard target for coding agents. Efficiency-driven benchmarks: Optimizing for performance is a different problem: agents must find the bottleneck, and success is measured by actual speedup rather than just correctness. At the kernel level, KernelBench [ 14 ] evaluates LLMs on generating efficient GPU kernels for 250 PyTorch ML workloads, introducing the fast_p metric to count solutions that are both correct and achieve greater than p\times speedup over a PyTorch baseline; even strong models succeed on fewer than 20% of tasks. Complementing this, TritonBench [ 10 ] offers two evaluation suites: TritonBench-G (GitHub-sourced operators) and TritonBench-T (PyTorch-aligned tasks), profiling Triton code against reference implementations and reporting both correctness and GPU efficiency. At the repository level, several recent benchmarks investigate if agents can optimize real-world codebases. SWE-Perf [ 7 ] constructs tasks from pull requests that improve performance, ensuring that patches are applied successfully, pass tests, and produce quantifiable speedups. GSO [ 15 ] evaluates agent patches against human expert commits rather than fixed thresholds, while SWE-fficiency [ 11 ] scales this methodology across a broader set of Python libraries, reporting how close agents come to expert-level improvements. Across all three repository-level benchmarks, agents consistently struggle to identify bottlenecks and understand low-level performance. Coding Agent Architectures: Recent work has shown that agent scaffolding significantly impacts software engineering performance. SWE-Agent [ 19 ] demonstrated that agent-computer interfaces are important for better code manipulation, while OpenHands [ 18 ] complements this by providing an open platform for designing and evaluating coding agents using standardized tools and benchmarks. TRAE-Agent [ 16 ] targets repository-level tasks by generating candidate patches, pruning them, and selecting a final solution. In contrast, commercial systems such as Claude Code [ 1 ] and Codex CLI [ 13 ] couple the model with proprietary scaffolding, which makes it difficult to separate gains from the underlying LLM versus the agent architecture. LLM-as-a-Judge for Code Evaluation: Using LLMs to evaluate code has gained traction as a scalable alternative to test-based evaluation. 20 showed that strong LLM judges can match human preferences with over 80% agreement. ICE-Score [ 22 ] applies this to code by guiding LLMs through step-by-step assessment, while CodeJudge [ 17 ] enhances accuracy by asking LLMs to think about error categories before scoring. Despite their promises, LLM judges suffer from biases such as favoring lengthier outputs and preferring their own generations, while code evaluation brings additional biases like susceptibility to misleading remarks [ 12 ] . 3 ISO-Bench Existing benchmarks evaluate either standalone kernel generation (KernelBench, TritonBench) or general repository optimization (GSO, SWE-Perf, SWE-fficiency), but none target the specific challenges of GPU-based inference serving systems. ISO-Bench fills this gap by focusing on execution-based metrics for real-world, GPU-based inference optimization workloads. Figure 1 shows the end-to-end ISO-Bench evaluation pipeline, from task inputs and agent-generated patches to hard/soft metrics and correctness validation. In this section, we describe how tasks are formulated, how the benchmark was constructed, and how we