SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Kepeng Lei, Yifan Yao, Xinping Lei, Wenqiang Zhu, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Wen Xiang, Ziqi Zhan, Yuanxing Zhang, Wuxuan Gong, Ziyuan · 2025
Read SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models by Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Kepeng Lei, Yifan Yao, Xinping Lei, Wenqiang Zhu, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Wen Xiang, Ziqi Zhan, Yuanxing Zhang, Wuxuan Gong, Ziyuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05459.txt.
Introduction
Jingxuan Xu and co-authors introduce SWE-Compass, a broad benchmark for the coding skills of agentic language models. Past software tests leaned hard on Python bug fixes or puzzle tasks, which miss much of real developer work and favor one tongue.
SWE-Compass spans eight task types, eight work scenes, and ten programming languages in one structured frame. It aims to match how code is built in production, from fixes to features. A unified score lets buyers and builders compare models on the full spread of software work, not a narrow slice.
Catalog record
| CREATOR | Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Kepeng Lei, Yifan Yao, Xinping Lei, Wenqiang Zhu, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Wen Xiang, Ziqi Zhan, Yuanxing Zhang, Wuxuan Gong, Ziyuan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-05459
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Kepeng Lei, Yifan Yao, Xinping Lei, Wenqiang Zhu, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Wen Xiang, Ziqi Zhan, Yuanxing Zhang, Wuxuan Gong, Ziyuan Gao, Guanxiang Wang, Yirong Xue, Mengtong Li, Mengfei Xie, Xiaojiang Zhang, Jinghui Wang, Wenhao Zhuang, Zheng Lin, Huiming Wang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Zhang, Bin Chen, Jiaheng Liu Kuaishou Technology , Nanjing University [email protected] , [email protected] , [email protected] Abstract Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass 1 1 1 https://huggingface.co/datasets/Kwaipilot/SWE-Compass , a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, we hope SWE-Compass can provide a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models. † † footnotetext: * Equal Contribution. † Corresponding Author. 1 Introduction Large language models (LLMs) trained on code have rapidly advanced from solving algorithmic puzzles to assisting with production-scale software development. Modern coding LLMs ( Team et al. 2025a ; Team et al. 2025b ; Anthropic 2025 ; Team et al. 2025c ; Team 2025 ) now exhibit strong multi-turn reasoning, long-context handling, and tool-use capabilities, enabling them to serve as autonomous coding agents that plan, edit, test, and deploy software. This shift has motivated a wave of benchmarks designed to measure their utility. However, existing evaluations fall short in capturing the full scope of real-world software engineering: most remain restricted to single-file tasks, Python-centric bug fixing, or synthetic algorithmic problems ( Zheng et al. 2023a ; Austin et al. 2021 ; Li et al. 2022 ; Jain et al. 2024 ; Zhuo et al. 2025 ) , leaving critical developer activities such as feature implementation, refactoring, configuration, and performance optimization underexplored. 0 . subfigureTask-Specific Resolve Rates 0 . subfigureLanguage Distribution Across Benchmarks (c) Comparative analysis: model performance across task types (left) and language coverage across benchmarks (right). Recent repository-grounded benchmarks, such as SWE-bench and its variants ( Jimenez et al. 2024 ; Yang et al. 2025a ; Badertdinov et al. 2025 ; Yang et al. 2025a ) , have improved ecological validity by embedding evaluations in real issues, integrating test oracles, and introducing multi-language ( Rashid et al. 2025 ; Yang et al. 2025a ) or multimodal ( Yang et al. 2025b ) extensions. Yet these efforts largely converge on bug fixing as the dominant evaluation axis. As a result, they neglect the breadth of software engineering workflows that unfold across diverse scenarios—ranging from infrastructure and security engineering to machine learning system development—and across heterogeneous programming ecosystems. This narrowness prevents systematic capability diagnosis and obscures whether strong performance arises from generalizable reasoning or from artifact-specific adaptation. To address these limitations, as shown in Figure 1 , we present SWE-Compass , a unified benchmark comprising 2,000 verified instances for evaluating LLMs’ agentic coding abilities. SWE-Compass spans 8 task types, 10 programming scenarios, and 10 programming languages, combining broad coverage with rigorous evaluation fidelity. Each instance is paired with executable environments and reproducible tests, enabling fair comparison across prompting and agent-based methods under controlled budgets. Importantly, SWE-Compass is built upon four design principles: (i) real-world alignment, ensuring data originates from genuine developer interactions; (ii) comprehensive coverage across diverse tasks and languages; (iii) systematic taxonomy, providing structured labeling and balanced distributions; and (iv) evaluation fidelity, guaranteeing that all instances are executable and verifiable. Together, these principles yield a benchmark that reflects the complexity, diversity, and reproducibility demanded by modern software engineering. Our contributions are threefold: • A comprehensive, execution-grounded benchmark for software engineering. We introduce SWE-Compass, a large-scale benchmark comprising 2,000 curated instances that span eight task types, eight programming scenarios, and ten programming languages. Each instance is drawn from real-world GitHub pull requests and paired with a reproducible execution environment, enabling rigorous and faithful evaluation of model performance in realistic development workflows. • A systematic evaluation framework aligned with real-world developer activities. SWE-Compass establishes a structured taxonomy to assess models across different dimensions such as feature implementation, refactoring, test generation, and deployment. This design enables fine-grained diagnosis of LLM capabilities and provides a principled foundation for comparing agentic coding systems under consistent conditions. • Comprehensive empirical analysis and insights into LLM coding behavior. Experiments with state-of-the-art LLMs and agentic systems reveal persistent gaps across tasks, languages, and scenarios, highlighting the difficulty of scaling beyond bug fixing and emphasizing the need for benchmarks that reflect the full complexity of real-world software engineering. 2 Related Works Coding LLMs and Agents. Code large language models (Code LLMs) ( Chen et al. 2021 ; Zhao et al. 2024 ; Chowdhery et al. 2023 ; Nijkamp et al. 2023 ; Fried et al. 2023 ; Xu et al. 2022 ; Roziere et al. 2023 ; Hui et al. 2024a ; Deng et al. 2025 ; Que et al. 2024 ) — excel at a wide range of programming tasks, including code generation, completion, repair, translation, code comprehension, documentation generation, and cross-language migration, among others. Crucially, modern Code LLMs combine ultra-long context support with robust tool-calling capabilities, enabling them to maintain global awareness across large codebases while actively invoking editors, shells, debuggers, or web browsers ( Liu et al. 2024a ; Liu et al. 2025 ; Wang et al. 2024 ) . This synergy has fueled the rise of agentic coding systems — such as SWE-Agents ( Yang et al. 2024 ) , OpenHands ( Wang et al. 2025 ) , and Claude Code ( Anthropic 2025 ) , QwenCode ( Team 2025 ) , Codex ( OpenAI 2025 ) , Cline ( Cline 2024 ) — that autonomously plan, search, edit, test, and even perform agentic browser use to fetch live API documentation or solutions. As evaluations move toward dynamic, repository-scale workflows, these agent-based systems are showing improved performance over traditional code-generation approaches — particularly in tasks requiring persistent context, environment interaction, and multi-step reasoning ( Liu et al. 2024b ; He et al. 2025 ) . Coding Benchmarks. Single-file code benchmarks — such as HumanEval ( Zheng et al. 2023a ) , MBPP ( Austin et al. 2021 ) , CodeContests ( Li et al. 2022 ) , LiveCodeBench ( Jain et al. 2024 ) and BigCodeBench ( Zhuo et al. 2025 ) — evaluate models on isolated algorithmic problems, abstracting away the structural, contextual, and environmental complexity inherent in real-world software engineering ( Liu et al. 2024c ) ; while SWE-bench ( Jimenez et al. 2024 ) and its variants — including