ComBench: A Repository-level Real-world Benchmark for Compilation Error Repair DOI: XXXXXXX.XXXXXXX Conference: ACM SIGSOFT International Symposium on Software Testing and Analysis; October 3–9, 2026; Oakland, California, USA ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software development techniques Automated program repair CCS: Computing methodologies Artificial intelligence Natural language processing Language models CCS: Software and its engineering Software development techniques Software testing and debugging CCS: Computing methodologies Artificial intelligence Machine learning Neural networks

Jia Li, Zeyang Zhuang, Zhuangbin Chen, Yuxin Su, Wei Meng, Michael R. Lyu · 2026

Read ComBench: A Repository-level Real-world Benchmark for Compilation Error Repair DOI: XXXXXXX.XXXXXXX Conference: ACM SIGSOFT International Symposium on Software Testing and Analysis; October 3–9, 2026; Oakland, California, USA ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software development techniques Automated program repair CCS: Computing methodologies Artificial intelligence Natural language processing Language models CCS: Software and its engineering Software development techniques Software testing and debugging CCS: Computing methodologies Artificial intelligence Machine learning Neural networks by Jia Li, Zeyang Zhuang, Zhuangbin Chen, Yuxin Su, Wei Meng, Michael R. Lyu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27333.txt.

Introduction

Li and co-authors present ComBench, a repo-level benchmark for repairing compilation errors in real projects. Existing benchmarks use isolated single files and miss repo complexity. ComBench mines real build failures from GitHub CI histories of large open-source projects. Its pipeline extracts ground-truth repair patches from version history. A reproduction step rebuilds the short-lived build setups. Every sample is run-verified before joining the benchmark. GPT-5 fixed syntax in 73 percent of cases but only 41 percent in meaning.

Catalog record

CREATORJia Li, Zeyang Zhuang, Zhuangbin Chen, Yuxin Su, Wei Meng, Michael R. Lyu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67410
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27333

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ComBench: A Repository-level Real-world Benchmark for Compilation Error Repair DOI: XXXXXXX.XXXXXXX Conference: ACM SIGSOFT International Symposium on Software Testing and Analysis; October 3–9, 2026; Oakland, California, USA ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering Software development techniques Automated program repair CCS: Computing methodologies Artificial intelligence Natural language processing Language models CCS: Software and its engineering Software development techniques Software testing and debugging CCS: Computing methodologies Artificial intelligence Machine learning Neural networks Jia Li, Zeyang Zhuang, Zhuangbin Chen, Yuxin Su, Wei Meng, Michael R. Lyu Abstract . Compilation errors pose pervasive and critical challenges in software development, significantly hindering productivity. Therefore, Automated Compilation Error Repair (ACER) techniques are proposed to mitigate these issues. Despite recent advancements in ACER, its real-world performance remains poorly evaluated. This can be largely attributed to the limitations of existing benchmarks, i.e. , decontextualized single-file data, lack of authentic source diversity, and biased local task modeling that ignores crucial repository-level complexities. To bridge this critical gap, we propose ComBench , the first repository-level, reproducible real-world benchmark for C/C++ compilation error repair. ComBench is constructed through a novel, automated framework that systematically mines real-world failures from the GitHub CI histories of large-scale open-source projects. Our framework contributes techniques for the high-precision identification of ground-truth repair patches from complex version histories and a high-fidelity mechanism for reproducing the original, ephemeral build environments. To ensure data quality, all samples in ComBench are execution-verified—guaranteeing reproducible failures and build success with ground-truth patches. Using ComBench , we conduct a comprehensive evaluation of 12 modern LLMs under both direct and agent-based repair settings. Our experiments reveal a significant gap between a model’s ability to achieve syntactic correctness (a 73% success rate for GPT-5) and its ability to ensure semantic correctness (only 41% of its patches are valid). We also find that different models exhibit distinct specializations for different error types. ComBench provides a robust and realistic platform to guide the future development of ACER techniques capable of addressing the complexities of modern software development. 1. Introduction Compilation errors ( 10 ) are a pervasive and disruptive challenge within the core “edit-compile-debug” software development cycle. According to a large-scale study ( 33 ) , C++ developers experience compilation failures in 37.4% of their 10.1 daily builds, while Java developers face a 29.7% failure rate across 6.98 daily builds. Resolving these errors manually is a time-consuming and tedious task, with reported median repair times of 5 mins for C++ and 12 mins for Java, exhibiting an order of magnitude variation across error types. This forces programmers to switch from feature development to troubleshooting, leading to blockages and delays that impact overall productivity. Automated compilation error repair (ACER) techniques ( 19 ; 26 ; 25 ; 41 ; 12 ) are proposed to alleviate these issues. The advent of LLMs has significantly enhanced the effectiveness of code generation, which boosts the development of LLM-based ACER tools ( 12 ) . However, fully assessing their performance is challenging due to a lack of benchmarks reflecting real-world compilation error complexity. Existing ACER benchmarks ( 19 ; 1 ) deviate significantly from the practices of real-world software development across three dimensions: data granularity , source authenticity , and task modeling . First, their decontextualized, single-file paradigm (typically around 20 lines) isolates errors from their project environment, preventing the capture of essential repository-level complexities, such as cross-file references and inter-module dependencies. Second, their narrow data sources have led to a lack of authenticity and diversity. Existing benchmarks are exclusively derived from introductory students’ programming assignments. This results in a relatively homogeneous set of error types ( e.g. , missing delimiters). Thus, they fail to represent the complex and diverse errors in large-scale projects. Lastly, this oversimplified data setting leads to biased task models that frame ACER as a local, sequence-to-sequence problem. It overlooks critical challenges such as global fault localization and comprehensive context collection within large repositories. Consequently, existing benchmarks create a significant disconnect between academic research and industrial practice. This highlights the urgent need for a more comprehensive and representative benchmark. To address these limitations, we propose ComBench , a repository-level real-world benchmark for compilation error repair with reproducible environments and developer-provided ground-truth fixes. ComBench aims to provide a unified platform for more accurately evaluating the performance of ACER methods. It features several distinct advantages. (1) Repository-level Context: ComBench ’s data is directly sourced from Git repository histories of open-source projects. Each error instance includes the complete project-level context, i.e. , all project code, file structure, and dependencies. This enables repair methods to consider cross-file context. (2) Authentic Real-world Errors : Compilation errors are collected from GitHub’s Continuous Integration (CI) system ( 11 ) , GitHub Actions ( 15 ) . Error-inducing code and its corresponding repair patches are derived from actual developer commits, ensuring the authenticity and representativeness of the data. (3) Comprehensive Error Types with Labels : The benchmark encompasses a wide variety of compilation error types, from simple syntax to complex cross-file issues. Each error instance includes detailed compiler diagnostics, complete code context, and the original developer’s repair patch, which is essential for evaluating repair intent and correctness. (4) Reproducible and Continuously Evolving : ComBench provides Docker containers for consistent compilation environments, ensuring reproducibility of reported errors. Furthermore, a reusable data collection pipeline ensures continuous updates, allowing the benchmark to evolve with new errors and development practices from GitHub. The construction of such a benchmark presents three significant challenges. First, the heterogeneity and noise in CI logs make it difficult to precisely extract structured compiler diagnostics. These logs lack a uniform format and are intermingled with massive irrelevant information, such as environment preparation, package installation, test results, and deployment details. Second, a more complex challenge lies in the causal inference of the ground truth repair , specifically locating the exact fix commit and isolating its minimal repair patch. This task is intricate because compilation failures often occur on ephemeral feature branches, complicating the direct traceability of fix commits and necessitating sophisticated search algorithms. Furthermore, identified fix commits frequently bundle the actual error repair with extensive, unrelated modifications like new features or refactoring. Precisely extracting the minimal, targeted patch from these multi-purpose changes is crucial for benchmark quality but inherently difficult. Finally, to ensure the benchmark’s usability for evaluation, the critical challenge is reproduction : accurately recreating the exact historical CI environment in which the failure originally occurred. This is often complicated by two factors, i.e. , inaccessible proprietary runner images, which prevent proper container runtime environment setup, and ambiguous job configurations lacking essential execution parameters, thus hindering a precise reconstruction of the job’s specific runtime environment. Our ComBench framework systematically addresses these challenges through a targeted, multi-faceted approach. To identify compilation errors from noisy and heterogeneous CI logs, we employ a dual-filtering algorithm to prune irrelevant entries and validate error lines, complemented by a bidirectional context collector for detailed error information extraction. For causally inferring the ground truth repair, ComBench utilizes a multi-source search algorithm to locate and rank candidate repair commits across all branches. Following this, compilation dependency analysis precisely isolates the minimal repair patch by filtering for only the modified files directly implicated in resolving the error, thereby ensuring high-quality ground truth. Finally, we

Read the complete text