AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context
Lei Zhang, Yongda Yu, Minghui Yu, Xinxin Guo, Zhengqi Zhuang, Guoping Rong, Dong Shao, Haifeng Shen, Hongyu Kuang, Zhengfeng Li, Boge Wang, Guoan Zhang, Bangyu Xiang, Xiaobin Xu · 2026
Read AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context by Lei Zhang, Yongda Yu, Minghui Yu, Xinxin Guo, Zhengqi Zhuang, Guoping Rong, Dong Shao, Haifeng Shen, Hongyu Kuang, Zhengfeng Li, Boge Wang, Guoan Zhang, Bangyu Xiang, Xiaobin Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19494.txt.
Introduction
Lei Zhang and fourteen colleagues build a test for a new kind of software tool. Large language models can now review code and comment on changes, much like a human reviewer. Judging whether such reviews are any good takes a benchmark, a fixed set of tasks with known answers. Many current tests look at single files or small snippets, which misses how real projects fit together.
Their benchmark, AACR-Bench, judges automatic code review with the whole repository in view. A reviewer that sees the full project can catch faults that only make sense across files. The tasks are built to reflect that wider context. Good scores on the bench should mean a tool is ready for real code bases. The work gives labs and firms a shared yardstick for this fast-growing use of language models.
Catalog record
| CREATOR | Lei Zhang, Yongda Yu, Minghui Yu, Xinxin Guo, Zhengqi Zhuang, Guoping Rong, Dong Shao, Haifeng Shen, Hongyu Kuang, Zhengfeng Li, Boge Wang, Guoan Zhang, Bangyu Xiang, Xiaobin Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19494
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context Lei Zhang, Yongda Yu, Minghui Yu, Xinxin Guo, Zhengqi Zhuang, Guoping Rong, Dong Shao, Haifeng Shen, Hongyu Kuang, Zhengfeng Li, Boge Wang, Guoan Zhang, Bangyu Xiang, Xiaobin Xu Abstract High-quality evaluation benchmarks are pivotal for deploying Large Language Models (LLMs) in a certain applicaiton field. However, existing benchmarks on Automated Code Review (ACR) suffer from two critical limitations: first, the reliance on noisy, incomplete ground truth derived from raw Pull Request (PR) comments, which constrains the scope of issue detection; second, the lack of multi-language support in repository-level contexts, which restricts the generalizability of evaluation results. To address these challenges, we introduce AACR-Bench , a comprehensive benchmark that provides full cross-file context across multiple programming languages. Unlike traditional datasets, AACR-Bench employs an “AI-assisted, Human expert-verified” annotation pipeline to uncover latent defects often overlooked in original PRs, resulting in a 285% increase in issue coverage. Extensive evaluations of mainstream LLMs on AACR-Bench reveal that previous assessments may have either misjudged or only partially captured model capabilities due to data limitations. Our work establishes a more rigorous standard for ACR evaluation and offers new insights on LLM based ACR, i.e., the granularity/level of context and the choice of retrieval methods significantly impact ACR performance, and this influence varies depending on the LLM, programming language, and the LLM usage paradigm e.g., whether an Agent architecture is employed. The code, data, and other artifacts of our evaluation set are available at Github 1 1 1 https://github.com/alibaba/aacr-bench . 1 Introduction Nowadays, empowered by Large Language Models (LLMs), automated code review(ACR) technologies have been widely studied and adopted ( Hou et al. 2024 ; Zhang et al. 2023 ) . Leveraging the impressive comprehension capabilities of LLMs, researchers have explored various methods for the end-to-end generation of review comments, for example, the pre-training model schemes proposed by Tufano et al., the fine-tuning strategies that balance cost and performance ( Lu et al. 2023 ) , and the optimization mechanisms based on reinforcement learning ( Le et al. 2022 ) . Recently, Agent-based code review systems have also begun to emerge ( Guo et al. 2025b ; Ren et al. 2025 ) . The flourishing research and adoption in ACR has intensified the need for credible benchmarks capable of comprehensively evaluating their performance. However, current evaluation frameworks still suffer from critical limitations, which can be summarized into two main issues. • Incomplete Issue Annotation. In general, existing benchmarks directly use raw review comments from real-world historical PRs as the Ground Truth, resulting in inherent limitations regarding issue coverage within the dataset ( Liu et al. 2025 ; Khoshnoud et al. 2022 ) . This, in turn, makes it impossible to authentically and faithfully characterize a model’s ability to uncover potential issues in a real-world code review scenarios. • Restricted Context Scope. Many code defects are inherently cross-file, requiring review systems to access repository-level context for accurate detection. Although some existing benchmarks do offer cross-file context awareness ( Zeng et al. 2025 ; Guo et al. 2025a ) , they are predominantly limited to a single programming language, such as Python. This language-specific focus not only constrains the generalizability of evaluation findings but may also introduce structural biases tied to the linguistic characteristics of that language. Consequently, such benchmarks fail to adequately represent the multilingual reality of contemporary software development. These deficiencies collectively undermine the credibility and scope of current ACR evaluations, highlighting the necessity for a more comprehensive and crediable benchmark. To address the aforementioned challenges, we propose AACR-Bench —a multilingual ACR benchmark that supports Repository-level Context-awareness. First, to address the limited issue annotation in existing datasets, we constructed a high-quality hybrid corpus. Beyond collecting 391 real-world review comments from GitHub PRs, we incorporated extensive manual issue annotation: 80 senior software engineers (each with 2+ years of industry experience) meticulously reviewed 2,145 comments generated by two ACR systems across six LLMs. This annotation strategy substantially improves issue coverage and enables AACR-Bench to more accurately and comprehensively assess a model’s capability to uncover latent issues compared to conventional datasets. Secondly, to address the issues of context and language diversity, AACR-Bench provides complete repository-level dependency information while broadly covering 10 mainstream programming languages. The full AACR-Bench dataset can be retrieved in the supplementary materials of this paper and will be open-sourced later on. Our main contributions are summarized as follows: • We combine multi-model generation and large-scale human annotation to improve issue coverage so as to establish a better groundtruth dataset. • We propose AACR-Bench , the first multilingual, repository-level context-aware benchmark for LLM enabled ACR tasks. • We conduct a comprehensive empirical evaluation of the mainstream LLMs with AACR-Bench and reveal new insights towards mainstream LLM’s ACR capabilities. 2 Related Work We position this study by examining two key dimensions: the current state of large language model (LLM)-enabled ACR, and the evolving landscape of corresponding specialized benchmark methodologies. 2.1 Automatic Code Review Based on LLM Recent advances in LLMs have driven growing research in automated code review (ACR), with methods spanning several categories. Early work includes T5-based ACR ( Tufano et al. 2022 ) and CodeReviewer ( Li et al. 2022 ) , pretrained on large diff–comment pairs. To improve efficiency, Llama-Reviewer ( Lu et al. 2023 ) employed Parameter-Efficient Fine-Tuning, while Sun et al. introduced a “Data Flywheel” for continuous model evolution via iterative feedback ( Sun et al. 2025 ; Yu et al. 2024a ) . Recent efforts align models with human preferences: Yu et al. ( Yu et al. 2024b ) used Kahneman–Tversky Optimization to enhance review utility, and Kapadnis et al. ( Kapadnis et al. 2025 ) combined static analysis with Direct Preference Optimization to better detect latent defects. To address limited context, Zhang et al. ( Zhang et al. 2025 ) applied Retrieval-Augmented Generation to incorporate project-level semantics. Further extending reasoning depth, agent-based frameworks employ multi-agent collaboration ( Ren et al. 2025 ; Li et al. 2025 ; Sharanarthi & Polineni 2025 ) and dialectical interactions such as debate ( Tang et al. 2024 ) to produce more comprehensive and objective reviews. 2.2 Code Review Benchmarks Benchmarks are essential for defining the capability boundaries of LLMs and guiding algorithmic evolution ( Jimenez et al. 2023 ) . In ACR, the rapid emergence of diverse methods underscores the need for a standardized benchmark to systematically assess performance and guide future optimization. Currently, ACR benchmarking remains nascent, with efforts falling into two categories: accompanying datasets from individual studies and dedicated evaluation datasets. The most widely used accompanying dataset is CodeReviewer ( Li et al. 2022 ) , employed in many recent works ( Lu et al. 2023 ; Yu et al. 2024a ; Yu et al. 2024b ; Kapadnis et al. 2025 ; Ren et al. 2025 ; Li et al. 2025 ) . However, it provides only diff-level code fragments and lacks repository-level context, which limits its realism relative to production environments. A few studies ( Sun et al. 2025 ; Sharanarthi & Polineni 2025 ) forgo static datasets altogether, relying solely on live user feedback. Dedicated benchmarks include SWR-Bench ( Zeng et al. 2025 ) (derived from SWE-Bench’s 12 Python projects) and CodeFuse-CR-Bench (