Benchmarking and Studying the LLM-based Code Review *These authors contributed equally to this work.

Zhengran Zeng 1 *, Ruikai Shi 1 *, Keke Han 1 , Yixin Li 1 , Kaicheng Sun 2, Yidong Wang 1 , Zhuohao Yu 1 , Rui Xie 1 , Wei Ye 1 , Shikun Zhang 1 · 2025

Read Benchmarking and Studying the LLM-based Code Review *These authors contributed equally to this work. by Zhengran Zeng 1 *, Ruikai Shi 1 *, Keke Han 1 , Yixin Li 1 , Kaicheng Sun 2, Yidong Wang 1 , Zhuohao Yu 1 , Rui Xie 1 , Wei Ye 1 , Shikun Zhang 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01494.txt.

Introduction

Automated code review promises to catch bugs before humans do. But existing benchmarks rarely mirror real project work. Zhengran Zeng, with nine co-authors, introduces SWR-Bench. It holds one thousand pull requests from GitHub, each checked by hand. Each case keeps the full project context. Reviews are judged at the pull request level, not on small code fragments.

A scoring method based on a language model checks the reviews. It asks whether issues in a ground truth list are covered. It agrees with human judgment about ninety percent of the time. Tests of leading tools and models showed weak results, with tools better at functional errors. Merging several reviews lifted scores sharply. The F1 score rose by up to about forty-four points.

Catalog record

CREATORZhengran Zeng 1 *, Ruikai Shi 1 *, Keke Han 1 , Yixin Li 1 , Kaicheng Sun 2, Yidong Wang 1 , Zhuohao Yu 1 , Rui Xie 1 , Wei Ye 1 , Shikun Zhang 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01494

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking and Studying the LLM-based Code Review *These authors contributed equally to this work. Zhengran Zeng 1 *, Ruikai Shi 1 *, Keke Han 1 , Yixin Li 1 , Kaicheng Sun 2, Yidong Wang 1 , Zhuohao Yu 1 , Rui Xie 1 , Wei Ye 1 , Shikun Zhang 1 Abstract Automated Code Review (ACR) is crucial for software quality, yet existing benchmarks often fail to reflect real-world complexities, hindering the evaluation of modern Large Language Models (LLMs). Current benchmarks frequently focus on fine-grained code units, lack complete project context, and use inadequate evaluation metrics. To address these limitations, we introduce SWR-Bench , a new benchmark comprising 1000 manually verified Pull Requests (PRs) from GitHub, offering PR-centric review with full project context. SWR-Bench employs an objective LLM-based evaluation method that aligns strongly with human judgment ( ∼ \sim 90% agreement) by verifying if issues from a structured ground truth are covered in generated reviews. Our systematic evaluation of mainstream ACR tools and LLMs on SWR-Bench reveals that current systems underperform, and ACR tools are more adept at detecting functional errors. Subsequently, we propose and validate a simple multi-review aggregation strategy that significantly boosts ACR performance, increasing F1 scores by up to 43.67%. Our contributions include the SWR-Bench benchmark, its objective evaluation method, a comprehensive study of current ACR capabilities, and an effective enhancement approach, offering valuable insights for advancing ACR research. I Introduction Code Review (CR) is an indispensable quality assurance practice in software development [ 1 , 2 , 3 , 4 , 5 ] , aimed at identifying and rectifying potential issues in code changes before integration. While crucial for enhancing code quality, traditional manual code review faces significant hurdles in modern software development. The increasing scale and complexity of projects mean that manual reviews are time-consuming, contributing to development costs and potential delays in feature releases [ 6 , 7 , 8 ] . To mitigate these challenges, Automated Code Review (ACR) has emerged as a vital field and a prominent research area within software engineering [ 9 , 10 , 11 , 12 , 13 ] , driving the creation of numerous tools and datasets. These ACR works strive to improve the efficiency and effectiveness of the code review process by shortening feedback cycles, reducing the manual burden on developers, and improving the consistency and scope of reviews, thus playing a crucial role in streamlining software development workflows and maintaining high standards of code quality. More recently, the explosive advancements in Large Language Models (LLMs) have catalyzed a significant shift in ACR research [ 14 , 15 , 16 , 17 ] , with a growing emphasis on LLM-based methodologies. However, a substantial portion of existing ACR benchmarks [ 10 , 11 , 12 , 13 ] were established primarily for deep learning models that predate the widespread capabilities and sophisticated understanding of modern LLMs. This temporal and methodological gap presents challenges in evaluating the true potential of contemporary LLM-based ACR tools using these benchmarks. Specifically, existing representative benchmarks [ 10 , 12 , 11 , 13 ] , while collecting code review data from platforms like Gerrit and GitHub, have limitations in their evaluation setups. They primarily centers its evaluation units on specific code snippets or single code modifications, rather than the complex interaction process of an entire Pull Request (PR). More critically, these benchmarks typically do not provide the complete project codebase as a necessary context for review, making it difficult for ACR tools to understand the global impact of code changes. Furthermore, the evaluation methodologies for these benchmarks are also problematic. They generally rely on traditional natural language generation metrics such as Exact\text{-}Match [ 10 ] , Bleu scores [ 18 ] and simple LLM-based ratings [ 19 , 20 , 15 ] . However, metrics like Bleu primarily measure textual similarity, while the reliability of LLM ratings is often questioned due to potential inherent biases; both approaches have been shown to be severely inadequate in assessing whether the crucial issues are truly identified in code review comments [ 21 , 22 ] . Furthermore, some studies [ 14 , 9 ] utilize expensive human annotation, but this approach is difficult to scale. Consequently, existing ACR benchmarks commonly exhibit several key limitations: 1) they are often detached from real-world, PR-centric review practices by focusing on overly fine-grained units instead of entire PRs; 2) they frequently lack comprehensive project context, such as the complete codebase, hindering an accurate assessment of change impact; and 3) they tend to employ inadequate evaluation metrics, such as textual similarity, simple LLM-based ratings and expensive human annotation. Consequently, evaluations on current benchmarks often misrepresent the true capabilities and practical value of LLMs in complex, real-world code review. To bridge this critical gap by constructing SWR-Bench ( S oft w are R eview Bench mark), a software review benchmark designed for more realistic evaluation. Specifically, SWR-Bench comprises 1000 manually verified PR instances from GitHub open-source projects. It directly confronts the issues in existing benchmarks through its core design features: • PR-centric Review: Each code review instance is based on a complete PR. This setup is intentionally more challenging, yet more aligned with real-world practices, as we aim to assess the ability of ACR tools to perform end-to-end code review. This includes identifying areas requiring review, pinpointing specific issues, and generating meaningful review comments. • Comprehensive Context: To simulate a realistic review environment, our benchmark provides all relevant code changes (commits) and a snapshot of the entire project repository for each PR, offering comprehensive context that enables tools to accurately assess the global impact of modifications. • Objective LLM-based Evaluation: We employ an automated evaluation where an LLM objectively verifies if issues from a structured ground truth report are covered in the generated review. This method offers a more objective and semantically relevant assessment than subjective “LLM-as-judge” ratings [ 19 , 20 , 15 ] or traditional text-similarity metrics [ 10 , 18 ] . After establishing SWR-Bench , we first validated its evaluation method, confirming strong ( \sim 90%) agreement with human judgment. We then systematically evaluated mainstream ACR tools and leading LLMs on SWR-Bench , uncovering key performance aspects in realistic scenarios: 1) current tools and LLMs do not yet perform sufficiently well; 2) furthermore, we found that LLMs trained with a reasoning-focused approach exhibit better code review capabilities; and finally, 3) existing tools are more adept at detecting functional errors, such as bugs, compared to identifying non-functional issues like outdated documentation. Motivated by these observations, we further introduce a simple enhancement scheme for LLM-based automated code review. This straightforward strategy empowers an LLM to synthesize feedback from multiple review sources by analyzing, filtering valid points, and discarding erroneous suggestions to produce a superior final report. Experiments demonstrate this strategy substantially boosts ACR performance (increasing issue detection F1 score by up to 43.67%), suggesting promising avenues for future research. Therefore, our main contributions in this work include: 1. Benchmark: We introduce SWR-Bench , a benchmark comprising 1,000 real-world PRs with full project context, providing a more realistic and challenging evaluation platform for ACR systems. 2. Evaluation Method: We developed and validated an objective LLM-based evaluation method to

Read the complete text