Code Review Automation using Retrieval Augmented Generation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06

Qianru Meng, Xiao Zhang, Zhaochun Ren, Joost Visser · 2025

Read Code Review Automation using Retrieval Augmented Generation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 by Qianru Meng, Xiao Zhang, Zhaochun Ren, Joost Visser (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05302.txt.

Introduction

Qianru Meng and co-authors ask when added retrieval hurts code review generation, the task of writing reviewer-style notes on a code change. Pure generation can be vague, while pure retrieval fits new cases poorly, so a blend seems natural.

They frame the blend as in-context learning: past reviews are placed in the prompt as guides. Their RARe framework picks those examples in a set way. Results show more retrieved text is not always better; poor picks can pull the review off point. The lesson guides how much past text a review tool should show.

Catalog record

CREATORQianru Meng, Xiao Zhang, Zhaochun Ren, Joost Visser
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05302

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Code Review Automation using Retrieval Augmented Generation DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 Qianru Meng, Xiao Zhang, Zhaochun Ren, Joost Visser Abstract . Code review is essential for maintaining software quality but is labor-intensive. Automated code review generation offers a promising solution to this challenge. Both deep learning-based generative techniques and retrieval-based methods have demonstrated strong performance in this task. However, despite these advancements, there are still some limitations where generated reviews can be either off-point or overly general. To address these issues, we introduce Retrieval-Augmented Reviewer (RARe), which leverages Retrieval-Augmented Generation (RAG) to combine retrieval-based and generative methods, explicitly incorporating external domain knowledge into the code review process. RARe uses a dense retriever to select the most relevant reviews from the codebase, which then enrich the input for a neural generator, utilizing the contextual learning capacity of large language models (LLMs), to produce the final review. RARe outperforms state-of-the-art methods on two benchmark datasets, achieving BLEU-4 scores of 12.32 and 12.96, respectively. Its effectiveness is further validated through a detailed human evaluation and a case study using an interpretability tool, demonstrating its practical utility and reliability. 1. Introduction Code review is an essential component of software quality assurance, requiring substantial human effort. To reduce this effort, automated code review generation has been used to assist human reviewers, for instance by alleviating their workload through the suggestion of review comments, or to replace human reviewers by autonomously generating reviews that can be used immediately by developers. Various approaches have been employed to automate the code review process, evolving from initial rule-based static analysis techniques ( Palvannan and Brown 2023 ; Balachandran 2013 ; Denney and Fischer 2009 ; Klinik et al. 2021 ; Chatley and Jones 2018 ) to more advanced deep learning (DL) methods ( Gupta and Sundaresan 2018 ; Siow et al. 2020 ; Tufano et al. 2022 ; Li et al. 2022a ; Li et al. 2022b ; Lu et al. 2023 ) . These DL-based methods address the code review task as a generation task, employing pre-trained language models ( Tufano et al. 2022 ; Li et al. 2022b ; Li et al. 2022a ) or LLMs ( Lu et al. 2023 ) to produce reviews. Alternatively, code review can also be formulated as an information retrieval task, where a given code change is used as a query to search for the most relevant reviews from a code review corpus. The retrieval-based approach has demonstrated its effectiveness with greater computational efficiency compared to DL-based generation methods ( Hong et al. 2022 ) . Although both retrieval-based and DL-based generative methods have shown effectiveness in code review generation, each still comes with limitations. On one hand, retrieval-based methods, as noted by prior research ( Hong et al. 2022 ) , heavily rely on existing reviews in the training dataset, which restricts their ability to generate new reviews. On the other hand, DL-based generative methods can compensate for this limitation by learning patterns from the given code-review context, thereby generating new reviews. In recent years, LLMs have demonstrated impressive capabilities, particularly when fine-tuned with domain-specific data ( Lu et al. 2023 ) . However, despite these advancements, significant challenges still remain in applying LLMs effectively to code review tasks. LLM-generated reviews often suffer from a lack of domain-specific knowledge, leading to outputs that are either overly general or lacking in relevance ( Li et al. 2022b ) . Moreover, while fine-tuning can improve the specificity of reviews by applying the model to learn patterns, it could results in reviews that are off-point due to the model’s overfitting on the limited training data. To mitigate these issues, integrating retrieval-based methods alongside generative models offers a promising solution. By introducing a wide-ranging external knowledge base, retrieval methods can provide relevant example reviews, thereby enriching the generated content with insights from similar past cases. This hybrid approach has the potential to significantly improve the relevance, accuracy, and contextual understanding of the generated reviews. Motivated by the limitations of both retrieval-based and generative approaches, we introduce the Retrieval-Augmented Reviewer (RARe), a novel method that combines the strengths of both paradigms for automated code review generation. Fundamentally, RARe embodies the concept of retrieval-augmented generation (RAG), which improves the quality and relevance of generative models by incorporating external knowledge from a retrieval codebase. RARe is composed of two main components: the retriever and the generator. The retriever first uses the given code snippet to search for the most relevant reviews within the codebase. These retrieved reviews, combined with the given code snippet and instructions, form a prompt that is then fed into the generator. The generator, utilizing this comprehensive prompt, produces a review tailored to the context. Extensive experiments demonstrate that RARe outperforms state-of-the-art methods in the code review generation task across two benchmark datasets, achieving the best BLEU-4 scores of 12.31 and 12.96, respectively. Compared to its performance without retrieval augmentation, RARe shows remarkable improvements, with a 153% increase in BLEU-4, a 240% rise in ROUGE-L, and a 228% gain in METEOR. In addition, we conduct a comprehensive human evaluations and a case study analysis using an interpretability tool to further confirm the effectiveness of RARe. The detailed investigation results show that by recommending relevant examples and leveraging the contextual understanding capabilities of LLMs, RARe effectively reduce the number of overly general or off-point reviews, providing more accurate and relevant reviews. Our main contributions are as follows: • We are the first to apply the RAG technique to the code review generation task, achieving state-of-the-art results across two datasets compared with the baselines. • We conduct comprehensive comparisons among various retrievers (Normal Dense Retrieval and Dense Passage Retrieval) and generators (Llama 3.1, Mistral, and CodeGemma), providing a detailed analysis of these components. • We are the first to utilize an interpretability tool to provide a novel evaluation insight for code review generation. • All of our code is available in anonymous repository: https://anonymous.4open.science/r/GAR-9EE2 The structure of the paper is as follows. We introduce the related work in Section 2 . Details of the RARe are provided in Section 3 . In Section 4 , we outline three research questions and detail the experiments conducted to investigate them. We then answer these questions in the analysis presented in Section 5 and Section 6 . Implications and limitations are discussed in Section 7 . Potential threats to validity are listed in Section 8 . Finally, Section 9 concludes the paper. 2. Related Work 2.1. Automation of Code Review Generation Static analysis tools are commonly used in early studies ( Palvannan and Brown 2023 ; Balachandran 2013 ; Denney and Fischer 2009 ; Klinik et al. 2021 ) to integrate predefined rules for automating review processes. For instance, Balachandran et al. ( Balachandran 2013 ) developed a Review Bot ( Balachandran 2013 ) to offer code modification suggestions by integrating outputs from multiple static analyzers. Recently, Palvannan et al. ( Palvannan and Brown 2023 ) developed a Suggestion Bot for GitHub, using a Python static analyzer. Moreover, Chatley et al. ( Chatley and Jones 2018 ) also employs data

Read the complete text