DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation
Junyi Lu, Xiaojia Li, Zihan Hua, Lei Yu, Shiqi Cheng, Li Yang (🖂), Fengjun Zhang, Chun Zuo · 2024
Read DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation by Junyi Lu, Xiaojia Li, Zihan Hua, Lei Yu, Shiqi Cheng, Li Yang (🖂), Fengjun Zhang, Chun Zuo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18291.txt.
Introduction
Junyi Lu and co-authors argue that grading machine-written code review comments by word overlap grades the wrong thing, in a 2024 paper. Human benchmark comments in open projects vary wildly in quality, and overlap scores barely track whether a comment finds real defects. The team built fresh criteria from past research and developer interviews, then regraded old methods with people and language models as judges.
Fewer than ten percent of benchmark comments proved good enough to train on. Their framework, DeepCRCEval, separated strong comments from weak ones far better, and model judges cut the cost and time by about ninety percent. A few-shot model baseline rounds out the study.
Catalog record
| CREATOR | Junyi Lu, Xiaojia Li, Zihan Hua, Lei Yu, Shiqi Cheng, Li Yang (🖂), Fengjun Zhang, Chun Zuo |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18291
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation Junyi Lu, Xiaojia Li, Zihan Hua, Lei Yu, Shiqi Cheng, Li Yang (🖂), Fengjun Zhang, Chun Zuo Abstract Code review is a vital but demanding aspect of software development, generating significant interest in automating review comments. Traditional evaluation methods for these comments, primarily based on text similarity, face two major challenges: inconsistent reliability of human-authored comments in open-source projects and the weak correlation of text similarity with objectives like enhancing code quality and detecting defects. This study empirically analyzes benchmark comments using a novel set of criteria informed by prior research and developer interviews. We then similarly revisit the evaluation of existing methodologies. Our evaluation framework, DeepCRCEval, integrates human evaluators and Large Language Models (LLMs) for a comprehensive reassessment of current techniques based on the criteria set. Besides, we also introduce an innovative and efficient baseline, LLM-Reviewer, leveraging the few-shot learning capabilities of LLMs for a target-oriented comparison. Our research highlights the limitations of text similarity metrics, finding that less than 10% of benchmark comments are high quality for automation. In contrast, DeepCRCEval effectively distinguishes between high and low-quality comments, proving to be a more reliable evaluation mechanism. Incorporating LLM evaluators into DeepCRCEval significantly boosts efficiency, reducing time and cost by 88.78% and 90.32%, respectively. Furthermore, LLM-Reviewer demonstrates significant potential of focusing task real targets in comment generation. 1 Introduction Since Fagan’s pioneering work on software inspections in 1976 [ 7 ] , code review practices have evolved significantly. Early software inspections were formal and resource-intensive, which limited their widespread adoption in the industry. In contrast, Modern Code Review (MCR) has become a vital and streamlined process in both Open-Source Software (OSS) [ 27 , 26 , 25 ] and industrial applications [ 28 , 29 ] . MCR typically involves developers submitting code changes for review, which reviewers then assess to provide feedback on potential issues or improvements. This process may involve mechanisms such as pull requests, changesets, or code patches, depending on the tools and platforms used. A pivotal aspect of MCR is the generation of insightful, constructive comments, essential for guiding developers and pinpointing problems, which is further elaborated in Section 0.A . Despite its effectiveness, MCR remains resource-intensive [ 34 ] . This challenge has spurred the exploration of automating code review comments to alleviate labor demands [ 11 ] . Initial attempts focused on retrieval-based methods using existing comments as references [ 8 , 30 ] . With the progression of deep learning, the emphasis has shifted to generative approaches. A variety of Code Review Comment Generators (CRCGs) have emerged, notably initiated by Tufano et al. with a T5 transformer architecture [ 32 ] , later augmented with code-technical language pre-training [ 31 ] . Following this trend, models like CodeReviewer [ 16 ] and AUGER [ 14 ] have further advanced the field, integrating specific pre-training for code review and utilizing review tags to enhance accuracy. Parallel developments include CommentFinder [ 10 ] , offering an efficient retrieval-based solution, and CCT5 [ 17 ] , which underscores the importance of considering code changes in comment generation. Later, with the emergence of LLMs, Lu et al. [ 20 ] investigate the factors influencing the code review process for both traditional pre-trained language models and LLMs, while Llama-Reviewer [ 21 ] marks the first attempt at training code review tasks on LLMs in a parameter-efficient approach. Nevertheless, these models typically employ text similarity metrics such as BLEU [ 22 ] and ROUGE [ 18 ] for evaluation. We question the suitability of text similarity as the primary metric for assessing code review comment automation. The dependability of human-written comments in OSS, commonly used as benchmarks, is often subject to scrutiny [ 1 , 3 , 2 , 12 , 23 , 9 , 33 ] . These comments can be arbitrary and inconsistent, at times offering little more than basic queries or directives like “Why do we need this?” or “Remove this.” Unlike traditional text-to-text tasks such as summarization or translation, which seek semantic equivalence, code review comments aim to aid in defect detection and code refinement, requiring deeper insight [ 1 ] . Hence, text similarity, as an indirect measurement, falls short of capturing the essence. Analysis of Benchmark Comments. The reliability of text similarity metrics for evaluating review comments hinges on the accuracy of the reference text. Yet, the quality and validity of benchmark comments in major datasets, such as the CodeReviewer (CRer) [ 16 ] and Tufano datasets [ 31 ] , often remain ambiguous. To address these ambiguities and provide a more nuanced understanding, we conducted an empirical study analyzing these datasets from four dimensions: 1) Quality: Assessing comments against established quality standards; 2) Category: Evaluating the effectiveness of comments in identifying defects and suggesting improvements, or other potential roles; 3) Tone: Analyzing whether comments clearly state issues or are merely interrogative; 4) Context: Determining if the comments are sufficiently supported by the associated code. For Quality evaluation, we developed criteria based on existing literature about developers’ views on code review quality [ 12 ] , supplemented by our semi-structured interviews and card sorting exercises using affinity diagrams. DeepCRCEval Stemming from our analysis of benchmark comments, we developed DeepCRCEval, an innovative evaluation framework incorporating both human and LLM evaluators. This framework utilized the criteria of high quality comments identified in the former analysis, and use both scoring and ranking to compare performances. It is geared towards identifying the intrinsic merits of review comments, moving away from the indirect approach of text similarity. LLM-Reviewer To validate the effectiveness of direct metrics in line with code review objectives, we propose LLM-Reviewer, a lightweight, training-free baseline tool. Traditional methods, reliant on text similarity metrics, may not capture the true essence of code review. In contrast, by harnessing the potential of Large Language Models (LLMs), LLM-Reviewer employs few-shot learning with meticulously crafted prompts. Unlike previous methods, LLM-Reviewer directly addresses the actual aims of code review and controlled the comments’ quality with criterion guidance in prompt, more closely mirroring developers’ concept of effective review commentary. This baseline also serves as a benchmark to gauge the effectiveness of evaluation frameworks in differentiating high and low-quality review comments. Revisiting the Evaluation of CRCGs Utilizing DeepCRCEval as our evaluation framework and LLM-Reviewer as our baseline, we reassessed the performance of current state-of-the-art (SOTA) CRCGs. Given the training-free nature of LLM-Reviewer, it was anticipated that other tuned methods would perform at least comparably. However, our findings, derived from both human and LLM evaluators, reveal a considerable shortfall in the performance of existing SOTA CRCGs compared to this benchmark. Our analysis of 1,000 typical code snippets containing defects or code smells demonstrated LLM-Reviewer’s proficiency in consistently pinpointing issues, providing well-rounded comments encompassing problem identification, detailed explanation, and possible solutions. In contrast, SOTA CRCGs often produced nonspecific and ambiguous comments, falling short of the review’s goals. Contributions Our significant contributions include: 1. Unveiling the biases in evaluating state-of-the-art (SOTA) CRCGs due to reliance on inappropriate metrics. 2. Developing DeepCRCEval, a versatile evaluation framework employing either human or LLM evaluators, which concentrates on the core essence of comments rather than indirect text similarity measures. 3.