CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models

Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude · 2025

Read CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models by Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16167.txt.

Introduction

Hong Yi Lin and four co-authors present CodeReviewQA, a test set for how well large language models grasp code review notes. Their study appeared in twenty twenty-five. Review notes are often brief, vague, and loose in tone, so a model has to grasp both code and human intent. Past tests score fixes by word match, which hides where a model fails.

The test set breaks a fix into three steps, each posed as a choice task. Steps cover what kind of change is asked, where it goes, and which fix is right. Nine hundred hand-picked cases span nine code tongues. Tests of many new models showed clear weak spots that fix scores alone had masked.

Catalog record

CREATORHong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE122 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16167

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude Abstract State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks, such as revising source code to address code reviews, hindering their practical use. Code review comments are often implicit, ambiguous, and colloquial, requiring models to grasp both code and human intent. This challenge calls for evaluating large language models’ ability to bridge both technical and conversational contexts. While existing work has employed the automated code refinement (ACR) task to resolve these comments, current evaluation methods fall short, relying on text matching metrics that provide limited insight into model failures and remain susceptible to training data contamination. To address these limitations, we introduce a novel evaluation benchmark, CodeReviewQA that enables us to conduct fine-grained assessment of model capabilities and mitigate data contamination risks. In CodeReviewQA, we decompose the generation task of code refinement into three essential reasoning steps : change type recognition (CTR), change localisation (CL), and solution identification (SI). Each step is reformulated as multiple-choice questions with varied difficulty levels, enabling precise assessment of model capabilities, while mitigating data contamination risks. Our comprehensive evaluation spans 72 recently released large language models on 900 manually curated, high-quality examples across nine programming languages. Our results show that CodeReviewQA is able to expose specific model weaknesses in code review comprehension, disentangled from their generative automated code refinement results. 1 1 1 Data Availability: https://huggingface.co/datasets/Tomo-Melb/CodeReviewQA Figure 1: CodeReviewQA results (%) of the top performing model (per scale class) based on ACR Accuracy. ACR: Automated Code Refinement, CTR: Change Type Recognition, CL: Change Localisation, SI: Solution Identification, E: Easy, H: Hard. 1 Introduction The proficiency of state-of-the-art large language models (LLMs) in code generation has garnered significant attention Zhuo et al. (2025) , demonstrating their ability to follow explicit instructions to author code. However, their competency in real-world software engineering environments remains limited Pornprasit and Tantithamthavorn (2024) , particularly in collaborative tasks involving colloquial and complex forms of communication. A quintessential example is code reviewing, where review comments Yang et al. (2023) ; Efstathiou and Spinellis (2018) represent natural communication between developers with a shared mental model, often resulting in under-specified, ambiguous, and implicit expressions of intent. For example, this comment “For all of the fuzz tests, does it make sense to have versions for ‘len_prefixed’ both ‘true’ and ‘false’?” is a rhetorical question that expresses an intended code change without explicitly detailing the literal change requirement and instructions. As a result, the ability to resolve code review comments not only requires proficiency in understanding and generating code, but also the ability to comprehend the communicative intent behind natural language code reviews in relation to the source code that they address. Therefore, assessing how LLMs resolve code review comments serves as a crucial testbed for their proficiency in understanding and following implicit, conversational instructions in software development. Success in this domain would significantly advance automated software development assistance, potentially reducing developer workload and improving code quality. To evaluate model capability in resolving code review comments, prior work has explored the automated code refinement (ACR) task using both small scale neural language models Tufano et al. (2022) ; Thongtanunam et al. (2022) and LLMs Guo et al. (2024) ; Pornprasit and Tantithamthavorn (2024) , aiming to automatically revise source code based on code review comments. While these efforts have advanced this direction, several critical challenges remain unaddressed. Firstly, current automatic evaluation approaches rely heavily on metrics such as exact match and BLEU Papineni et al. (2002) , which merely capture surface-level token similarities between the generated output and the ground truth, without reflecting intermediate comprehension capabilities. Secondly, as these evaluation benchmarks typically use popular GitHub projects, they risk data contamination from training data in LLMs Sallou et al. (2024) , potentially obfuscating true model capabilities. As a result, there are currently no suitable approaches for assessing the capabilities of state-of-the-art LLMs in automated code refinement. To address these challenges, we introduce a novel evaluation benchmark that enables comprehensive assessment of automated code refinement capabilities. Our benchmark decomposes the original one-step generative task into three underlying reasoning steps: change type recognition (CTR), change localisation (CL), and solution identification (SI). These components represent essential cognitive processes required for understanding code review comments, and subsequently generating the required code revision. By reflecting explicit intermediate reasoning steps, our benchmark provides fine-grained feedback of model failures to support future model development. To mitigate potential data contamination, we formulate each reasoning step as a multiple-choice question answering (MCQA) probe with synthetic answers. This approach transforms the original task into unfamiliar formats with unseen solutions, demanding proficiency in code review comprehension, rather than sequence memorisation of contaminated data Zhu et al. (2024) . Furthermore, we leverage MCQA’s flexibility to introduce distractor variation strategies, enabling assessment of model understanding across different difficulty levels. To avoid the pervasive issue of noisy examples present in past benchmarks Tufano et al. (2024) and ensure high-quality evaluation data, we manually verify and curate 900 valid code refinement examples that cannot be automated by traditional software engineering tools. These examples are sourced from 199 repositories, reflecting nine of the most popular programming languages on GitHub. Finally, we evaluate 72 state-of-the-art code-intelligent LLMs, providing an extensive benchmark to facilitate future research. Our contributions can be summarised as follows: 1) CodeReviewQA Benchmark. The first ACR evaluation approach to include intermediate reasoning probes for detailed feedback on code review comprehension capabilities of LLMs. It is also the first approach to counteract the effects of data contamination in ACR assessment, allowing for the reuse of existing code review data in LLM evaluation. The benchmark consists of four tasks (three MCQA probes and one generative task) in total. 2) Clean Code Review Evaluation Set. The first code review evaluation set that has been completely manually verified for examples that are noisy or do not faithfully represent the task of ACR. To achieve diversity, the examples represent 199 repositories across nine of the most popular programming languages on GitHub. The evaluation set consists of 900 real code reviews in total. 3) Comprehensive Evaluation of LLMs. A large scale evaluation of state-of-the-art open source LLMs that have been trained on both code and natural language. These models span across five different scales, ranging from 1B to 72B parameters. This includes code focused models e.g., CodeLlama-70b-Instruct-hf, general purpose models e.g., Qwen2.5-72B-Instruct, and the latest reasoning models e.g., QwQ-32B. In total, we include 72 LLMs developed by 18 different organisations. 2 Background and Related Work Recently, LLMs have shown promise in various software engineering tasks involving natural language artifacts. However, these artifacts vary significantly in their linguistic nature and structure. Some tasks involve explicit, non-conversational language, such as bug reports Saha et al. (2018) and GitHub issues Jimenez et al. (2024) , which typically contain detailed specifications of defects or feature requests. Other tasks involve static monologues, like commit messages Jiang et al. (2017) , code comments Hu et al. (2018) , and pull request descriptions Liu et al. (2019) , which aim to clearly explain source code or code changes. In contrast, code reviews are unique as

Read the complete text