CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude Abstract State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks, such as revising source code to address code reviews, hindering their practical use. Code review comments are often implicit, ambiguous, and colloquial, requiring models to grasp both code and human intent. This challenge calls for evaluating large language models’ ability to bridge both technical and conversational contexts. While existing work has employed the automated code refinement (ACR) task to resolve these comments, current evaluation methods fall short, relying on text matching metrics that provide limited insight into model failures and remain susceptible to training data contamination. To address these limitations, we introduce a novel evaluation benchmark, CodeReviewQA that enables us to conduct fine-grained assessment of model capabilities and mitigate data contamination risks. In CodeReviewQA, we decompose the generation task of code refinement into three essential reasoning steps : change type recognition (CTR), change localisation (CL), and solution identification (SI). Each step is reformulated as multiple-choice questions with varied difficulty levels, enabling precise assessment of model capabilities, while mitigating data contamination risks. Our comprehensive evaluation spans 72 recently released large language models on 900 manually curated, high-quality examples across nine programming languages. Our results show that CodeReviewQA is able to expose specific model weaknesses in code review comprehension, disentangled from their generative automated code refinement results. 1 1 1 Data Availability: https://huggingface.co/datasets/Tomo-Melb/CodeReviewQA Figure 1: CodeReviewQA results (%) of the top performing model (per scale class) based on ACR Accuracy. ACR: Automated Code Refinement, CTR: Change Type Recognition, CL: Change Localisation, SI: Solution Identification, E: Easy, H: Hard. 1 Introduction The proficiency of state-of-the-art large language models (LLMs) in code generation has garnered significant attention Zhuo et al. (2025) , demonstrating their ability to follow explicit instructions to author code. However, their competency in real-world software engineering environments remains limited Pornprasit and Tantithamthavorn (2024) , particularly in collaborative tasks involving colloquial and complex forms of communication. A quintessential example is code reviewing, where review comments Yang et al. (2023) ; Efstathiou and Spinellis (2018) represent natural communication between developers with a shared mental model, often resulting in under-specified, ambiguous, and implicit expressions of intent. For example, this comment “For all of the fuzz tests, does it make sense to have versions for ‘len_prefixed’ both ‘true’ and ‘false’?” is a rhetorical question that expresses an intended code change without explicitly detailing the literal change requirement and instructions. As a result, the ability to resolve code review comments not only requires proficiency in understanding and generating code, but also the ability to comprehend the communicative intent behind natural language code reviews in relation to the source code that they address. Therefore, assessing how LLMs resolve code review comments serves as a crucial testbed for their proficiency in understanding and following implicit, conversational instructions in software development. Success in this domain would significantly advance automated software development assistance, potentially reducing developer workload and improving code quality. To evaluate model capability in resolving code review comments, prior work has explored the automated code refinement (ACR) task using both small scale neural language models Tufano et al. (2022) ; Thongtanunam et al. (2022) and LLMs Guo et al. (2024) ; Pornprasit and Tantithamthavorn (2024) , aiming to automatically revise source code based on code review comments. While these efforts have advanced this direction, several critical challenges remain unaddressed. Firstly, current automatic evaluation approaches rely heavily on metrics such as exact match and BLEU Papineni et al. (2002) , which merely capture surface-level token similarities between the generated output and the ground truth, without reflecting intermediate comprehension capabilities. Secondly, as these evaluation benchmarks typically use popular GitHub projects, they risk data contamination from training data in LLMs Sallou et al. (2024) , potentially obfuscating true model capabilities. As a result, there are currently no suitable approaches for assessing the capabilities of state-of-the-art LLMs in automated code refinement. To address these challenges, we introduce a novel evaluation benchmark that enables comprehensive assessment of automated code refinement capabilities. Our benchmark decomposes the original one-step generative task into three underlying reasoning steps: change type recognition (CTR), change localisation (CL), and solution identification (SI). These components represent essential cognitive processes required for understanding code review comments, and subsequently generating the required code revision. By reflecting explicit intermediate reasoning steps, our benchmark provides fine-grained feedback of model failures to support future model development. To mitigate potential data contamination, we formulate each reasoning step as a multiple-choice question answering (MCQA) probe with synthetic answers. This approach transforms the original task into unfamiliar formats with unseen solutions, demanding proficiency in code review comprehension, rather than sequence memorisation of contaminated data Zhu et al. (2024) . Furthermore, we leverage MCQA’s flexibility to introduce distractor variation strategies, enabling assessment of model understanding across different difficulty levels. To avoid the pervasive issue of noisy examples present in past benchmarks Tufano et al. (2024) and ensure high-quality evaluation data, we manually verify and curate 900 valid code refinement examples that cannot be automated by traditional software engineering tools. These examples are sourced from 199 repositories, reflecting nine of the most popular programming languages on GitHub. Finally, we evaluate 72 state-of-the-art code-intelligent LLMs, providing an extensive benchmark to facilitate future research. Our contributions can be summarised as follows: 1) CodeReviewQA Benchmark. The first ACR evaluation approach to include intermediate reasoning probes for detailed feedback on code review comprehension capabilities of LLMs. It is also the first approach to counteract the effects of data contamination in ACR assessment, allowing for the reuse of existing code review data in LLM evaluation. The benchmark consists of four tasks (three MCQA probes and one generative task) in total. 2) Clean Code Review Evaluation Set. The first code review evaluation set that has been completely manually verified for examples that are noisy or do not faithfully represent the task of ACR. To achieve diversity, the examples represent 199 repositories across nine of the most popular programming languages on GitHub. The evaluation set consists of 900 real code reviews in total. 3) Comprehensive Evaluation of LLMs. A large scale evaluation of state-of-the-art open source LLMs that have been trained on both code and natural language. These models span across five different scales, ranging from 1B to 72B parameters. This includes code focused models e.g., CodeLlama-70b-Instruct-hf, general purpose models e.g., Qwen2.5-72B-Instruct, and the latest reasoning models e.g., QwQ-32B. In total, we include 72 LLMs developed by 18 different organisations. 2 Background and Related Work Recently, LLMs have shown promise in various software engineering tasks involving natural language artifacts. However, these artifacts vary significantly in their linguistic nature and structure. Some tasks involve explicit, non-conversational language, such as bug reports Saha et al. (2018) and GitHub issues Jimenez et al. (2024) , which typically contain detailed specifications of defects or feature requests. Other tasks involve static monologues, like commit messages Jiang et al. (2017) , code comments Hu et al. (2018) , and pull request descriptions Liu et al. (2019) , which aim to clearly explain source code or code changes. In contrast, code reviews are unique as they represent routine conversations in highly collaborative scenarios. As such, they are informal, free-flowing, and can lean on the interlocutor’s shared technical knowledge, without being overly specific Yang et al. (2023) . Thus, interpreting code reviews requires a deep understanding of conversational language in a highly technical context, posing challenges for computational models. The automated code refinement task involves revising source code to address code review comments, which requires an understanding of both technical implications and the reviewer’s unstated expectations. Such nuanced communication makes automated code refinement an ideal testbed for evaluating LLMs’ ability to bridge both technical and conversational understanding in software development. The automated code refinement task was typically framed as a sequence-to-sequence neural machine translation problem, where models “translate” pre-review code submissions into post-review code revisions that reflect the intent of the accompanying code review comments. Formally, this problem requires the following estimation: P(H_{post}|H_{pre},R_{nl}) (1) where H_{pre} denotes the submitted pre-review code hunk, R_{nl} denotes the natural language code review comment, and H_{post} denotes the expected post-review revision of that code hunk. The “hunk” refers to the code snippet within the file, where the code review comment was inlined. See Figure 2 for a concrete example. While prior work has applied various neural language models, such as recurrent neural networks Tufano et al. (2019) and transformers Tufano et al. (2022) ; Thongtanunam et al. (2022) , the task remains a challenging problem, even for recent LLMs such as GPT-4 Guo et al. (2024) ; Lu et al. (2023) ; Tufano et al. (2024) . Indeed, prior work has highlighted several limitations in the evaluation Guo et al. (2024) . Traditional evaluation approaches have relied heavily on text matching metrics such as exact match and BLEU Tufano et al. (2024) ; Guo et al. (2024) , which are either too strict or fail to provide meaningful feedback. The emergence of LLMs has introduced additional challenges, as they are trained on extensive code repositories, creating significant risks of training data contamination. While some researchers have attempted to address this by collecting code reviews that outpace training cutoff dates Guo et al. (2024) , such approaches lack long-term sustainability as real code reviews take time to naturally occur. Furthermore, existing benchmarks have been constructed automatically through large-scale mining using general heuristics, as a result, significant proportions of noise have been reported Tufano et al. (2024) ; Liu et al. (2025) , undermining the reliability of past results. Table 1 summarises the limitations of existing ACR evaluation benchmarks, underscoring the need for a new evaluation approach and dataset to reliably assess the capabilities of state-of-the-art LLMs. Our proposed CodeReviewQA benchmark focuses on addressing these gaps. Code Review Benchmark Size #Lang Metric DC MV VD Tufano 2021 Tufano et al. (2021) 1.7k 1 Text Match T5CR Tufano et al. (2022) 16.8k 1 Text Match CodeReviewer Li et al. (2022) 13.1k 9 Text Match CodeReview-New Guo et al. (2024) 14.6k 16 Text Match CodeReviewQA (Ours) 900 9 Text Match & Probe DC: Addresses Data Contamination, MV: Manual Verification, VD: Varied Difficulty Table 1: Benchmarks for ACR. Pre-Review Code Submission ( H_{pre} ): ⬇ 1 from hypothesistooling . projects . hypothesispython import PYTHON_SRC 2 from hypothesistooling . scripts import pip_tool , tool_path 3 4 PYTHON_VERSIONS = [ f "3.{v}" for v in range (7, 11)] 5 6 def test_mypy_passes_on_hypothesis (): Code Review ( R_{nl} ) : I think I’d prefer to write these out as literals, unless we can pull them out of the autoupdated CI config? Just thinking about how they’ll stay up to date. I think we can also test against 3.11? —————————————————————————————— What type of change is the code review asking for? A. Only add new lines of code B. Only delete existing lines of code C. Modify the code \checkmark —————————————————————————————— Which line numbers is the code review asking to modify code? A. line number 1    B. line number 2 C. line number 4 \checkmark D. line number 6 —————————————————————————————— Which code revision is the code review asking for? A. ⬇ 4 - PYTHON_VERSIONS = [ f "3.{v}" for v in range (7, 11)] 4 + PYTHON_VERSIONS >= [ "3.7" , "3.8" , "3.9" , "3.10" , "3.11" ] B. ⬇ 4 - PYTHON_VERSIONS = [ f "3.{v}" for v in range (7, 11)] 4 + PYTHON_VERSIONS <= [ "3.7" , "3.8" , "3.9" , "3.10" , "3.11" ] C. ⬇ 4 - PYTHON_VERSIONS = [ f "3.{v}" for v in range (7, 11)] 4 + PYTHON_VERSIONS != [ "3.7" , "3.8" , "3.9" , "3.10" , "3.11" ] D. \checkmark ⬇ 4 - PYTHON_VERSIONS = [ f "3.{v}" for v in range (7, 11)] 4 + PYTHON_VERSIONS = [ "3.7" , "3.8" , "3.9" , "3.10" , "3.11" ] —————————————————————————————— Post-Review Code Revision ( H_{post} ) : ⬇ 1 from hypothesistooling . projects . hypothesispython import PYTHON_SRC 2 from hypothesistooling . scripts import pip_tool , tool_path 3 4 PYTHON_VERSIONS = [ "3.7" , "3.8" , "3.9" , "3.10" , "3.11" ] 5 6 def test_mypy_passes_on_hypothesis (): Figure 2: The ACR task with intermediate reasoning steps presented as MCQA Probes. 3 CodeReviewQA: MCQA Probes Effective ACR relies heavily on the ability to comprehend R_{nl} under the context of H_{pre} . Rather than focusing this task as a sequence-to-sequence translation problem like the prior works, we argue that the model must be able to: 1) reason about the type of change R_{nl} is requesting and 2) identify the relevant lines of code in H_{pre} that is the subject of the change; and 3) formulate the required code changes from a wide action space of potential code edits that can be performed on H_{pre} , before generating the code revision H_{post} . The inability to perform the final code generation step may be caused by any failure point amongst this multi-step reasoning process. Additionally, any failure within the intermediary reasoning steps might be propagated from a failure in a prior reasoning step, which obfuscates the specific incompetencies of the model. To assess the proficiency of LLMs in ACR, we propose three MCQA probes, each representing a specific intermediate reasoning step. See Figure 2 for an example of the MCQA structure. Below, we describe the construction of each MCQA probe. 3.1 Change Type Recognition (CTR) This is a closed set intent classification task that probes the model’s ability to infer the intended type of code change. Specifically, given H_{pre} , the model must infer which general type of code change is being requested by R_{nl} . Formally, this problem requires the following estimation: P(C_{type^{+}}|H_{pre},R_{nl}) (2) where C_{type^{+}}\hskip-3.00003pt\in\hskip-3.00003pt\{add,delete,modify\} denotes the correct code change type. There are three general types. Firstly, add requests involve only adding new lines of code. Secondly, delete requests involve only deleting existing lines of code. Lastly, modify requests involve altering the existing code by both deleting existing segments and adding new ones. The C_{type^{-}} distractors are the remaining two incorrect code change types. This preliminary understanding serves as crucial conditional information that refines the problem space, providing the correct C_{type} context to subsequently locate where the code changes need to occur and identify what needs to be implemented. 3.2 Change Localisation (CL) This is a coreference resolution task that probes the model’s ability to locate where the intended code change is to occur. Specifically, given R_{nl} , the model must locate the precise lines of code within H_{pre} where the intended C_{type} code change should be applied. Formally, this problem requires the following estimation: P(C_{loc^{+}}|H_{pre},R_{nl},C_{type}) (3) where C_{loc^{+}} denotes the exact set of line numbers that is the target of the intended code change. When C_{type}\hskip-3.00003pt\in\hskip-3.00003pt\{delete,modify\} , these are the exact lines of code that need to be deleted or modified. When C_{type}=\{add\} , these are the lines of code above where the new code needs to be added. The C_{loc^{-}} distractors are different sets of lines sampled from H_{pre} . We ensure |C_{loc^{-}}|=|C_{loc^{+}}| , such that set sizes do not reveal additional information. As shown in Figure 2 , natural code review comments often do not directly specify the exact location of the intended code change, rather this is implicitly conveyed based on a shared understanding between the reviewer and code author. Thus, the model must possess the ability to conduct anaphora resolution across modalities, between anaphors in R_{nl} and antecedents in H_{pre} . Inferring the incorrect C_{loc} , would subsequently hinder the model’s ability to identify the H_{post} that accurately reflects the intended code change. 3.3 Solution Identification (SI) This task probes the model’s ability to both conduct open intent extraction from R_{nl} and identify the H_{post} that accurately reflects that intent. Given R_{nl} , the model must identify the correct H_{post} that reflects the intended C_{type} change on C_{loc} in H_{pre} . The intuition behind this task design is that if a model is able to generate a correct H_{post+} revision, it should at least be able to identify that exact H_{post+} solution amongst a solution space with incorrect H_{post^{-}} alternatives. Formally, this problem requires the following estimation: P(H_{post+}|H_{pre},R_{nl},C_{type},C_{loc}) (4) where H_{post+} denotes the diff of the ground truth post-review code revision. We only include cases where C_{type}\in\{add,modify\} , as \{delete\} cases merely delete C_{loc} located in the previous task. 3.4 Variation of Distractor Difficulty The MCQA format allows flexibility in varying the difficulties of the distractors (i.e., the incorrect answer options). This not only allows us to stress test the models’ level of understanding, but also enables the ability to evolve the benchmark against performance saturation. We specify the process of generating easy and hard distractors for Change Localisation and Solution Identification , as these tasks allow for variation in the solutions. Change Localisation Distractors . We vary the difficulty based on the degree of overlap between the sets of the provided C_{loc} options. For the easy distractors, we sample C_{loc^{-}} distractors from H_{pre} , such that the Jaccard Similarity between all answer options are as low as possible. This ensures that all answers are easy to distinguish from each other and the ground truth is more obvious to locate. For the hard distractors, we sample C_{loc^{-}} distractors, such that the Jaccard Similarity between all answer options are as high as possible. This ensures that all answers are hard to discern from the ground truth, and requires the model to locate every exact line of the intended code change. Solution Identification Distractors . To create distractor options H_{post-} , we generate modified versions of H_{post+} by perturbing code elements in C_{loc} , ensuring the intended code change is no longer correctly implemented. To create plausible but incorrect distractors that imitate possible mistakes that the models would make, we use a surrogate LLM 2 2 2 We use a competitive surrogate LLM of code ( Codestral-22B-v0.1 ) with a temperature of 3.5 to encourage diversity. to 1) identify the code element with the highest average surprisal in H_{post+} , 2) mask it, and 3) retroactively fill the masks with diverse candidates. We keep candidates that are not equivalent to H_{post+} as valid H_{post-} distractor candidates. All generated distractors are manually verified for semantic in-equivalence to the ground truth. The algorithm for constructing H_{post-} distractors is illustrated in Algorithm 1 in the Appendix. We vary the difficulty based on the degree of semantic similarity between the H_{post^{-}} distractors and the H_{post^{+}} ground truth. For the easy distractors, we retain the H_{post^{-}} distractors which yield the lowest cosine similarity against H_{post^{+}} in the embedding space of the surrogate model. This ensures that each H_{post^{-}} is substantially different to H_{post^{+}} , such that it is easy to discern. For the hard distractors, we retain the H_{post^{-}} distractors which yield the highest cosine similarity against H_{post^{+}} . This ensures that each H_{post^{-}} is only marginally different from H_{post^{+}} , such that it is hard to discern. See Figure 7 in the Appendix for examples of variation in difficulty. 4 Dataset Preparation Data Source. We built our benchmark based on the most recently published automated code refinement dataset Guo et al. (2024) . This multilingual dataset was constructed from code reviews that occurred after January 1, 2022. To ensure that we have a sizable amount of clean data for each of the programming languages in our benchmark, we only include the nine most popular programming languages on GitHub i.e. C, C++, C#, Go, Java, Javascript, PHP, Python, Ruby. These 9,367 examples were mined from 259 repositories, filtered from a list of the most starred GitHub projects. Data Sampling. To ensure diversity and quality in our benchmark, we conducted stratified sampling Baltes and Ralph (2022) across all nine programming languages in the dataset, and discarded any examples that were noisy or unfaithfully represented the task of code refinement. For each of the nine languages, we sampled until there were 100 clean examples each, resulting in 900 total examples in our benchmark. Within each language partition, we also conducted stratified sampling across projects to maintain diversity. This mitigated bias towards the code reviews of any specific project, the nature of which are influenced by their particular software development tools Paschali et al. (2017) , processes Viggiato et al. (2019) and issues Linares-Vásquez et al. (2014) . Data Curation. We discard examples that were noisy or unfaithfully represented the task of code refinement. The noisy examples refer to code review comments that are unclear, ignored, not asking for a code change, or linking to wrong code hunks. See Appendix A for a detailed explanation of these noise types. These kind of review comments were reported as critical quality issues with existing code review datasets by prior work Tufano et al. (2024) . Unfaithful examples refer to the scenarios that do not faithfully represent the ACR task i.e., reviews directly including the intended H_{post} revision implementation, reviews regarding simple code formatting, reviews that are not self-contained Tufano et al. (2024) ; Lin et al. (2024) . Instead, examples in the benchmark should represent meaningful quality improving code reviews that are beyond the capacity of traditional rule-based software engineering tools. See Appendix B for a detailed explanation of these unfaithful examples. To discard noisy and unfaithful examples, we first applied heuristic filters as detailed in Appendix C , before manual verification. This resulted in 3,761 out of 9,367 examples being discarded from the source dataset. The manual discarding was conducted by the first and third authors, who are currently pursuing computer science PhDs focused on AI for software engineering. In the end, both annotators independently annotated 3k examples, and resolved all conflicts together across 46 rounds. The \mu and \sigma of the Cohen’s Kappa were 0.89 and 0.11, respectively. For C, JavaScript and Ruby, less than 100 clean examples could be obtained from the source dataset, thus, the remaining examples were sampled from code reviews conducted in 2021 Li et al. (2022) . The overall retention rate was 13%, highlighting critical quality issues in the source datasets, necessitating manual curation of ACR benchmarks for accurate and reliable evaluation. The final benchmark includes 199 of the original 259 GitHub repositories. Table 5 in the Appendix shows benchmark statistics. 5 Experimental Setup 5.1 MCQA Setup To support the MCQA probes in CodeReviewQA , we detail our prompt design, answer extraction, and evaluation framework for invariance testing. Prompt. We use multiple-choice prompting that takes an input containing three components: task definition, question, and options. The task definition specifies the broad purpose (e.g., “tests code review comprehension”). The question section presents the code review scenario within a template that includes programming language markers, H_{pre} , and R_{nl} . Finally, the options section lists multiple choice answers labelled alphabetically (A, B, C, D), with explicit instructions to respond with only the letter symbol. This prompt structure is used across all tasks, varying only the specific task parameters and answer options. See Figure 3 in the Appendix for all prompt templates used. Answer Extraction. We use multiple choice prompting with a max output length of one, where the symbol token \in\{A,B,C,D\} with the highest log probability is considered as the selected answer. This style of prompting avoids the conflation of likelihood of sequence and likelihood of answer, eliminates the need for normalisation and allows for direct comparison between answers Robinson and Wingate (2023) . Our implementation uses the vLLM inference framework Kwon et al. (2023) with guided decoding targeting the option symbols. Invariant Test and Evaluation. To reduce the likelihood of random correct guesses, for each question, we exposed the models to every order combination of the answer options. This resulted in N! runs per question, where N is the number of answer options provided. Thus, the likelihood of guessing the correct answer for all combinations of a question is merely (\frac{1}{N})^{N!} . With this, we assess the models’ invariability in selecting the correct answer, regardless of the position of that answer. To be counted as correctly answering that question, the models must select the correct answer for all N! runs, which is a more reliable indicator of the models’ understanding Wang et al. (2025) . 5.2 Model Selection Criteria We list the criteria that determines whether a LLM is appropriate for this benchmark. MCQA Proficiency. The LLM must have achieved state-of-the-art results in MCQA style benchmarks e.g., MMLU Hendrycks et al. (2021) . This accounts for format as a confounding factor. MCSB Proficiency. The LLM must demonstrate proficiency in multiple choice symbol binding (MCSB; Robinson and Wingate (2023) ). This ensures that the answer extraction method is not a confounding factor. We report the Proportion of Plurarity Agreement (PPA), which measures the degree of order invariance in selecting the symbol of the plurarity answer. Formally, PPA is calculated as the average of \frac{k}{N!} over a dataset, where k is the number of times the plurarity answer’s symbol yielded the highest log probability for a given question and N! is the aforementioned number of order combinations for N answer options. MCSB proficiency is demonstrated when a PPA significantly higher than the random baseline of \frac{1}{N} is achieved. Coding Proficiency. In addition to understanding the natural language in R_{nl} , the model must also be able to understand the code in H_{pre} and H_{post} . Therefore, the LLM must have demonstrated proficiency in coding related benchmarks e.g., HumanEval Chen et al. (2021) and MBPP Austin et al. (2021) . In total, we selected 72 state-of-the-art open source LLMs, that have satisfied the three criteria. The included models are considered state-of-the-art as of March, 2025. See Table 7 in Appendix F for descriptions of all 72 models. The models are grouped into five scales based on their model parameters: \leq 3B, \leq 9B, \leq 16B, \leq 34B, and \leq 72B. We selected models with \leq 72B parameters, as it is the largest non-quantised model class that we can support locally to extract answer probabilities. 6 Results Overview To compare how models of different scales perform on CodeReviewQA , we conducted experiments using all 72 models. Due to space limitations, complete results on ACR and all three MCQA probes are presented in Appendix F . In Table 2 , we compare the MCQA probe results of the top-2 models in terms of ACR from each scale class. ACR vs MCQA Probes. Table 2 (column ACR) shows that Llama-3.1-70B-Instruct achieved the highest exact match rate of 50.3%. As expected, we find that larger LLMs tend to achieve higher exact match rates on average in ACR. However, the performance improvements appear to be diminishing when scaling past the \leq 16B class of models. Specifically, we find that Llama-3.1-70B-Instruct only achieves a 3.7% increase over Qwen2.5-Coder-14B-Instruct, despite wielding a 5-fold increase in parameter count. Interestingly, we find that ACR performances are not always congruent with how models rank in MCQA probes. For example, Qwen2.5-72B-Instruct achieves an exact match rate approximately 2% lower than the top performer, but performs vastly better (> 10% increase) in terms of invariant accuracy in both CTR and SI. This type of inconsistency also occurs throughout the smaller scales. For example, Qwen2.5-Coder-14B-Instruct, gemma-2-27b-it, and Mistral-Small-Instruct-2409 achieve comparable exact match rates ( \leq 1% difference) in ACR. However, their performances on CL and SI probes are substantially different. These additional insights highlight the benefits of intermediate feedback, beyond simple text matching evaluations on final code revision outputs. Below, we discuss results on the three MCQA probing tasks. Model ACR CTR CL {}_{\text{E}} CL {}_{\text{H}} SI {}_{\text{E}} SI {}_{\text{H}} Qwen2.5-Coder-3B-Instruct 30.3 77.7 1.8 1.8 12.2 8.0 Llama-3.2-3B-Instruct 25.9 78.8 0.8 0.4 9.9 7.6 Qwen2.5-Coder-7B-Instruct 41.0 78.6 13.8 10.7 67.6 55.2 gemma-2-9b-it 39.0 74.1 59.2 52.0 58.8 49.6 Qwen2.5-Coder-14B-Instruct 46.6 73.9 46.7 37.3 65.5 56.2 phi-4 37.1 76.6 50.9 44.8 84.4 77.5 gemma-2-27b-it 46.4 74.0 70.1 58.7 76.2 65.7 Mistral-Small-Instruct-2409 45.6 76.7 38.4 31.6 63.8 60.1 Llama-3.1-70B-Instruct 50.3 68.4 74.7 69.0 84.2 76.7 Qwen2.5-72B-Instruct 48.7 79.8 64.2 58.3 97.1 90.9 ACR: Automated Code Refinement, CTR: Change Type Recognition CL: Change Localisation, SI: Solution Identification, E: Easy, H: Hard Table 2: MCQA Probe Accuracy (%) of top-2 models (per scale class) in terms of Exact Match Rate on ACR. CTR Results. Table 2 (column CTR) shows that most of the \leq 3 B models were already competent in this task, with Llama-3.2-3B-Instruct achieving 78.8% invariant accuracy. Despite the promising results of small models, this ability plateaus as we scale model size. In fact, Qwen2.5-72B-Instruct only achieved a 1% improvement, despite having 24 times the amount of parameters. Interestingly, Llama-3.1-70B-Instruct even regresses in this capability, achieving the worst invariant accuracy across the top performing models. As knowledge of the general change type sets the entire context for deciding where and how to revise the code, the inability to improve on CTR presents a fundamental limit to downstream ACR performance. CL Results. Table 2 (columns CL {}_{\text{E}} and CL {}_{\text{H}} ) show that change localisation tends to be the most difficult comprehension task in the benchmark, where proficiency depends more heavily on scale. Table 10 in Appendix F shows the full results for CL {}_{\text{E}} , where most of the \leq 3 B models achieved invariant accuracies between 0%-3%. The only exceptions were Qwen2.5-3B-Instruct and Phi-3-mini-128k-instruct, which could achieve 39.3% and 34.1%, respectively. The discrepancy between ACR and CL results across the under parameterised models suggest a reliance on surface-level pattern recognition for ACR, that is not grounded in a robust understanding of the code review. In contrast, we find that many models from the \leq 34 B and \leq 72 B classes could achieve invariant accuracies of more than 70% for CL {}_{\text{E}} and more than 60% for CL {}_{\text{H}} . Nevertheless, proficiency in identifying exactly where the code revisions need to occur remains the largest challenge. SI Results. Table 2 (columns SI {}_{\text{E}} and SI {}_{\text{H}} ) show that proficiency in solution identification also strengthens with size. Despite this, Table 12 in Appendix F shows a few anomalies that outperform their scale class average by a large margin. For example, Phi-3-mini-128k-instruct can achieve an invariant accuracy of 58.2% for SI {}_{\text{E}} , whilst the majority of \leq 3B models achieve less than 13%. In contrast, many models from the \leq 72B class can achieve more than 80% for SI {}_{\text{E}} and more than 70% for SI {}_{\text{H}} . Most notably, Qwen2.5-72B-Instruct, could achieve a near perfect score of 97.1% for SI {}_{\text{E}} , and 90.9% for SI {}_{\text{H}} , despite previously achieving underwhelming results for change localisation. These results suggest that many larger models can identify the correct code revision at least under a drastically reduced solution space with semantically ambiguous distractors. 7 Insights from Probing ACR Failures What does MCQA probes reveal about model failures in ACR? We use our MCQA probes to investigate ACR failure cases of the top performing models in Table 2 . We define a failure as the case when the model does not generate a H_{post} revision that exactly matches the ground truth. For each failure case, we examine whether the model also fails on the MCQA probes. Through this analysis, we aim to identify specific weaknesses in the models’ code review comprehension capabilities that may contribute to their ACR failures. The results are presented in Table 3 , including the percentage of failure on each of the MCQA probes, 3 3 3 For CL and SI, a failure in either difficulties is counted. and the overall failure on at least one probe ( \geq 1 Probe). We find that all five models failed at least one MCQA probe for 76.5%-99.8% of the failure cases, indicating that the model struggled at the code review comprehension stage. CTR failures are seldom associated with the ACR failures, as this capability only accounts for 22.2%-37.4% of cases. In contrast, CL failures are often associated with ACR failures, particularly for the three smaller models, where 74.0%-99.4% of the cases overlap with ACR failures. SI failures account for 95.8% of cases for Qwen2.5-Coder-3B-Instruct. Thus, for the smallest model, most non-exact matches are associated with failures in both CL and SI. For both Qwen2.5-Coder-7B-Instruct and Qwen2.5-Coder-14B-Instruct, CL failure is the major factor associated with ACR failure. For the larger models, gemma-2-27b-it and Llama-3.1-70B-Instruct, the probe failures are more evenly distributed, varying based on the model’s specific incompetencies. On the other hand, we also analysed the successful cases where the model achieves an exact match. Intuitively, if a model can achieve exact match on an example, it should be able to fully comprehend the code review, thus correctly answering all probes. However, the models could not accurately answer all probes for 49.0%-99.6% of their successful cases. This trend shows a strict inverse relationship with model size, where Llama-3.1-70B-Instruct and Qwen2.5-Coder-3B-Instruct could not consistently complete all of the probes for 49.0% and 99.6% of their successful cases, respectively. The ability to complete these ACR examples verbatim without a prerequisite understanding of the intent behind the code reviews allude to prior exposure and rote memorisation of these examples. %Fail Model \geq 1 Probe CTR CL SI Qwen2.5-Coder-3B-Instruct 99.8 23.4 99.4 95.8 Qwen2.5-Coder-7B-Instruct 96.6 22.2 93.0 55.0 Qwen2.5-Coder-14B-Instruct 87.3 29.3 74.0 56.0 gemma-2-27b-it 75.5 25.7 53.7 41.4 Llama-3.1-70B-Instruct 76.5 37.4 46.3 32.7 CTR: Change Type Recognition, CL: Change Localisation, \geq 1 Probe: Failed at least one probe, SI: Solution Identification Table 3: MCQA Failure (%) in Non-Exact Match cases of the top performing model (per scale class) in ACR. 8 Evaluating Data Contamination To what extent is CodeReviewQA resistant to data contamination? We utilise two canonical metrics for measuring data contamination, perplexity Jelinek et al. (1977) and n-gram accuracy Xu et al. (2024) . Perplexity is an information-theoretic metric, which quantifies the uncertainty of a language model in a token sequence Jelinek et al. (1977) , which can be formulated as: \text{PPL}(\textbf{X})=\exp(-\frac{1}{t}\sum^{t}_{t=0}\log p_{\theta}(x_{i}|x_{ FutureTerms = new List < string > 6 { 7 @ "esea" 8 }; 9 } 10 } 11 \ No newline at end of file [/CSharp] [CODE REVIEW] There are two issues here. One, this term is not a "future" term, so we need a better name. Second, this value is incorrect in PT, it should be "esse", "essa", "este", "esta". [/CODE REVIEW] ### Possible answers: A. only add new lines of code B. only delete existing lines of code C. modify the code ### Answer with the letter symbol only. Answer: C Question: Given this {lang} code snippet, what type of change is the code review asking for? [{lang}] {code_snippet} [/{lang}] [CODE REVIEW] {code_review} [/CODE REVIEW] ### Possible answers: A. only add new lines of code B. only delete existing lines of code C. modify the code ### Answer with the letter symbol only. Answer: Figure 4: Few-Shot CTR Prompt Template. Change Localisation (CL) {change_type} = add new lines of code under/delete code/modify code ### The following are multiple choice questions (with answers) that tests code review comprehension. Question: Given this Java code snippet, which line numbers is the code review asking to modify code? [Java] ⬇ 1 private void launchZoomActivityAfterPermissionCheck ( final View view ) { 2 final Context ctx = view . getContext (); 3 final Intent zoomableIntent = new Intent ( ctx , ZoomableActivity . class ); 4 zoomableIntent . setData ( Uri . parse ( media . getImageUrl ())); 5 zoomableIntent . putExtra ( "Origin" , "MediaDetail" ); 6 ctx . startActivity ( 7 zoomableIntent 8 ); [/Java] [CODE REVIEW] Could you please rename to "MediaDetails"? [/CODE REVIEW] ### Possible answers: A. 3 B. 4 C. 5 D. 6 ### Answer with the letter symbol only. Answer: C Question: Given this CSharp code snippet, which line numbers is the code review asking to modify code? [CSharp] ⬇ 1 public static class DateTimeDefinitions 2 public const string QuarterTypeRegex = @ "(trimestral(mente)?)$" ; 3 public const string SemiAnnualTypeRegex = @ "(semestral(mente)?)$" ; 4 public const string YearTypeRegex = @ "(anual(mente)?)$" ; 5 public static readonly IList < string > FutureTerms = new List < string > 6 { 7 @ "esea" 8 }; 9 } 10 } 11 \ No newline at end of file [/CSharp] [CODE REVIEW] There are two issues here. One, this term is not a "future" term, so we need a better name. Second, this value is incorrect in PT, it should be "esse", "essa", "este", "esta". [/CODE REVIEW] ### Possible answers: A. 5, 7 B. 2, 3 C. 1, 4 D. 8, 11 ### Answer with the letter symbol only. Answer: A Question: Given this {lang} code snippet, which line numbers is the code review asking to {change_type}? [{lang}] {code_snippet} [/{lang}] [CODE REVIEW] {code_review} [/CODE REVIEW] ### Possible answers: A. {option_a} B. {option_b} C. {option_c} D. {option_d} ### Answer with the letter symbol only. Answer: Figure 5: Few-Shot CL Prompt Template. Solution Identification (SI) ### The following are multiple choice questions (with answers) that tests code review comprehension. Question: Given this Java code snippet, which code revision is the code review asking for? [Java] ⬇ 1 private void launchZoomActivityAfterPermissionCheck ( final View view ) { 2 final Context ctx = view . getContext (); 3 final Intent zoomableIntent = new Intent ( ctx , ZoomableActivity . class ); 4 zoomableIntent . setData ( Uri . parse ( media . getImageUrl ())); 5 zoomableIntent . putExtra ( "Origin" , "MediaDetail" ); 6 ctx . startActivity ( 7 zoomableIntent 8 ); [/Java] [CODE REVIEW] Could you please rename to "MediaDetails"? [/CODE REVIEW] ### Possible answers: A. ⬇ 5 - zoomableIntent . putExtra ( "Origin" , "MediaDetail" ); { 5 + zoomableIntent . postExtra ( "Origin" , "MediaDetails" ); B. ⬇ 5 - zoomableIntent . putExtra ( "Origin" , "MediaDetail" ); { 5 + zoomableIntent . getExtra ( "Origin" , "MediaDetails" ); C. ⬇ 5 - zoomableIntent . putExtra ( "Origin" , "MediaDetail" ); { 5 + zoomableIntent . putExtra ( "Origin" , "MediaDetails" ); D. ⬇ 5 - zoomableIntent . putExtra ( "Origin" , "MediaDetail" ); { 5 + zoomableIntent . renameExtra ( "Origin" , "MediaDetails" ); ### Answer with the letter symbol only. Answer: C Question: Given this CSharp code snippet, which code revision is the code review asking for? [CSharp] ⬇ 1 public static class DateTimeDefinitions 2 public const string QuarterTypeRegex = @ "(trimestral(mente)?)$" ; 3 public const string SemiAnnualTypeRegex = @ "(semestral(mente)?)$" ; 4 public const string YearTypeRegex = @ "(anual(mente)?)$" ; 5 public static readonly IList < string > FutureTerms = new List < string > 6 { 7 @ "esea" 8 }; 9 } 10 } 11 \ No newline at end of file [/CSharp] [CODE REVIEW] There are two issues here. One, this term is not a "future" term, so we need a better name. Second, this value is incorrect in PT, it should be "esse", "essa", "este", "esta". [/CODE REVIEW] ### Possible answers: A. ⬇ 5 - public static readonly IList < string > FutureTerms = new List < string > 5 + public static readonly IList < string > ThisTerms = new List < string > 7 - @ "esea" 7 + @ "esse" 8 + @ "essa" 9 + @ "este" 10 + @ "esta" B. ⬇ 5 - public static readonly IList < string > FutureTerms = new List < string > 5 + public static readonly IList < string > NotTerms = new List < string > 7 - @ "esea" 7 + @ "esse" 8 + @ "essa" 9 + @ "este" 10 + @ "esta" C. ⬇ 5 - public static readonly IList < string > FutureTerms = new List < string > 5 + public static readonly IList < string > futureTerms = new List < string > 7 - @ "esea" 7 + @ "esse" 8 + @ "essa" 9 + @ "este" 10 + @ "esta" D. ⬇ 5 - public static readonly IList < string > FutureTerms = new List < string > 5 + public static readonly IList < string > betterTerms = new List < string > 7 - @ "esea" 7 + @ "esse" 8 + @ "essa" 9 + @ "este" 10 + @ "esta" ### Answer with the letter symbol only. Answer: A Question: Given this {lang} code snippet, which code revision is the code review asking for? [{lang}] {code_snippet} [/{lang}] [CODE REVIEW] {code_review} [/CODE REVIEW] ### Possible answers: A. {option_a} B. {option_b} C. {option_c} D. {option_d} ### Answer with the letter symbol only. Answer: Figure 6: Few-Shot SI Prompt Template. Pre-Review Code Submission ⬇ 1 def main ( args : argparse . Namespace ): 2 ) 3 host_environment = host_environments . pop () 4 5 module_dir_paths = sort_and_dedup_paths ([ 6 iree_artifacts . get_module_dir_path ( config . module_generation_config ) 7 for config in run_configs 8 ]) 9 10 output_map [ device_name ] = { 11 "host_environment" : dataclasses . asdict ( host_environment ), Code Review: Huh, would be nice if the path was just naturally serializable —————————————————————————————————————————————————— Which line numbers is the code review asking to modify code? A. \checkmark line numbers 5, 6, 8 Change Localisation (Easy) B. line numbers 1, 2, 3   C. line numbers 4, 7, 9   D. line numbers 9, 10, 11 Change Localisation (Hard) B. line numbers 1, 5, 6   C. line numbers 3, 5, 6   D. line numbers 3, 5, 8 —————————————————————————————————————————————————— Which code revision is the code review asking for? A. \checkmark ⬇ 5 - module_dir_paths = sort_and_dedup_paths ([ 5 + module_dir_paths = sorted ( set ( 6 - iree_artifacts . get_module_dir_path ( config . module_generation_config ) 6 + str ( iree_artifacts . get_module_dir_path ( config . module_generation_config )) 8 - ]) 8 + )) Solution Identification (Easy) B. ⬇ 5 - module_dir_paths = sort_and_dedup_paths ([ 5 + module_dir_paths = sorted ( set ( 6 - iree_artifacts . get_module_dir_path ( config . module_generation_config ) 6 + str ( struct_lucule . get_module_dir_path ( config . module_generation_config )) 8 - ]) 8 + )) C. ⬇ 5 - module_dir_paths = sort_and_dedup_paths ([ 5 + module_dir_paths = sorted ( set ( 6 - iree_artifacts . get_module_dir_path ( config . module_generation_config ) 6 + str ( assertTrue_localhost . get_module_dir_path ( config . module_generation_config )) 8 - ]) 8 + )) D. ⬇ 5 - module_dir_paths = sort_and_dedup_paths ([ 5 + module_dir_paths = sorted ( set ( 6 - iree_artifacts . get_module_dir_path ( config . module_generation_config ) 6 + str ( indexChat_retry . get_module_dir_path ( config . module_generation_config )) 8 - ]) 8 + )) Solution Identification (Hard) B. ⬇ 5 - module_dir_paths = sort_and_dedup_paths ([ 5 + module_dir_paths = sorted ( set ( 6 - iree_artifacts . get_module_dir_path ( config . module_generation_config ) 6 + str ( View_DEF . get_module_dir_path ( config . module_generation_config )) 8 - ]) 8 + )) C. ⬇ 5 - module_dir_paths = sort_and_dedup_paths ([ 5 + module_dir_paths = sorted ( set ( 6 - iree_artifacts . get_module_dir_path ( config . module_generation_config ) 6 + str ( develop_weight . get_module_dir_path ( config . module_generation_config )) 8 - ]) 8 + )) D. ⬇ 5 - module_dir_paths = sort_and_dedup_paths ([ 5 + module_dir_paths = sorted ( set ( 6 - iree_artifacts . get_module_dir_path ( config . module_generation_config ) 6 + str ( register_access . get_module_dir_path ( config . module_generation_config )) 8 - ]) 8 + )) Figure 7: Examples of Variation in Difficulty. Comment Length = 8 FYI, this will spam console when running ‘aaa‘. —————————————————————————————————————————————————— Comment Length = 18 By the format string it looks like parameters shall be reversed. Type shall be 1st and exception 2nd —————————————————————————————————————————————————— Comment Length = 23 I’m wondering if it’s useful to show the message from the exception in this debug message, at least in the case of IOException. Figure 8: Examples of Different Comment Lengths. Code Edit Distance = 9 ⬇ 1 OnConflictAction onconflict_action = ts_chunk_dispatch_get_on_conflict_action ( dispatch ); 2 ResultRelInfo * resrelinfo , * relinfo ; 3 - bool has_compressed_chunk = ( chunk -> fd . compressed_chunk_id != 0); 3 + bool is_compressed = ( chunk -> fd . compressed_chunk_id != 0) 4 /* permissions NOT checked here; were checked at hypertable level */ 5 if ( check_enable_rls ( chunk -> table_id , InvalidOid , false ) == RLS_ENABLED ) Code Review: Should ‘is_compressed‘ now be given by the chunk’s compression status flag? Otherwise it looks like we have different ways of determining compression status. —————————————————————————————————————————————————— Code Edit Distance = 25 ⬇ 1 void Server_Card :: resetState () 2 setPT ( QString ()); 3 setAnnotation ( QString ()); 4 setDoesntUntap ( false ); 5 - setFaceDown ( false ); 6 } 7 8 QString Server_Card :: setAttribute ( CardAttribute attribute , const QString & avalue , bool allCards ) Code Review: this causes a major bug: cards have their state reset when moved between the battlefield and the deck, their facedown state is then checked afterwards to determine what event to show to other players. with this change moving a facedown card to your deck (unknown to unknown) will tell all your opponents (but not you) what card it was. —————————————————————————————————————————————————— Code Edit Distance = 67 ⬇ 1 public unsafe LazyStringValue GetDocumentId ( LazyStringValue key ) 2 if ( index == -1) 3 return null ; 4 5 - _tmpLazyStringInstance = _context . GetLazyString ( key . Buffer , index ); 5 + return _context . GetLazyString ( key . Buffer , index ); 6 - return _tmpLazyStringInstance ; 7 } 8 9 // TODO unify if possible with AllowedPathsValidator Code Review: Why do you store that in the temporary variable? Figure 9: Examples of Different Code Edit Distances. Code Element Ratio = 0 Should this really be a compile time error? The fact that is can be imported multiple times does not mean that it will be. —————————————————————————————————————————————————— Code Element Ratio = 0.13 For all of the fuzz tests, does it make sense to have versions for ‘len_prefixed‘ both ‘true‘ and ‘false‘ ? —————————————————————————————————————————————————— Code Element Ratio = 0.37 I think you’re missing a ‘flb_free(seq_index_str);‘ there. Other than that, would you mind change that comparison to ‘if (tmp_key == NULL) {‘ instead? I’d really appreciate it. Figure 10: Examples of Different Code Element Ratios. Specification Ratio = 0.67 ⬇ 1 void hpx_thread_buffer :: resize ( const std :: size_t num_threads , 2 } 3 4 void * hpx_thread_buffer :: get ( std :: size_t thread_num ) const noexcept { 5 - KOKKOS_ASSERT ( thread_num < m_num_threads ); 5 + KOKKOS_EXPECTS ( thread_num < m_num_threads ); 6 if ( m_data == nullptr ) { 7 return nullptr ; 8 } 9 return & m_data [ thread_num * m_size_per_thread ]; 10 } 11 12 void * hpx_thread_buffer :: get_extra_space () const noexcept { 13 - KOKKOS_ASSERT ( m_extra_space > 0); 13 + KOKKOS_EXPECTS ( m_extra_space > 0); 14 if ( m_data == nullptr ) { 15 return nullptr ; 16 } Code Review: This is fine but just pointing out there is also a ‘KOKKOS_EXPECTS‘ that was meant for checking preconditions —————————————————————————————————————————————————— Specification Ratio = 37.60 ⬇ 1 private String addNashornJavaScriptEngineIfNecessary ( String cp ) { 2 } 3 4 private boolean requiresNashornJavaScriptEngine () { 5 - String version = System . getProperty ( "java.specification.version" ); 5 + return getJavaVersion () >= 15; // Nashorn was removed in Java 15 6 - if ( version . startsWith ( "1." )) { 7 - version = version . substring (2); 8 - } 9 - return Integer . parseInt ( version ) >= 15; // Nashorn was removed in Java 15 10 } 11 12 } Code Review: You can use ‘getJavaVersion()‘ here. Figure 11: Examples of Different Specification Ratios. Appendix E Implementation Details The experiments were carried out on a single node consisting of 64 cores (Intel(R) Xeon(R) Platinum 8462Y+ @ 2.80GHz), 928GBs of RAM and four GPUs (NVIDIA H100-80GB SXM5). For efficient inference, we utilised the default vLLM implementation with Hugging Face models. 4 4 4 https://huggingface.co/docs/hub/en/models-the-hub For all tasks, we set the temperature to zero for greedy decoding. Appendix F Experimental Results on 72 models This section shows our complete results. The details of all evaluated LLMs are presented in Table 7 , and their achieved exact match rate on ACR in Table 8 . For the MCQA probes, we report both their proportion of plurarity agreement and invariant accuracy. If the achieved proportion is far higher than the stated random probability on any probe, the model is considered to have multiple choice symbol binding proficiency. Table 9 shows the full results on CTR. Table 10 and Table 11 shows full results on the easy and hard variations of CL, respectively. Table 12 and Table 13 shows full results on the easy and hard variations of SI, respectively. Appendix G Advanced Prompting Since our main results were based on zero-shot prompting, we further investigated the effects of advanced prompting i.e., few-shot, chain-of-thought, on MCQA probe performance. Specifically, we were interested if advanced prompting could increase the performance of the top performing model i.e., Llama-3.1-70B-Instruct. For few-shot prompting, we considered both one-shot and two-shot scenarios with real examples held out from the benchmark. For the few-shot prompt templates, see Figure 4 for CTR, Figure 5 for CL and Figure 6 for SI. For chain-of-thought, we only considered zero-shot due to the lack of real reasoning traces to use as examples. This was invoked by appending " Let’s think step by step " to the end of the zero-shot prompts and instructing the model to finalise their answer with " The final answer is ". Prompt Strategy CTR CL {}_{\text{E}} CL {}_{\text{H}} SI {}_{\text{E}} SI {}_{\text{H}} Zero-Shot 68.4 74.7 69.0 84.2 76.7 One-Shot 74.1 68.6 61.9 59.9 55.6 Two-Shot 76.9 75.8 68.7 57.3 54.6 Chain-of-Thought 68.6 65.8 60.1 66.6 60.0 CTR: Change Type Recognition, E: Easy, H: Hard CL: Change Localisation, SI: Solution Identification Table 6: MCQA Probe Accuracy (%) of Llama-3.1-70B-Instruct w/ Advanced Prompting. The results are shown in Table 6 . Overall, we find that advanced prompting does not outperform the zero-shot strategy. For CTR, we find that zero-shot performed similar to chain-of-thought, where both accuracies were approximately 68%. In contrast, the few-shot methods could achieve 74.1-76.9%, where two-shot prompting was the top performing prompt strategy. In terms of CL easy, we find that one-shot and chain-of-thought were the worst performers, achieving 68.6% and 65.8%, respectively. In contrast, zero-shot and two-shot performed similarly, achieving 74.7% and 75.8%, respectively. For the remaining MCQA probes, all advanced prompting techniques underperformed against the zero-shot strategy, especially for SI. Whilst chain-of-thought consistently degraded the model’s performance on the MCQA probes, few-shot prompting showed promise. Specifically, two-shot prompting was the top performing strategy in two of the five probes, without optimising for the best examples. We did not optimise this selection due to the lack of clean code review examples. As a result, we only used two held out examples collected during the manual curation process. Future researchers can investigate the effects of advanced example selection e.g., retrieval-augmented generation, for few-shot prompting strategies. Algorithm 1 Create H_{post^{-}} Distractors for Solution Identification Input: Surrogate LLM {\color[rgb]{0.75,0,0.25}f_{\theta}} , Temperature {\color[rgb]{0,0.5,0.5}k} , No. of Distractors {\color[rgb]{1,0.5,0}N} , Difficulty {\color[rgb]{0.75,0.5,0.25}\phi} Output: Set of H_{post^{-}} Distractors {\color[rgb]{0.5,0,0.5}D} // Identify code elements in the changed lines of the post-review code revision Lines\leftarrow\text{GetChangedLines}(H_{post^{+}}) ; AST\leftarrow\text{GetAbstractSyntaxTree}(H_{post^{+}}) ; Nodes\leftarrow\text{GetLeafNodes}(AST,Lines) ; // Calculate average token surprisal for each identified code element S_{token},S_{node},C_{distractor},{\color[rgb]{0.5,0,0.5}D}\leftarrow\emptyset,\emptyset,\emptyset,\emptyset ; for c_{t}\in H_{post^{+}} do \bm{h}_{t-1}\leftarrow{\color[rgb]{0.75,0,0.25}f_{\theta}(}H_{pre},R_{nl},c_{