CRScore++: Reinforcement Learning with Verifiable Tool and AI Feedback for Code Review
Manav Nitin Kapadnis Atharva Naik Carolyn Rosé · 2025
Read CRScore++: Reinforcement Learning with Verifiable Tool and AI Feedback for Code Review by Manav Nitin Kapadnis Atharva Naik Carolyn Rosé (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00296.txt.
Introduction
Manav Kapadnis, Atharva Naik, and Carolyn Rose train code review models with mixed feedback in this 2025 paper. Review comments are free text, which makes reward signals hard. Verifiable feedback is reliable but narrow. AI feedback is flexible but subjective. CRScore plus plus combines both into one training reward.
The base metric joins language model judgment with linters and code smell detectors. Those signals become rewards for reinforcement learning. A weaker student model improves through supervised steps plus critique from a stronger teacher. The gains carried to programming languages outside training. Better automated review could cut maintainer load on open source projects. The framework shows how to reward messy, human facing outputs.
Catalog record
| CREATOR | Manav Nitin Kapadnis Atharva Naik Carolyn Rosé |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00296
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CRScore++: Reinforcement Learning with Verifiable Tool and AI Feedback for Code Review Manav Nitin Kapadnis Atharva Naik Carolyn Rosé Abstract Reinforcement learning (RL) to improve code review comment generation requires handling unstructured outputs, making reinforcement learning (RL) feedback challenging. The two main RL approaches, namely RL with Verifiable Feedback (RLVR) and RL with AI Feedback (RLAIF), offer trade-offs: RLVR provides reliable feedback for structured tasks like code generation, while RLAIF works for unstructured outputs but is subjective. We bridge this gap with CRScore++, an RL framework that leverages both LLM-based subjective feedback and verifiable signals for training. Extending CRScore, a code review evaluation metric integrating LLMs with verifiers like linters and code smell detectors, CRScore++ transforms these signals into training rewards. We show that CRScore++ improves a weaker student model through a combination of supervised fine-tuning and RL critique from a stronger teacher model, thus enabling generalization to novel programming languages. 1 Introduction In recent years, large language models (LLMs) have been increasingly adopted for software engineering tasks Zhang et al. 2023 . This trend has been driven in part by advancements in reinforcement learning (RL)-based training methods, particularly those incorporating verifiable rewards (RLVR), such as execution feedback Le et al. 2022 ; Dou et al. 2024 ; Shojaee et al. 2023 . While these learning signals have proven effective for tasks with structured outputs like code generation, applying verifiable rewards to natural language generation (NLG) tasks, such as code review, remains challenging. For NLG tasks, the research community has primarily relied on supervised fine-tuning (SFT) with high-quality examples, RL from human feedback (RLHF) Ouyang et al. 2022 , or RL from AI feedback (RLAIF) Lee et al. 2024 for greater scalability. While RLVR is both scalable and verifiable, RLHF and RLAIF depend on subjective preference signals OpenAI et al. 2024 ; Bai et al. 2022 , which can introduce unintended biases into models. On the other hand, some of these biases may be desirable, as evidenced by studies on using LLMs as evaluators (a form of RLAIF) for software engineering tasks, which demonstrate high agreement with human evaluators and similar score distributions He et al. 2025a ; Wang et al. 2025a . Applying RLVR to code review is particularly difficult due to the open-ended nature of textual outputs. As a result, recent work has explored integrating LLMs with verifiable static analysis tools, such as linters and code smell detectors, either during inference or for data generation. These approaches have been shown to improve both the evaluation Naik et al. 2024 and generation Jaoua et al. 2025a of review comments. Building on these ideas, we take steps toward leveraging verifiers in training by incorporating verifiable “facts” from static analysis tools alongside more subjective LLM feedback (RLAIF). This results in a new training framework, which we call CRScore++. Our framework consists of two stages: (1) learning from demonstration (SFT) and (2) learning from critique (Preference Optimization or PO), both of which incorporate verifier-generated “facts” into step-by-step reasoning. In the SFT stage, a stronger teacher model generates code review demonstrations for a student model using a chain-of-thought (CoT) approach that integrates relevant linter messages and code smells into the review demonstration. This builds on recent findings that show how incorporating external tools enhances the reasoning abilities of long CoT models Li et al. 2025 . In the PO stage, the teacher model critiques the student model’s review comments by first generating a list of expected review points (pseudo-references generated by CRScore Naik et al. 2024 ), incorporating tool outputs, and then applying CoT reasoning to evaluate whether these points are addressed. Following the findings of Sharma et al. 2024 , we use the same teacher model for both the SFT and PO stages * * * To reduce OpenAI API usage costs, we distill the target model into an open-source reward model for the PO stage . Beyond grounding RLAIF with verifiable facts from tools, our framework also indirectly trains the model to detect code smells and linter-related issues through CoT reasoning. A key advantage of our approach is its remarkable cross-language generalization capability. Models trained exclusively on Python code achieve nearly equivalent performance when generating reviews for Java and JavaScript, despite having no exposure to these languages during training. This result challenges the prevailing assumption that language-specific training data is essential for high-quality code review systems. Our work makes three primary contributions: • Introducing a novel RL method for training code models that combines the benefits of RLVR with the flexibility of RLAIF. • Applying this framework to integrate linters, code smell detectors, and security analyzers to measure review quality for code model training, which to the best of our knowledge hasn’t been explored in prior work. • Enabling generalizable learning of skills such as code smell detection, code efficiency evaluation, and assessments of security, readability, and maintainability through step-by-step reasoning. • Demonstrating robust cross-language generalization of code quality analysis abilities. Models trained exclusively on Python data maintain most of their performance when reviewing Java and JavaScript code without any language-specific fine-tuning. 2 Related Work 2.1 Code Review Automation Recent work on automated code review leverages large language models (LLMs) to generate reviews that resemble human-written reviews Li et al. 2022 . While systematic assessment can be done using benchmarking datasets Tufano et al. 2022 , conventional metrics like BLEU fail to measure semantic quality of generated reviews. Prior work demonstrated the potential of LLMs as automatic code reviewers He et al. 2025b and identified key reasoning abilities needed for code review comprehension Lin et al. 2025 , but less attention has been paid to evaluating review comment quality. Hybrid approaches for review comment generation combining static analyzers and LLMs Jaoua et al. 2025b are promising but concentrate largely on training data or inference time augmentation and, to a lesser extent, on the utilization of verification signals as rewards during training. Wang et al. 2025b ; Jaoua et al. 2025b and Naik et al. 2024 all show that LLM-as-a-judge methods achieve a significant correlation with human scores for software engineering and code review evaluation. Moreover, CRScore Naik et al. 2024 shows that integrating static analysis tools with LLMs and using pseudo-references can lead to a more accurate signal of review quality. Based on past evidence of how LLMs combined with tools can provide strong signals of review quality, our work extends this idea by using them as a reward signal for training LLMs to generate better review comments. Additionally, we also show that supervision and knowledge obtained from a tool for one language (e.g. Python) can lead to better reviews for unseen languages at test time (e.g. Java and Javascript). 2.2 Static Analysis Integration Static analysis tools provide structured signals regarding code quality by detecting code smells and enforcing style and maintainability standards Pecorelli et al. 2022 . Early work limited integration with LLMs to post-generation validation, but recent research has explored deeper synergy. Jaoua et al. 2025c systematically combined static analyzers and LLMs at multiple stages, including data augmentation, retrieval-augmented generation, and naive output concatenation, thus demonstrating that hybrid strategies can improve review relevance and coverage. However, their evaluation primarily relies on LLM as a judge relative comparison, whereas our