AI-Assisted Fixes to Code Review Comments at Scale
Chandra Maddila, Negar Ghorbani, James Saindon, Parth Thakkar, Vijayaraghavan Murali, Rui Abreu, Jingyue Shen, Brian Zhou, Nachiappan Nagappan, Peter C. Rigby · 2025
Read AI-Assisted Fixes to Code Review Comments at Scale by Chandra Maddila, Negar Ghorbani, James Saindon, Parth Thakkar, Vijayaraghavan Murali, Rui Abreu, Jingyue Shen, Brian Zhou, Nachiappan Nagappan, Peter C. Rigby (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13499.txt.
Introduction
Chandra Maddila and co-authors deploy AI fixes for code review at Meta in a 2025 paper. Tens of thousands of review comments land every week. Their system, MetaMateCR, drafts patches that answer reviewer comments directly.
A benchmark of sixty-four thousand comment and patch pairs trained the models. Fine-tuned Llama models form the core of the system. Offline tests compared them against a leading commercial model. Randomized safety trials then guarded the production rollout. Review time was watched so the tool could not slow humans down. Readers get a rare inside report on language models working at industrial scale.
Catalog record
| CREATOR | Chandra Maddila, Negar Ghorbani, James Saindon, Parth Thakkar, Vijayaraghavan Murali, Rui Abreu, Jingyue Shen, Brian Zhou, Nachiappan Nagappan, Peter C. Rigby |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-13499
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AI-Assisted Fixes to Code Review Comments at Scale Chandra Maddila, Negar Ghorbani, James Saindon, Parth Thakkar, Vijayaraghavan Murali, Rui Abreu, Jingyue Shen, Brian Zhou, Nachiappan Nagappan, Peter C. Rigby Abstract Aim. There are 10 s of thousands of code review comments each week at Meta. We developed Metamate for Code Review ( MetaMateCR ) that provides AI-assisted fixes for reviewer comments in production at scale. Method. We developed an internal benchmark of 64 k \langle review\_comment,patch\rangle data points to fine-tune Llama models. Once our models achieve reasonable offline results, we roll them into production. To ensure that our AI-assisted fixes do not negatively impact the time it takes to do code reviews, we conduct randomized controlled safety trials as well as full production experiments. Offline Results. As a baseline, we compare GPT-4o to our small and large Llama models. In offline results, our LargeLSFT model creates an exact match patch 68% of the time outperforming GPT-4o by 9 percentage points (pp). The internal models also use more modern Hack functions when compared to the PHP functions suggested by GPT-4o . Safety Trial. When we roll MetaMateCR into production in a safety trial that compares no AI patches with AI patch suggestions, we see a large regression with reviewers taking over 5% longer to conduct reviews. After investigation, we modify the UX to only show authors the AI patches, and see no regressions in the time for reviews. Production. When we roll LargeLSFT into production, we see an ActionableToApplied rate of 19.7%, which is a 9.2pp improvement over GPT-4o . Our results illustrate the importance of safety trials in ensuring that AI does not inadvertently slow down engineers, and a successful review comment to AI patch product running at scale. I Introduction Code review is important to ensure that high quality code is released into production. The process has evolved from a heavyweight inspection process to increasingly lightweight tool-based approaches that rely on reviewer expertise [ 1 , 2 , 3 ] . With the rise of generative models (LLMs), code review research has shifted towards automating more complex activities. For example, Cihan et al. [ 4 ] propose a method to do LLM-based code review and evaluate it in an enterprise setting. AutoTransform [ 5 ] proposed a neural-network based method for transforming code during the code review process. Recent works have used LLMs to generate code fixes from reviewer comments. For example, Google [ 6 ] reports on the creation of a production system. Over the last few years, we have also been developing and deploying Metamate for Code Review ( MetaMateCR ). We discuss the motivation for each of our models, using an illustrative example of a reviewer’s comment, and the AI patch fixes in Figure 1(c) . The figure also shows the UX of Metamate for Code Review ( MetaMateCR ) in production. We conduct safety trials to ensure that AI fix suggestions do not inadvertently slow down code review, and also rollout at scale. Below we summarize our motivation and briefly highlight our contributions through our research results. I-A GPT-4o : Public model When we started this research (in 2024), GPT-4o was the state-of-the-art public model for coding. We conducted an offline test on an internal benchmark using GPT-4o . We calculate how often the model produces the exact match (EM) patch from the comment and the number times that it creates a patch that can compile, that does not contain static analysis and lint warnings, and passes all tests (SPG). Results summary. As illustrated in Figure 1(c) a, we see that GPT-4o is able to correctly fix the patch, but because it only has publicly available information, it uses an outdated PHP function rather than the modern performant Hack function. Our offline evaluation shows that GPT-4o generates an exact match 59% of the time. When we roll it out into production, we find that authors accept the patch 13% of the time. I-B SmallLSFT : Llama-SFT on on a small human-labeled data set To fine-tune a model for our internal context, we turn to the Llama-3-70B. This open source public model is trained on both public documents and source code and serves as our foundation model for subsequent work [ 7 ] . Using this model, we pre-train on Meta’s internal code to create iCodeLlama-3-70B. We then curate a small high-quality human-labeled annotation data set and use supervised fine tuning (SFT) on iCodeLlama-3-70B. We call this model SmallLSFT . Results summary. As illustrated in Figure 1(c) b, we see that SmallLSFT uses a performant Hack function because it has been fine-tuned on the internal codebase. We find that SmallLSFT has an offline exact match of 63% and, in production experiments, authors accept the patch 16.13%. In percentage points (pp), this represents a 4 pp and 1.25 pp improvement over GPT-4o . I-C LargeLSFT : Llama-SFT on a large LLM-labeled code review data set It is difficult to manually curate a large training set of code review comments and appropriate change. We use the manually created SmallLSFT data set to train iCodeLlama-3-8B as a classifier to curate a large SFT dataset. We are able to obtain 64k \langle review\_comment,patch\rangle datapoints. We train iCodeLlama-3-70B on this large dataset to create LargeLSFT . Results summary. As illustrated in Figure 1(c) c, we see that LargeLSFT not only avoids PHP and uses performant Hack functions, but it writes clean code based on the large set of review comments that lead to accepted patches that it was trained on. LargeLSFT has an offline exact match of 68% which is an improvement of 5pp and 9pp over SmallLSFT and GPT-4o , respectively. The corresponding production value is the ActionableToApplied rate of 19.75%, which is a 3.6 pp and 6.6 pp improvement over SmallLSFT and GPT-4o , respectively. I-D Safety Trials in Production Concurrently to the development of the models and offline tests, the engineering team designed a new user experience for MetaMateCR and conducted safety trials, similar to those used in medicine [ 8 ] , to ensure that there were no unintended side effects of AI patch fixes. We conducted two randomized controlled safety experimental trials to ensure that AI patches from code review comments did not have a negative impact on the amount of time diffs spent in review, TimeInReview, and that reviewers did not spend more time actively examining diffs, TimeSpent, and the total WallClock from when the diff is published until it is closed. Results summary. The amount of time needed to do reviews increased by over 5%. Investigation revealed that reviewers were checking the AI patches. In the second experiment, we only showed the AI suggestions to the authors and collapsed them for the reviewers. This change reduced the reviewer burden and led to a safe user experience that did not have any regressions in the time it takes for code review. I-E Full Production Experiments Unlike medical trials [ 8 ] , we are able to monitor the entire population of code reviews to ensure and continuously evaluate our goal and safety metrics [ 9 ] . We want all engineers to experience the best AI patch suggestion possible, so we rolled out the new