Understanding Dominant Themes in Reviewing Agentic AI-authored Code Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil
Md. Asif Haider, Thomas Zimmermann · 2026
Read Understanding Dominant Themes in Reviewing Agentic AI-authored Code Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil by Md. Asif Haider, Thomas Zimmermann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19287.txt.
Introduction
Md. Asif Haider and Thomas Zimmermann study how human reviewers react to code written by AI agents. Past work graded how well agents generate code, but little was known about the review side of real practice. Their large scale study reads code review on pull requests that agents authored.
They map the dominant themes in what reviewers say and ask for. Review is where agent code meets human judgment, trust, and standards. Knowing what reviewers balk at tells agent builders what to fix. It is not just whether the code runs. A human must also accept it into the project.
Catalog record
| CREATOR | Md. Asif Haider, Thomas Zimmermann |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19287
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Dominant Themes in Reviewing Agentic AI-authored Code Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793566 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering CCS: Computing methodologies Artificial intelligence Md. Asif Haider, Thomas Zimmermann Abstract . While prior work has examined the generation capabilities of Agentic AI systems, little is known about how reviewers respond to AI-authored code in practice. In this paper, we present a large-scale empirical study of code review dynamics in agent-generated PRs. Using a curated subset of the AIDev dataset, we analyze 19,450 inline review comments spanning 3,177 agent-authored PRs from real-world GitHub repositories. We first derive a taxonomy of 12 review comment themes using topic modeling combined with large language model (LLM)-assisted semantic clustering and consolidation. According to this taxonomy, we then investigate whether zero-shot prompts to LLM can reliably annotate review comments. Our evaluation against human annotations shows that open-source LLM achieves reasonably high exact match (78.63%), macro F1 score (0.78), and substantial agreement with human annotators at the review comment level. At the PR level, the LLM also correctly identifies the dominant review theme with 78% Top-1 accuracy and achieves an average Jaccard similarity of 0.76, indicating strong alignment with human judgments. Applying this annotation pipeline at scale, we find that apart from functional correctness and logical changes, reviews of agent-authored PRs predominantly focus on documentation gaps, refactoring needs, styling and formatting issues, with testing and security-related concerns. These findings suggest that while AI agents can accelerate code production, there remain gaps requiring targeted human review oversight. † † cc-license: by 1. Introduction Autonomous coding agents, also referred to as Agentic-AI Software Engineers, are now assisting humans in end-to-end development tasks, including coding, debugging, testing, submitting pull requests (PRs), acting on reviews, suggesting repairs, and generating documentation. This new paradigm of human-AI interaction faces some core challenges, such as establishing trust ( 18 ) , as seen in human-driven workflows. Developers still hesitate to accept and merge AI-generated PRs, and the huge surge in the volume of code generated by AI agents just makes the situation worse. As reported in Hassan et al. ( 6 ) , nearly 70% of agent-authored PRs face longer review times, remain completely unreviewed, or get rejected soon, making the review process a major performance bottleneck. Hence, understanding the dynamics of AI-authored code reviews can fill this exposed gap on our path to enhancing human-AI collaboration. While datasets exist that capture how human-generated reviews are treated, finding large-scale empirical data on agentic-PRs has remained a challenge. Recent research in the mining software repository community presents datasets ( 9 ; 21 ) comprising PRs with rich metadata, authored by leading AI agents on the market. Using such data, we aim to address the following timely research questions: • RQ1: How accurately can LLMs annotate themes of code reviews commented on Agentic PRs? • RQ2: What aspects/topics/themes are the most prevalent ones that receive attention while reviewing Agentic PRs? • RQ3: What comment themes are dominant in successful vs rejected PRs? Insights from these findings can be further leveraged to reduce failure rates, improve human-AI collaboration, and boost AI model training performance that prioritizes learning from mistakes in real-world projects. We provide our replication package via Zenodo ( 1 ) . 2. Dataset We utilize the AIDev ( 9 ) dataset in our study. This is a large-scale dataset comprising Agentic-PRs from real-world GitHub projects. AIDev contains 932,791 Agentic-PRs authored by five most popular AI coding agents: OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code across 116,211 repositories involving 72,189 developers. Each PR in the dataset is linked to its corresponding repository and developer, along with additional metadata. The authors also released a curated subset of 33,596 Agentic-PRs from 2,807 repositories with more than 100 GitHub stars. This enriched subset provides review comments, commit-level diffs, event timelines, and related issues. For our analysis, we focus solely on this subset of AIDev. Specifically, we work on Zenodo version v2 of the data ( 8 ) . Table 1. Taxonomy of Review Comment Themes Tag Thematic Category Significant Keywords secu Security, Safety & Reliability vulnerability, security, tokens, safety, permissions, reliability, attack, defense test Testing, Assertions & Validation test, assertion, expected, behavior, sql-checks, database, validation style Styling & Formatting Adjustments line-level, text, formatting, rename, variable, attributes, readability, indentation, lint, consistency, style docs Documentation, Logs & Developer Guidance documentation, api, guide, reference, section, markdown, changelog, typo, notes, logs chore Dependency, Module & Import Management dependencies, module, import, export, package, sdk, optional-deps, resolution, cleanup build Build, Configuration & Default Behavior Management build, permission, config, default, maven, gradle, cargo, script ci CI/CD Workflow Management workflow, ci, cd, versions, install, migration, github-actions, integration, deploy, travis feat Core Implementation & Feature Development feature, new, implementation, logic, pattern, approach, functionality, helper, utility refactor Code Restructure, Simplification & Redundancy Elimination unused, redundant, unnecessary, false-positive, refactor, dead-code, maintainability, simplify, restructure cmd Command-Line Tools & Developer Utilities cli, commands, batch, python-tools, dev-utilities undo Reverts, Rollbacks & Change Rejection revert, rollback, undo, restore, preserved, unchanged, rejection perf Performance Improvement memory, consumption, execution, speed, performance, monitor 3. RQ1: LLM as Review Theme Annotator Here, we describe our approach to utilizing open-source LLMs for review comment topic categorization and evaluate LLM performance relative to human annotations. Since each PR may contain multiple review comments from reviewers, we first narrow our focus down to the pr_review_comments table of the AIDev dataset ( 8 ) . This particular table contains inline code review comments with rich file-level context, including file path, diff hunks/code patches, and timestamps. To retrieve the PR level information and link it to the detailed review comments, we first run a join query combining the following tables: pull_request , pr_reviews , and pr_review_comments . This leaves us with 19,450 review comments from 3,177 unique PRs. 3.1. Deriving a Taxonomy of Categories To derive a taxonomy for review comment categorization, we first take inspiration from the conventional commit specifications (CCS) proposed by Zeng et al ( 22 ) , an approach adopted in recent literature for PR body content categorization ( 21 ; 9 ) as well. However, review comments differ from commit messages and PR bodies; hence, to identify review-specific textual signals in the review comment corpora, we further adopt a topic modeling-based approach to cluster potential themes with their corresponding keywords. In particular, we concatenate all review comment body field entries for each PR into a document, and use transformer-based BERTopic modeling ( 4 ) on the documents to identify unique potential themes and the most significant keywords associated with each hierarchical cluster. Based on the keywords and 49 initial clusters, we then utilize ChatGPT (GPT-5) ( 15 ) to gain a deeper understanding of the probable underlying definition of each cluster. To combine relevant clusters together, we also prompt ChatGPT to form groups of related clusters, keeping the unique keywords intact as much as possible. Thus, we identify 42 fine-grained topics and group them into 12 distinct thematic categories. Table 1 outlines the proposed set of Thematic Category with the most Significant Keywords associated with each topic.