On the Footprints of Reviewer Bots’ Feedback on Agentic Pull Requests in OSS GitHub Repositories Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793599 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software engineering Agentic software engineering

Syeda Kaneez Fatima, Yousuf Abrar, Abdul Rehman Tahir, Amelia Nawaz, Shamsa Abid, Abdul Ali Bangash · 2026

Read On the Footprints of Reviewer Bots’ Feedback on Agentic Pull Requests in OSS GitHub Repositories Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793599 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software engineering Agentic software engineering by Syeda Kaneez Fatima, Yousuf Abrar, Abdul Rehman Tahir, Amelia Nawaz, Shamsa Abid, Abdul Ali Bangash (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24450.txt.

Introduction

Fatima and five colleagues study reviewer-bot feedback on agentic pull requests in GitHub projects. Analyzing 7,416 bot comments on 4,532 pull requests, they find comments focus on bug fixes, testing, and documentation, are civil and prescriptive. Bots give clear, concise feedback, though relevance to code changes is moderate. Higher bot activity volume links to longer resolution times and lower feedback quality, as more comments dilute pertinence. Feedback quality shows no meaningful link to workflow outcomes, suggesting bots should favor targeted high-relevance feedback over sheer volume.

Catalog record

CREATORSyeda Kaneez Fatima, Yousuf Abrar, Abdul Rehman Tahir, Amelia Nawaz, Shamsa Abid, Abdul Ali Bangash
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21549
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24450

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Footprints of Reviewer Bots’ Feedback on Agentic Pull Requests in OSS GitHub Repositories Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793599 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software engineering Agentic software engineering Syeda Kaneez Fatima, Yousuf Abrar, Abdul Rehman Tahir, Amelia Nawaz, Shamsa Abid, Abdul Ali Bangash Abstract . Autonomous coding agents are reshaping software development by creating pull requests (PRs) on GitHub, referred to as agentic PRs. In parallel, the review process is also becoming autonomous, thereby making reviewer bots key actors in the assessment of these agentic PRs. However, their influence on PR acceptance and resolution remains unclear. This study empirically investigates the relationship between reviewer-bot feedback and PR outcomes by analyzing how Reviewer Bot Feedback Quality (relevance, clarity, conciseness) and Reviewer Bot Activity Volume (comment count) are associated with PR acceptance and resolution time. We analyze 7,416 7,416 reviewer-bot comments on 4,532 4,532 PRs from the AI_Dev dataset (a dataset that captured AI agents’ PRs in GitHub projects). Our results show that reviewer-bot comments mainly focus on bug fixes, testing, and documentation, are civil in tone, and are prescriptive in nature. Reviewer bots generally produce clear and concise feedback, though the semantic relevance of comments to underlying code changes is moderate. We find that higher Reviewer Bot Activity volume is associated with longer PR resolution times and lower average feedback quality, showing that as bots generate more comments on a PR, the average pertinence of that feedback appears to degrade. At the same time, Reviewer Bot Feedback Quality shows no meaningful association with workflow outcomes. Our findings suggest that, in agentic PR workflows, reviewer bots should prioritize targeted high-relevance feedback over generating large numbers of comments. † † cc-license: by 1. Introduction Autonomous coding agents introduce a distinct class of pull requests (PRs) in which code is generated or modified entirely by AI systems ( 20 ) . We refer to these contributions as agentic pull requests (agentic PRs) . In parallel, autonomous systems increasingly support the code review process by automatically generating feedback on submitted changes ( 10 ; 4 ) . These systems, commonly termed as automated code review agents ( 11 ) or reviewer bots ( 19 ) , are now frequently involved in reviewing agentic PRs ( 2 ; 4 ) , contributing to the broader automation of modern software workflows ( 1 ; 26 ; 5 ) . While prior work has examined human perceptions of AI-generated code ( 14 ; 7 ) , the interaction between reviewer bots and agentic PRs remains underexplored ( 20 ) . In particular, there is limited empirical evidence on how reviewer bot communication characteristics affect workflow outcomes. We argue that understanding the influence of Bot Feedback Quality (defined in terms of relevance, clarity, and conciseness), and Bot Activity Volume (measured as the number of bot comments per PR), is critical for evaluating review effectiveness, developer experience, and CI/CD pipeline reliability ( 17 ) . This paper presents a quantitative and correlational analysis of reviewer-bot comments on agentic PRs using review data from the AI_Dev dataset ( 10 ) . We analyze the types of issues highlighted by bots, the nature and civility of their comments, and assess feedback quality using the categorization and scoring framework proposed by Sghaier et al. ( 15 ) (detailed in Section 3.2 ). We then empirically examine how these communication characteristics relate to PR outcomes. Specifically, we test whether higher Reviewer Bot Feedback Quality is associated with reduced PR resolution time and higher acceptance rates, and contrast this effect with that of Reviewer Bot Activity Volume ( 1 ) . Accordingly, we address the following research questions: RQ1: How can reviewer bot comments on agentic pull requests be characterized in terms of type, nature, civility, and Bot Feedback Quality (relevance, clarity, and conciseness)? RQ2: To what extent do these variables, specifically the Bot Feedback Quality and the Bot Activity Volume on a pull request, correlate with PR resolution time and acceptance rate? Replication Kit: Our replication kit is available at ( 18 ) . 2. Related Work Research on automated code review has expanded rapidly, with recent surveys documenting the growing use of large language models (LLMs) in software engineering tasks ( 24 ) . Prior studies, including those by Schäfer et al. ( 14 ) and Tufano et al. ( 19 ) , evaluate automation for activities such as test generation and code review. While Watanabe et al. ( 20 ) , and Wessel et al. ( 22 ; 23 ) identified this shift; Li et al. ( 10 ) released the AI_Dev dataset to study it at scale, existing work focuses largely on coarse-grained metrics such as turnaround time and explicitly notes that the dynamics of automated review on agent-generated code remain unexplored. We address this gap by systematically profiling bot comment quality (RQ1) and assessing their correlation with PR resolution and acceptance rates (RQ2). Unlike prior work ( 7 ; 13 ; 21 ) that evaluates automated reviewers primarily in terms of their utility to human developers or treats bot feedback as a black box, we quantify specific interaction characteristics, i.e., relevance, clarity, and conciseness, and empirically test whether feedback quality or activity volume drives workflow delays. 3. Methodology 3.1. Dataset We use the AI_Dev dataset by Li et al. ( 10 ) , which is a large-scale empirical corpus of AI based PRs, review comments, and associated metadata collected from open-source software (OSS) projects on GitHub. We analyze 7,416 reviewer-bot comments from the AI_Dev dataset. Each record contains the bot-authored review comment and the associated diff_hunk (the specific block of modified code). After we pre-process the data and filter out non-bot comments, the final dataset contains 7,416 reviewer-bot comments generated by 29 distinct bots across 4,532 agentic PRs. 3.2. Categorization and Scoring Framework To identify the interaction patterns of reviewer bots with agentic PRs (RQ1), we adopt the framework by Sghaier et al. ( 15 ) and use it to evaluate the characteristics and quality of reviewer-bot comments. Their framework classifies comments along three Categorical dimensions: Type, Nature, and Civility , and scores each subcategory according to three Scoring Criteria Metrics: Relevance, Clarity, and Conciseness . Among the categories, Type captures focus (e.g., Bugfix, Testing, Documentation), Nature captures expression style (e.g., Prescriptive, Clarification), and Civility captures tone (Civil or Uncivil) of the comment. Similarly, in the scoring framework, Relevance measures pertinence to the code change, Clarity measures how clearly the comment conveys its message, and Conciseness measures brevity and efficiency, on a scale of 1-10. A score of 1 indicates a very irrelevant, unclear, and non-concise comment, and 10 shows a highly relevant, clear, and concise review comment, respectively. To annotate reviewer-bot comments, we use GPT-5.1 as an automated annotator following the framework of Sghaier et al. ( 15 ) . The model categorizes each comment by Type , Nature , and Civility , and assigns 1–10 ordinal scores for Relevance , Clarity , and Conciseness . The annotation prompt defines all categorization and scoring criteria, includes an illustrative example, and provides

Read the complete text