Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793575 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering Collaboration in software development CCS: Software and its engineering Software configuration management and version control systems CCS: Software and its engineering Software development techniques

Sien Reeve O. Peralta, Fumika Hoshi, Hironori Washizaki, Naoyasu Ubayashi, Inase Kondo, Yoshiki Higo, Hiroki Mukai, Norihiro Yoshida, Kazuki Kusama, Hidetake Tanaka, Youmei Fan · 2026

Read Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793575 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering Collaboration in software development CCS: Software and its engineering Software configuration management and version control systems CCS: Software and its engineering Software development techniques by Sien Reeve O. Peralta, Fumika Hoshi, Hironori Washizaki, Naoyasu Ubayashi, Inase Kondo, Yoshiki Higo, Hiroki Mukai, Norihiro Yoshida, Kazuki Kusama, Hidetake Tanaka, Youmei Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22534.txt.

Introduction

Peralta and team ask why agent pull requests merge or fail, from 11,048 closed agent PRs, 9,799 human-checked, with 717 read cases. Failed outcomes exceed agent error. Only 35.7 percent of rejected PRs showed clear agentic faults; 31.2 percent stemmed from workflow constraints and 33.1 percent lacked seen reasons. Among merged PRs, 15.4 percent needed review help, and 5.5 percent showed no talk trace. Copilot and Devin appear more in review-led workflows; Codex and Cursor PRs merge with little talk. PR outcomes alone miss agent skill; talk-aware judging is needed.

Catalog record

CREATORSien Reeve O. Peralta, Fumika Hoshi, Hironori Washizaki, Naoyasu Ubayashi, Inase Kondo, Yoshiki Higo, Hiroki Mukai, Norihiro Yoshida, Kazuki Kusama, Hidetake Tanaka, Youmei Fan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20516
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.22534

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793575 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering Collaboration in software development CCS: Software and its engineering Software configuration management and version control systems CCS: Software and its engineering Software development techniques Sien Reeve O. Peralta, Fumika Hoshi, Hironori Washizaki, Naoyasu Ubayashi, Inase Kondo, Yoshiki Higo, Hiroki Mukai, Norihiro Yoshida, Kazuki Kusama, Hidetake Tanaka, Youmei Fan Abstract . AI coding agents increasingly submit pull requests (Agentic-PRs) to open-source repositories, yet their performance is commonly assessed using merge and rejection outcomes alone. We hypothesized that these outcome labels do not reliably reflect agent capability without considering review interactions. To test this, we conducted a decision-oriented analysis of 11,048 closed Agentic Pull Requests, refined to 9,799 human-reviewed PRs, and manually inspected 717 representative cases to recover decision rationale from interaction artifacts. We found that rejection outcomes substantially overstate agent error: only 35.7% of rejected PRs reflected clear agentic failures, while 31.2% were driven by workflow constraints and 33.1% lacked observable decision rationale. Among merged PRs, 15.4% required explicit reviewer involvement through feedback or direct commits, and 5.5% showed no visible interaction trace. We further observed systematic differences across agents, with Copilot and Devin more often embedded in reviewer-mediated workflows, while Codex and Cursor PRs were typically merged with minimal interaction. These results reject the assumption that PR outcomes alone capture agent performance and demonstrate the need for interaction-aware evaluation grounded in review behavior. † † cc-license: by-nc-nd 1. Introduction AI coding agents increasingly submit Agentic Pull Requests (Agentic-PRs) that participate directly in real-world software development. Evidence from the AIDev dataset shows that agent-authored PRs span thousands of repositories ( 11 ) , reflecting a broader shift toward AI-native software engineering (SE 3.0) ( 8 ) . Prior work reports productivity gains from AI-assisted coding ( 5 ; 16 ) , changes in review practices ( 2 ) , and growing concerns related to maintainability ( 13 ) and security risks ( 15 ) . Despite widespread adoption, it remains unclear why Agentic-PRs are merged or rejected. Existing studies predominantly rely on outcome-based measures such as merge rates or approval frequencies, which provide limited insight into how review decisions are formed. In practice, pull request decisions are shaped by interaction-level factors, including reviewer comments, CI outcomes, commit history, and workflow actions, rather than PR states alone. This gap complicates the interpretation of both rejection and merge outcomes. Rejections may reflect workflow constraints or undocumented reviewer decisions rather than agentic failures, while merged PRs may depend on varying degrees of human involvement. Without examining interaction evidence, outcome-based evaluations risk misrepresenting agent capability. To address this limitation, we analyze closed Agentic-PRs with evidence of human evaluation. Our dataset includes 11,048 closed PRs from repositories with at least 500 stars, refined to 9,799 human-reviewed PRs. We manually inspect a stratified sample of 717 PRs to examine how merge and rejection decisions are formed in practice. This study is guided by two research questions: RQ1: What drives the rejection of Agentic-PRs? Motivation: Rejections may stem from agentic failures, workflow constraints, or undocumented reviewer decisions. Distinguishing among these factors is necessary to avoid misattributing rejection outcomes to agent capability alone. RQ2: How does human involvement shape successful Agentic-PRs? Motivation: We examine whether merged PRs reflect autonomous agent performance or depend on collaborative refinement through reviewer feedback or direct human intervention. Key Takeaway. Analysis of 717 manually inspected Agentic-PRs demonstrates that pull request outcomes alone are insufficient for assessing agent performance. Only 35.7% of rejections correspond to clear agentic failures, while 31.2% are driven by workflow factors and 33.1% lack observable decision rationale. Moreover, 15.4% of merged PRs require explicit reviewer involvement, indicating that both success and failure outcomes depend on interaction-level processes that are invisible in merge or rejection labels alone. Contributions. This paper makes three contributions: (1) a decision-oriented empirical analysis that disentangles agentic failures from workflow-driven and undocumented rejection outcomes in Agentic-PRs; (2) a quantitative characterization of human involvement in successful Agentic-PRs, including feedback loops and reviewer-applied commits; and (3) empirical evidence that outcome-based metrics systematically conflate agent capability with repository workflows, motivating interaction-aware evaluation of AI coding agents. 2. Motivation Prior empirical studies show that AI coding agents can improve developer productivity and accelerate task completion ( 5 ; 16 ) , while also introducing challenges related to review effort, maintainability, and security ( 13 ; 15 ) . Large-scale analyses of Agentic-PRs, including those enabled by the AIDev dataset ( 11 ) , predominantly assess agent performance using outcome-based measures such as approval rates or merge frequency. However, these measures provide limited insight into how merge and rejection decisions are formed. Software engineering research has shown that pull request decisions rely heavily on interaction-level signals, including discussion threads, CI outcomes, and iterative revisions, rather than final PR states alone ( 7 ) . Studies of AI-assisted code review similarly indicate that reviewer judgments are often conveyed through unstructured comments or small corrective actions that are not captured by metadata or formal review labels ( 4 ; 14 ) . As a result, outcome-based evaluations may conflate agent behavior with workflow conventions, reviewer intervention, or project-specific practices. A decision-oriented analysis grounded in interaction evidence is therefore necessary to distinguish agentic failures from workflow-driven and human-mediated outcomes. Such separation enables more accurate interpretation of agent performance and supports a clearer understanding of Human-AI collaboration in software development ( 19 ; 12 ; 9 ) . 3. Data Collection Figure 1. Overview of the study pipeline. Overview of the study pipeline: (A) data collection and initial filtering, (B) refined dataset definition, (C) stratified sampling strategy, and (D) manual qualitative coding process. Step A in Figure 1 summarizes the initial construction of the dataset. We build on the AIDev dataset ( 11 ) and restrict the scope to closed Agentic-PRs, defined as pull requests with explicit merge or rejection decisions. To limit the analysis to repositories with established review activity, we apply a threshold of at least 500 stars. After this filtering step, the dataset contains 11,048 Agentic-PRs, including 6,774 merged and 4,274 rejected submissions. The refined dataset definition is shown in step B in Figure 1 . We exclude bot-only reviewed submissions, defined as agent-authored PRs without any human comments, to reduce cases where no human decision context is available. This step removes 1,249 PRs and yields a final dataset of 9,799 human-reviewed Agentic-PRs across multiple agents and repositories. Table 1 summarizes the composition of the refined dataset by agent, reporting total, merged, and rejected PR counts. The dataset is unevenly distributed across agents: OpenAI Codex and Devin together account for the majority of Agentic-PRs, while Copilot and Cursor contribute smaller but substantial portions, and Claude Code appears least frequently. Across all agents, merged PRs are more common than rejected PRs, reflecting the overall composition of the refined dataset rather than agent-specific merge behavior. Table 1. Per-agent distribution of merged and rejected Agentic Pull Requests in the refined dataset after removing bot-only reviews (Figure

Read the complete text