Let’s Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 1
Haruhiko Yoshioka, Takahiro Monno, Haruka Tokumasu, Taiki Wakamatsu, Yuki Ota, Nimmi Weeraddana, Kenichi Matsumoto · 2026
Read Let’s Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 1 by Haruhiko Yoshioka, Takahiro Monno, Haruka Tokumasu, Taiki Wakamatsu, Yuki Ota, Nimmi Weeraddana, Kenichi Matsumoto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18749.txt.
Introduction
Haruhiko Yoshioka and six colleagues study what makes a code change count, for both human hands and smart agents. Code hosts now fill with change asks made by agents. Those asks are fast and cheap to make, but they are merged at a lower rate than asks by human hands. The team ask why, in a large study of past asks from a shared data set.
Their set holds tens of thousands of asks, by both kinds of maker. They weigh traits such as size, tests, and the words that go with each ask. Traits that track with a merge for one kind may not track for the next. A clear map could help agent teams write asks that land. It could also help hosts set fair bars for all makers.
Catalog record
| CREATOR | Haruhiko Yoshioka, Takahiro Monno, Haruka Tokumasu, Taiki Wakamatsu, Yuki Ota, Nimmi Weeraddana, Kenichi Matsumoto |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18749
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Let’s Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793567 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering Collaboration in software development CCS: Software and its engineering Software creation and management Haruhiko Yoshioka, Takahiro Monno, Haruka Tokumasu, Taiki Wakamatsu, Yuki Ota, Nimmi Weeraddana, Kenichi Matsumoto Abstract . The automatic generation of pull requests (PRs) using AI agents has become increasingly common. Although AI-generated PRs are fast and easy to create, their merge rates have been reported to be lower than those created by humans. In this study, we conduct a large-scale empirical analysis of 40,214 PRs collected from the AIDev dataset. We extract 64 features across six families and fit statistical regression models to compare PR merge outcomes for human and agentic PRs, as well as across three AI agents. Our results show that submitter attributes dominate merge outcomes for both groups, while review-related features exhibit contrasting effects between human and agentic PRs. The findings of this study provide insights into improving PR quality through human–AI collaboration. † † cc-license: by 1. Introduction In software development, creating pull requests (PRs) requires substantial effort from developers ( 7 ) . Beyond implementing code changes, developers must also provide contextual information, such as clear statements of intent, references to related issues, and responses to peer-reviewer feedback. Existing literature ( 6 ; 20 ; 21 ; 13 ; 16 ; 1 ) has extensively analyzed the factors influencing the acceptance or rejection of developer-authored PRs. For example, Gousios et al. ( 6 ) have revealed that PR acceptance is influenced by change size, file activity, project scale, contributor’s track record, and the review process. Furthermore, others have shown that the richness of descriptions, the nature of discussions, and the presence of tests ( 20 ) impact the acceptance of developers’ PRs. To help developers create PRs, Artificial Intelligence (AI) agents are used to automatically create change sets and submit PRs ( 15 ) . However, AI-generated PRs (i.e., Agentic PRs ) are not without their limitations ( 5 ; 2 ; 3 ; 4 ; 15 ) . In fact, AI-generated code has been reported to be structurally simple and repetitive, prone to containing vulnerabilities ( 5 ) , and identifiable by specific lexical and syntactic features ( 2 ) . Besides, PRs that include at least one ChatGPT shared link take longer to be closed (merged or abandoned) ( 3 ) . Then, Li et al. ( 15 ) demonstrated that the acceptance rate of Agentic PRs (i.e., the proportion of Agentic-PRs that are successfully merged) in AIDev dataset is significantly lower than that of humans. However, little is known about the quality and acceptability of such PRs compared to those authored by humans ( 15 ; 5 ) . The primary goal of this study is to enhance PR quality by leveraging the complementary strengths of humans and GenAI. To compare the factors that characterize the accepted PRs that were created by humans and agents, we first extract 64 features across 6 families from a dataset of 6,618 human-authored PRs and 33,596 agentic PRs from the AIDev dataset ( 15 ) . These features include change set size, number of files added/deleted/changed, task types, external links, and AI agent type (e.g., OpenAI Codex, Copilot). For humans and each of the AI models in AIDev dataset, we fit and analyze regression models to uncover the features of PRs that make them difficult for GenAI agents to get their PR merged but easy for humans (and vice versa). By gaining insights into such features, we provide actionable evidence regarding the behavior of AI agents in real-world projects, offering guidelines for developers and project managers to optimize development processes where AI and humans collaborate ( 15 ) . This study addresses the following three research questions (RQs): RQ1: To what extent can statistical regression models determine whether a PR created by a human or an AI agent will be merged? As a foundation for our comparative analysis between humans and AI agents, we analyze the extent to which the constructed regression models can perform on real-world PRs. RQ2: How do the features contributing to merge probability differ between PRs created by humans and those created by AI agents? This RQ aims to uncover the respective strengths and weaknesses of humans and agents in getting their PRs merged. We investigate the explanatory power of each feature family in our statistical models to identify which ones contribute most to PR merging, providing insights into which aspects are most critical for humans versus agents. RQ3: How do the features contributing to merge probability differ across PRs generated by different AI agents? In this RQ, we compare the importance of feature families in predicting the merge outcomes of agent-authored PRs across different AI agents, with the goal of providing insights into which AI agents are better suited for specific development contexts. Our key findings reveal that: (1) submitter attributes dominate merge outcomes for both human and agentic PRs; (2) increased review activity is associated with higher merge likelihood for human PRs but lower for agentic PRs; and (3) AI agents show distinct merge patterns. Figure 1. An overview of our study design 2. Study Design Table 1. Families of Features Family # Features PR Change Size & Commit Characteristics 18 PR Description 8 Submitter Attributes 6 Repository Attributes 3 Issue Linkage & Surrounding Context 3 Reviews & Discussion 26 Total number of features 64 In this section, we describe our method for data preparation (Section 2.1 ) and the model creation (Section 2.2 ). Figure 1 overviews our study design. 2.1. Data Preparation Collect metadata from AIDev dataset. We begin with the AIDev dataset (Nov. 2025) released by Li et al. ( 15 ) , which consists of 932,791 Agentic PRs created by five major autonomous AI agents. Of these, 33,596 include rich metadata such as comments, issues, reviews, commits, repositories, timelines, and user information. Collect metadata for human-authored PRs. For comparison purposes, the AIDev dataset contains 6,618 human-authored PRs created by 2,515 developers (hereafter referred to as human PRs ) across 818 repositories. However, for these human PRs, the AIDev dataset provides only repository-level information and task type, and does not include other metadata such as comments, reviews, or commits. To enable a fair comparison between agentic PRs and human PRs, it is necessary to use the same set of features for both. Therefore, we collect the same set of metadata for human PRs as those available for agentic PRs by using the same GitHub API endpoints as Li et al. when constructing the AIDev dataset ( 15 ) . Extract features from human PRs and agentic PRs, covering distinct families of features. We use our dataset of metadata to extract 64 features across six families (while consulting prior work ( 21 ; 19 ; 20 ) ) to represent different dimensions of a PR, ranging from code changes to social interactions. Table 1 shows these families of features. For