Early-Stage Prediction of Review Effort in AI-Generated Pull Requests Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazi

Dao Sy Duy Minh, Huynh Trung Kiet, Nguyen Lam Phu Quy, Pham Phu Hoa, Tran Chi Nguyen, Nguyen Dinh Ha Duong, Truong Bao Tran · 2026

Read Early-Stage Prediction of Review Effort in AI-Generated Pull Requests Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazi by Dao Sy Duy Minh, Huynh Trung Kiet, Nguyen Lam Phu Quy, Pham Phu Hoa, Tran Chi Nguyen, Nguyen Dinh Ha Duong, Truong Bao Tran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00753.txt.

Introduction

Dao Sy Duy Minh and six co-authors study, in 2026, the review load that AI coding agents place on human teams. The agents now file whole code changes, called pull requests, on their own. The team read over thirty-three thousand such requests. About twenty-eight in each hundred merged at once, with no fuss.

The worst cases end in ghosting, where the agent drops the task after hard feedback. That tail of costly requests acts as a hidden tax on staff time. The authors build a guard model that flags costly requests at birth. It uses plain cues, such as file types and the size of the change. At a fifth of the review budget, it caught about sixty-nine in each hundred of the costly ones. Teams could thus fail cheap requests fast and speed the easy wins.

Catalog record

CREATORDao Sy Duy Minh, Huynh Trung Kiet, Nguyen Lam Phu Quy, Pham Phu Hoa, Tran Chi Nguyen, Nguyen Dinh Ha Duong, Truong Bao Tran
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00753

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Early-Stage Prediction of Review Effort in AI-Generated Pull Requests Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793609 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering Software evolution Dao Sy Duy Minh, Huynh Trung Kiet, Nguyen Lam Phu Quy, Pham Phu Hoa, Tran Chi Nguyen, Nguyen Dinh Ha Duong, Truong Bao Tran Abstract . As AI coding agents evolve from autocomplete tools to autonomous “AI workforce” teammates, they introduce a critical new bottleneck: human maintainers must now manage complex interaction loops rather than just reviewing code. Analyzing 33,707 agent-authored PRs, we uncover a stark two-regime reality: agents excel at narrow automation (28.3% of PRs merge instantly), but frequently fail at iterative refinement , leading to “ghosting” (abandonment) when faced with subjective feedback. This creates a hidden “attention tax” on maintainers. We introduce a creation-time Circuit Breaker model to predict high-maintenance PRs before human review begins. By leveraging simple static complexity cues (e.g., file types, patch size), our model identifies the “expensive tail” of contributions with AUC 0.96 , enabling a gated triage process. At a 20% review budget, this approach captures 69% of the high-effort PRs, effectively allowing maintainers to fast-fail costly, low-quality agent contributions while fast-tracking simple fixes. † † cc-license: by 1. Introduction As AI agents evolve from assistants to autonomous teammates ( 12 ) , they flood repositories with code. While some contributions force-multiply productivity, others devolve into “approval churning”—agents submit changes without resolving core issues, ultimately ghosting the reviewer. We identify a two-regime behavioral pattern distinguishing agents from humans: seamless success in narrow automation , versus frequent failure in iterative refinement . This distinction matters because it splits the maintainer’s role: they are either stamping simple fixes or rescuing flailing agents. This study motivates a shift to Automated Governance : can we identify high-effort drains before any human context switch? Research Questions. RQ1: Can we predict review effort at creation time using only structural signals? RQ2: What specific behaviors (e.g., lack of planning) correlate with agentic ghosting? Contributions. (1) Concept of Agentic Ghosting : We operationalize and quantify the prevalence of agents abandoning PRs during review. (2) Circuit Breaker Model : We demonstrate that high-effort PRs are structurally distinct, predictable with AUC 0.96 using simple complexity features ( RQ1 ). (3) Behavioral Insights : We show that “unplanned” conceptual changes are the primary driver of interaction failure ( RQ2 ). Artifact: https://zenodo.org/records/17993901 . 1.1. Related Work PR review effort and lifetime are well-studied: work practices ( 7 ) , size/complexity determinants ( 18 ; 15 ) , and interventions like automated reminders ( 17 ) inform triage strategies. This aligns with modern code review (MCR) change quality estimation ( 8 ) , but shifts the focus from code defects to review burden. Our focus on agent-authored PRs extends these insights to autonomous coding agents, where non-deterministic changes ( 3 ; 16 ) differ from traditional bot automation ( 10 ) . We target review effort (comment/review volume) rather than latency (time-to-merge; prior work shows weak correlations between PR size and merge latency ( 9 ) , while our effort metric [review/comment count] exhibits stronger size correlation, suggesting different constructs); this choice reflects maintainer attention cost directly. We ask whether static creation-time features (patch size, file types) suffice for zero-latency governance, and observe a bimodal outcome pattern (instant merge vs iterative failure) that contrasts with gradual review distributions reported for human PRs ( 1 ) . 2. Methodology We use the AIDev dataset v1.0 ( 11 ) : 33,707 agent-authored PRs from 2,807 repositories ( > 100 stars), identified via AIDev metadata (type=’Bot’) plus generative agent names (Codex, Claude, Devin, Copilot), excluding deterministic bots. Manual audit confirmed 94% precision. We extract 35 features across Intent, Context, and Complexity at two stages: T0 (Creation-Time) captures signals available at PR submission (Complexity: additions , deletions , changed_files , entropy ; Intent: body_length , and has_plan —a boolean flag detected via regex matching keywords like "plan:" or "steps:", validated with 91% precision); Context: language , agent , file types). T1 (Pre-Review) adds CI status and bot comments. We frame triage as binary classification targeting High Cost PRs (see Table 1 ); sensitivity shows 99% label agreement excluding bots. Effort correlation is dominated by size ( r\approx 0.6 ), but significant residual signals exist for touches_tests and has_plan . Using lightGBM, we achieve AUC 0.958 , establishing a strong baseline for automated triage. Table 1. Operational Definitions of Target Variables Target Definition High Cost Top 20% of PRs by Effort Score (Sum of all reviews and comments, including bot messages; we validated robustness by recomputing effort excluding bot messages, achieving 99% label agreement, confirming minimal leakage risk) in the training set. True Ghosting PR Status = Rejected AND Received Human Feedback AND No follow-up commit >14 days after feedback. 2.1. Label Audit We analyzed 2,364 PRs with human feedback: overall ghosting rate is 3.8%. The gap from prior estimates reflects our strict definition requiring clear evidence of abandonment (inactive > 14d after feedback). Per-agent details in Table 2 . Table 2. Per-Agent Statistics: Scale, Speed, and Abandonment (Ghosting % conditioned on Rejected+Feedback). Agent Total PRs Instant % Ghosting % Codex ( 4 ) 21,799 42.9 10.0 Claude 3.5 ( 2 ) 523 2.9 3.1 Devin ( 5 ) 4,827 1.0 0.9 GitHub Copilot ( 14 ) 5,017 0.1 2.3 Figure 1. Label Audit: ECDF of time from feedback to close. ECDF. 3. Results and Analysis 3.1. RQ1: Predictability of Effort Table 3 demonstrates that high-cost PRs are predictable at creation time using static signals. Our model consistently achieves AUC > 0.95 on chronological splits, significantly outperforming text-based approaches like CodeBERT (AUC 0.52), which struggle to distinguish effort based on description summary alone. This stark gap confirms that for triage, structural footprint metrics (files, lines, entropy) are far more reliable signals than author intent. Even when tested across different repositories (repo-disjoint split), the model retains strong discrimination (AUC \approx 0.83), validating that the "complexity fingerprint" of high-effort PRs is universal rather than repo-specific. At a 20% review budget (i.e., reviewing only the top 20% of PRs ranked by predicted effort), we successfully intercept over two-thirds of the most expensive PRs. Table 3. Model Performance (AUC and PR-AUC): From Baselines to SOTA. Splits: Temporal = chronological 80/20; Repo-Disjoint = agent/repo separated. Size-Only AUC drops (0.93 \to 0.65) on repo-disjoint, confirming temporal confounders. Patch-tokens (TF-IDF on file paths) yield AUC 0.75; combined with text (AUC 0.80) still trails structural model (0.83), validating core claim. 95% CIs via bootstrap ( 6 ) . Model Features AUC PR-AUC Baselines (Repo-Disjoint): TF-IDF (0.57), Patch Tokens (0.75), CodeBERT (0.52), Size-Only (0.65) LightGBM (Temporal Split) T0 0.9571 0.8812 Size-Only (Temporal Split) \log(total\_changes) 0.9330 0.8700 LightGBM (Repo-Disjoint) T0 0.8345 0.8719 Stacking Ensemble (Repo-Disjoint) T0 0.8342 0.8747 Why this matters. The dominance of structural features over semantics (CodeBERT) suggests maintainers can rely on "cheap" metadata for triage. We manually inspected false negatives and found a pattern of “silent abandonment”: small PRs that look safe (no CI touches) but stall because the agent cannot handle subjective feedback. This implies

Read the complete text