Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories

Arsham Khosravani, Audris Mockus · 2026

Read Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories by Arsham Khosravani, Audris Mockus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24429.txt.

Introduction

Khosravani and Mockus count how much open-source code AI agents are actually writing, and find the usual yardstick misses most of it. Generative coding agents leave often hidden traces across repositories. Their multi-layered detection framework scans configuration files, commit messages, author-identity patterns, and bot signatures. It runs across the World of Code infrastructure, spanning 180 million Git repositories.

No single method captures more than a fraction of the activity. Multi-method detection finds hundreds of thousands of Claude Code commits in one snapshot. Bot account lookup, the signal adoption studies lean on most, recovers only a thirtieth of them, a thirtyfold recall gap. Every pattern is hand-validated. Tracking snapshots from late 2024 into spring 2026, such agents generate hundreds of thousands of commits monthly, led by Claude Code across tens of thousands of projects. Single-signal prevalence estimates, the authors warn, are biased badly low.

Catalog record

CREATORArsham Khosravani, Audris Mockus
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24429

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories Arsham Khosravani, Audris Mockus Abstract Generative AI coding agents are rapidly entering the open-source software supply chain, yet their diverse and often invisible traces leave their true prevalence poorly understood. We introduce a multi-layered detection framework that integrates configuration-file scanning, commit-message analysis, author-identity pattern matching, and bot-signature lookup across the World of Code (WoC) infrastructure, spanning more than 180 million Git repositories. Our four-type taxonomy shows that no single detection method captures more than a fraction of total AI activity. Multi-method detection identifies 850,157 Claude Code commits in the V2510 snapshot, of which bot-account lookup, the signal most adoption studies rely on, recovers only 28,154 (3.3%): a 30 × 30\times relative-recall gap. Absolute recall is unknown; the union is a lower bound, so single-signal prevalence estimates are biased low by at least this factor. Every detection pattern is hand-validated (495 labels) with per-cell precision and Wilson confidence intervals. Applying the framework across snapshots from December 2024 through April 2026, we find that commit-attributed agents collectively generate over 320,000 commits per month by V2604. Claude Code leads with 886,122 commits across 17,295 projects, followed by Jules with 215,804 commits. The Type D configuration-file census further establishes Claude as the dominant silent agent in V2604, appearing in 21,078 projects (888,177 blob occurrences), a category entirely absent in V2412. Adoption timing is bimodal: projects either integrate agents at inception (born-with-AI) or after years of development. Comparing against an independent pull-request-based census (AIDev), we find the two channels capture nearly disjoint agent populations: Codex is the largest agent by pull requests but near-absent from commits, while Claude Code is the largest by commits but near-absent from pull requests, so a PR-based census misses 79% of commit-detected Claude Code adopters and a commit-based census misses essentially all Codex adopters. No single detection channel is representative—and the channels differ in kind , not just population: via AIDev’s task labels, PR-deployed cloud agents (Codex, Cursor) surface as feature work while commit-deployed in-editor agents (Claude Code, OpenHands, Aider) surface as maintenance, so the observed agent work profile follows deployment and detection mode rather than the tool itself. 1 Introduction Generative AI (GenAI) coding agents, tools that autonomously generate, modify, or review source code, are increasingly embedded in open-source software (OSS) development workflows. Tools such as GitHub Copilot [ 26 ] , Cursor [ 6 ] , Replit [ 27 ] , and OpenHands [ 31 ] now participate in activities ranging from autocompletion to autonomous pull-request generation. Understanding how widely these agents are adopted and what effect they have on development outcomes is important for at least three reasons. First, from a software engineering perspective , quantifying AI-assisted contributions enables researchers to study how GenAI tools affect development effort, code quality, and project sustainability [ 17 ] . Without reliable detection, any impact study conflates human and machine contributions. Second, from a software supply-chain perspective, AI-generated code introduces distinct risk profiles. It may embed subtle vulnerabilities, propagate biased training-data patterns, or create negative feedback loops where generated code degrades the training corpus of future models [ 18 ] . Identifying AI-generated contributions is therefore a prerequisite for supply-chain auditing. Third, from an ecosystem measurement perspective, the explosive growth of coding agents documented in 2025 [ 28 ] demands baseline prevalence estimates grounded in repository-level evidence rather than self-reported surveys. Despite this need, detection is non-trivial. Agents differ dramatically in the traces they leave: some operate through dedicated bot accounts, others embed signatures in commit messages, and many are entirely “silent,” modifying code under the developer’s own identity with no distinguishing marker. A detection strategy that relies on any single signal will systematically undercount adoption, and prior censuses rely on exactly one signal (pull requests or configuration files). These gaps frame three research questions: RQ1 (prevalence). How prevalent are AI coding agents across the OSS ecosystem, how much do single-signal methods miss, and do different detection channels see the same agents and the same kinds of work? RQ2 (adoption timing). When in a project’s lifetime are agents adopted? RQ3 (velocity association). Does adoption coincide with changes in development velocity? RQ1 demands the multi-method detection our taxonomy provides; RQ3 is deliberately descriptive, establishing association rather than a controlled causal estimate. This paper makes three contributions that answer these questions: (1) A detection taxonomy (Types A–D) that classifies AI coding agents by the traces they leave in version-control history, together with a multi-layered detection framework implemented on the World of Code infrastructure [ 21 ] . (2) A prevalence census of twelve agents across 180M+ repositories and three snapshots (December 2024 to April 2026), revealing that silent and distributed agents account for a substantial share of AI activity invisible to prior file-based and PR-based surveys. A direct comparison with an independent PR-based census (AIDev [ 19 ] ) shows the commit and PR channels capture nearly disjoint agent populations and different kinds of work—feature generation in the PR channel, maintenance in the commit channel—so the observed work profile follows deployment and detection mode, not the tool, and no single-channel census is representative. (3) A hand-labeled validation of every cell in the detection grid (495 labels) with per-cell precision and Wilson confidence intervals, a construct-validity analysis separating agent-authored from agent-trailered commits, and a documented audit trail of detector corrections. 2 Related Work Adoption surveys. Robbes et al. [ 28 ] provide the most comprehensive adoption study to date, documenting the rapid uptake of coding agents on GitHub during early 2025 by tracking bot-authored pull requests. Their methodology is PR-centric, capturing Type A agents (centralized bot accounts) but missing silent tools (Type D) and distributed-attribution agents (Type C). Li et al. [ 19 ] likewise build a large PR-based census (AIDev, 932,791 agent-authored pull requests); we compare against it directly (Section 4 ) and show that the PR channel and our commit channel see nearly disjoint agent populations. He et al. [ 17 ] use repository configuration files ( .cursorrules ) as an adoption proxy for Cursor, which corresponds to a single Type D signal in our taxonomy. Our framework generalizes this approach. The same .cursorrules proxy recovers Cursor’s 28,909 configuration projects (Table 1 ), but taken alone it would report zero commit-level activity and would entirely miss Cursor’s pull-request channel (32,941 PRs in AIDev; Table 6 ). Conversely, it is blind to commit-channel agents such as Claude Code. The single-signal proxy is thus correct but partial. We quantify exactly what the additional detection tiers contribute. A 2023 GitHub developer survey [ 12 ] reports that 92% of surveyed developers use AI coding tools, but self-reported surveys are susceptible to social-desirability bias and cannot quantify actual commit-level contributions. Each of these is a single -channel measurement; None of these prior approaches establishes how much the other channels miss, or whether the different detection channels even observe the same agents and activity. This is precisely the representativeness gap that RQ1 targets. Adoption timing (RQ2) is likewise unmeasured at ecosystem scale. Bot detection in software repositories. The problem of identifying non-human contributors in version control predates GenAI. Dey et al. [ 7 ] developed methods to detect and characterize bots

Read the complete text