Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793620 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering Integrated and visual development environments CCS: Software and its engineering Automatic programming CCS: Software and its engineering Software evolution CCS: Software and its engineering Maintaining software
Suzuka Yoshimoto † \dagger , Shun Fujita † \dagger , Kosei Horikawa † \dagger , Daniel Feitosa § \lx@sectionsign , Yutaro Kashiwa † \dagger , Hajimu Iida † \dagger · 2026
Read Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793620 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering Integrated and visual development environments CCS: Software and its engineering Automatic programming CCS: Software and its engineering Software evolution CCS: Software and its engineering Maintaining software by Suzuka Yoshimoto † \dagger , Shun Fujita † \dagger , Kosei Horikawa † \dagger , Daniel Feitosa § \lx@sectionsign , Yutaro Kashiwa † \dagger , Hajimu Iida † \dagger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13724.txt.
Introduction
Suzuka Yoshimoto and co-authors study test generation by AI agents in real-world software development. Agent-based coding tools autonomously create, modify, and execute code, including tests, inside repositories. Little was known about how they generate tests or how those tests compare with human-written ones. Using the AIDev dataset, the team extracted 2,232 commits with test-related changes. AI authored about a sixth of all commits adding tests in real repositories. AI test methods were longer, with denser assertions and lower cyclomatic complexity, and coverage gains matched human tests.
Catalog record
| CREATOR | Suzuka Yoshimoto † \dagger , Shun Fujita † \dagger , Kosei Horikawa † \dagger , Daniel Feitosa § \lx@sectionsign , Yutaro Kashiwa † \dagger , Hajimu Iida † \dagger |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21731 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage Conference: 23rd International Conference on Mining Software Repositories; April 13–14, 2026; Rio de Janeiro, Brazil 23rd International Conference on Mining Software Repositories (MSR ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793302.3793620 ISBN: 979-8-4007-2474-9/2026/04 CCS: Software and its engineering Integrated and visual development environments CCS: Software and its engineering Automatic programming CCS: Software and its engineering Software evolution CCS: Software and its engineering Maintaining software Suzuka Yoshimoto † \dagger , Shun Fujita † \dagger , Kosei Horikawa † \dagger , Daniel Feitosa § \lx@sectionsign , Yutaro Kashiwa † \dagger , Hajimu Iida † \dagger Abstract . Agent-based coding tools have transformed software development practices. Unlike prompt-based approaches that require developers to manually integrate generated code, these agent-based tools autonomously interact with repositories to create, modify, and execute code, including test generation. While many developers have adopted agent-based coding tools, little is known about how these tools generate tests in real-world development scenarios or how AI-generated tests compare to human-written ones. This study presents an empirical analysis of test generation by agent-based coding tools using the AIDev dataset. We extracted 2232 2232 commits containing test-related changes and investigated three aspects: the frequency of test additions, the structural characteristics of the generated tests, and their impact on code coverage. Our findings reveal that (i) AI authored 16.4% of all commits adding tests in real-world repositories, (ii) AI-generated test methods exhibit distinct structural patterns, featuring longer code and a higher density of assertions while maintaining lower cyclomatic complexity through linear logic, and (iii) AI-generated tests contribute to code coverage comparable to human-written tests, frequently achieving positive coverage gains across several projects. † † cc-license: by 1. Introduction Software testing is an essential activity that forms the foundation of quality assurance ( 6 ; 20 ; 18 ) . Code lacking adequate tests cannot only delay bug detection but also increase the risk of regressions and undermine developers’ confidence in making changes ( 38 ; 21 ) . Furthermore, low test coverage can discourage code modifications and extensions, thereby hindering the evolution of projects ( 19 ; 34 ) . To avoid such problems, developers are encouraged to write tests in parallel with feature implementation; however, due to time constraints and the tedious nature of test creation, sufficient tests are often not written in practice ( 33 ; 7 ) . To address this, automated test generation has evolved from traditional techniques like symbolic execution ( 5 ) to recent approaches leveraging Large Language Models (LLMs) ( 15 ) . More recently, the mainstream approach has been shifting from single-prompt-based methods to approaches utilizing AI agents. In agent-based approaches, by understanding the context of the entire repository and autonomously repeating cycles of test creation, execution, and modification, more sophisticated test generation becomes possible ( 14 ) . Although existing studies have evaluated the quality and coverage of LLM-generated tests in controlled experimental environments ( 32 ) , it is not sufficiently understood to what extent agent-based tools generate tests in actual development settings and how the generated tests differ from those created by humans. This represents an important research challenge for evaluating the practicality of agent-based test generation technology. This study aims to quantitatively analyze the reality of test generation by agent-based coding tools. Specifically, we extracted commits containing test-related changes from the AIDev dataset ( 22 ) and investigated the frequency of test generation, characteristics, and impact on coverage in those commits. We identified 2232 commits and compared test methods generated by AI with those created by humans, thereby conducting a multifaceted analysis of the effects of test generation by agent-based tools. 2. Data Collection To identify projects suitable for our analysis, we applied the following steps using the AIDev dataset ( 22 ) . 1. Language Filtering : We analyzed the AIDev dataset to identify the languages with the most AI-authored pull requests (PRs). We found TypeScript to be one of the most common languages across all agents (22.4% of PRs) and, thus, selected it for this study. 2. Framework Filtering : To measure coverage systematically (for RQ3), we decided to standardize the test framework use to make this study feasible. In particular, we investigated the 2,807 unique repositories from the AIDev dataset that contain PRs, finding that 650 were identified as TypeScript projects, of which 400 had detectable test frameworks. Among these, vitest was found to be the most prevalent, used in 179 projects (44.8% of projects with test frameworks). Given vitest ’s dominance in the TypeScript ecosystem within our dataset and its straightforward coverage measurement capabilities, we decided to target vitest projects for our analysis. 3. Project Identification : Next, focusing on the identified vitest projects, we returned to the AIDev dataset to find those with a high volume of AI-authored test additions. We inspected the patch information of commits within AI-authored PRs. We filtered for PRs that contained new test method definitions (e.g., describe() , it() ) on added lines (prefixed with + ). Final Dataset: We selected the top 10 projects with the highest AI-authored test activity to ensure a focus on the most significant contributions. This sample size was determined by the technical complexity of dynamic analysis; measuring test coverage (RQ3) requires manual environment setup and dependency resolution for each repository, making a larger-scale execution impractical. Table 1. Prevalence of AI-Generated Test Addition Commits in Selected Projects. Project Test-adding Commits All AI-authored (%) cal.com ( 8 ) 894 129 (14.4%) azure-sdk-for-js ( 4 ) 366 7 (1.9%) liam ( 23 ) 352 43 (12.2%) appkit ( 31 ) 257 29 (11.3%) helper ( 2 ) 123 13 (10.6%) primer-react ( 29 ) 89 31 (34.8%) alchemy ( 1 ) 76 54 (71.1%) total-typescript-monorepo ( 25 ) 26 26 (100.0%) llmgateway ( 35 ) 25 14 (56.0%) capgo ( 9 ) 24 20 (83.3%) 3. Research Questions RQ_{1} : How often do developers add tests with Agent Coding Tools? Motivation. Understanding the prevalence of AI-generated test contributions is the first step in assessing their overall impact. We examine the proportion of test-added commits among all commits made by AI agents to gauge their activity level in this specific task. Approach. We analyzed 10 selected TypeScript projects to quantify the contribution of AI agents to test code generation. Our methodology focuses on identifying commits where AI agents explicitly introduced new test methods. The process consists of four steps: 1. Defining the Temporal Scope. We determined the analysis period for each project based on the available data in the AIDev dataset. By examining the pull_request table, we extracted the minimum and maximum created_at timestamps for each repository. This defined interval served as the boundary for our data collection, ensuring that our commit analysis corresponds precisely to the period of AI activity captured in the dataset. 2. Commit Retrieval and Filtering. We utilized the GitHub API to retrieve all commits associated with pull requests within the defined period. To isolate substantive code contributions, we filtered this set to exclude merge commits. Furthermore, we narrowed our focus to commits that modified at least one test file, as these are the potential candidates for test additions. 3. Detecting