Testing Framework Migration with Large Language Models Conference: 7th ACM/IEEE International Conference on Automation of Software Test (AST 2026); April 13–14, 2026; Rio de Janeiro, Brazil 7th ACM/IEEE International Conference on Automation of Software Test (AST 2026) (AST ’26), April 13–14, 2026,
Altino Alves, João Eduardo Montandon, Andre Hora · 2026
Read Testing Framework Migration with Large Language Models Conference: 7th ACM/IEEE International Conference on Automation of Software Test (AST 2026); April 13–14, 2026; Rio de Janeiro, Brazil 7th ACM/IEEE International Conference on Automation of Software Test (AST 2026) (AST ’26), April 13–14, 2026, by Altino Alves, João Eduardo Montandon, Andre Hora (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02964.txt.
Introduction
Migrating Python test suites between frameworks with language models is tested by Altino Alves and 2 co-authors in a 2026 paper. Many projects want to move from unittest to Pytest for simpler assertions and reusable fixtures. Doing that migration by hand is slow, so automation could modernize suites faster.
The authors built a dataset of real migrations from the top hundred Python open-source projects. They evaluated two leading models under three prompting strategies and two temperature settings. Generated migrations were actually run in their test suites. Slightly under half passed, so models can accelerate migration but with caveats. One model migrated conservatively, keeping class-based tests. The other transformed more aggressively toward function-based tests.
Catalog record
| CREATOR | Altino Alves, João Eduardo Montandon, Andre Hora |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02964
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Testing Framework Migration with Large Language Models Conference: 7th ACM/IEEE International Conference on Automation of Software Test (AST 2026); April 13–14, 2026; Rio de Janeiro, Brazil 7th ACM/IEEE International Conference on Automation of Software Test (AST 2026) (AST ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793654.3793749 ISBN: 979-8-4007-2476-3/2026/04 CCS: Software and its engineering Software testing and debugging Altino Alves, João Eduardo Montandon, Andre Hora Abstract . Python developers rely on two major testing frameworks: unittest and Pytest . While Pytest offers simpler assertions, reusable fixtures, and better interoperability, migrating existing suites from unittest remains a manual and time-consuming process. Automating this migration could substantially reduce effort and accelerate test modernization. In this paper, we investigate the capability of Large Language Models (LLMs) to automate test framework migrations from unittest to Pytest . We evaluate GPT 4o and Claude Sonnet 4 under three prompting strategies (Zero-shot, One-shot, and Chain-of-Thought) and two temperature settings (0.0 and 1.0). To support this analysis, we first introduce a curated dataset of real-world migrations extracted from the top 100 Python open-source projects. Next, we actually execute the LLM-generated test migrations in their respective test suites. Overall, we find that 51.5% of the LLM-generated test migrations failed, while 48.5% passed. The results suggest that LLMs can accelerate test migration, but there are often caveats. For example, Claude Sonnet 4 exhibited more conservative migrations (e.g., preserving class-based tests and legacy unittest references), while GPT-4o favored more transformations (e.g., to function-based tests). We conclude by discussing multiple implications for practitioners and researchers. † † cc-license: by 1. Introduction Software testing is fundamental to avoiding regressions and catching bugs. Currently, Python developers can rely on two main testing frameworks: unittest ( 55 ) and Pytest ( 41 ) . Pytest provides some advantages compared to unittest, including simpler assertions, reuse of fixtures, and interoperability ( 41 ; 4 ) . Multiple Python projects—such as Pandas, NumPy, ScikitLearn, Requests, and Flask—have migrated to or are currently migrating to Pytest due to these benefits ( 4 ) . For example, Figure 1 presents a migration from unittest to Pytest in pyvim. 1 1 1 https://github.com/prompt-toolkit/pyvim/commit/7e1c7bfb505cefba468 This migration consists of three major modifications. The first chunk shows that the unittest setUp method is being split into four @Pytest.fixtures properties. The second and third chunks modify test methods to receive these fixtures and use assert statements during the test verification. Section 2 explains this migration in more detail. Figure 1 . Migration from unittest to Pytest (pyvim). Migration from unittest to Pytest (pyvim). Pytest allows projects to run tests written in unittest, thus enabling a gradual migration process. However, this migration can be time-consuming given the complexity of test suites: a recent study found that some projects may take months or years to conclude the migration or simply never conclude it ( 4 ) . Besides, using two different testing frameworks at the same time can increase the effort needed to maintain the test suite. Such projects could benefit from an automated solution to assist developers in migrating these tests faster and effectively. Recently, Large Language Models (LLMs) have been evaluated in multiple software engineering tasks, including source code and tests generation, bug fix, code smells detection, and to support the code review process ( 40 ; 20 ; 38 ; 36 ; 54 ; 22 ; 27 ; 26 ; 52 ; 16 ) . Other studies investigated the use of LLMs to support API migration ( 1 ; 29 ) . However, to the best of our knowledge, no study has explored the migration of the test frameworks. In this paper, we investigate the capability of LLMs to assist developers during the migration of test frameworks. For this purpose, we built a dataset with over 900 real-world migrations from unittest to Pytest , manually implemented by developers ( 3 ) . Next, we selected a subset of migrated tests that still exist by September 2025, and thus remain executable. This process resulted in a final set of 40 isolated migrations that are investigated in this paper. In our experiment, we rely on GPT-4o and Claude Sonnet 4 to migrate Python tests from unittest to Pytest . We applied three prompting strategies—Zero-shot, One-shot, and Chain-of-Thought, under two temperature settings (0.0 and 1.0), generating a total of 480 migration variants. To ensure the migration proposed by the LLMs was correct, we reproduced the test environment of each migration and actually executed both LLM-migrated and developer-migrated tests. We then compared the results of both executions in terms of test correctness (i.e., test was successfully executed) and test coverage (i.e., test coverage remained the same). Both models achieved an overall correctness rate of 48.54%, while successful migrations maintained the same test coverage, indicating that LLMs did not modify runtime behaviour. However, 25 out of 40 migrations presented at least one failing configuration (247 total failures), mostly related to dependency handling, fixture adaptation, or setup inconsistencies. While Claude Sonnet 4 preserved class-based structures and unittest patterns, GPT-4o favored function-based rewrites aligned with Pytest , revealing distinct yet complementary migration styles. We conclude by discussing implications for researchers and practitioners. \noindentparagraph Contribution. The contributions of this paper are twofold: (1) we provide the first study to explore test migration with LLMs and (2) we discuss multiple implications for researchers and practitioners. 2. Background and Motivation Unittest ( 55 ) and Pytest ( 41 ) are the most popular testing frameworks in Python ( 30 ) . Unittest belongs to the Python standard library, while Pytest is a third-party testing framework. Unittest relies on classes and inheritance to create tests (i.e., the test class needs to extend the unittest class TestCase ), whereas Pytest tests can be regular functions, with the test prefix. Consequently, Pytest tests tend to be less verbose than unittest ones. Another difference is the assertions: unittest provides self.assert* methods (e.g., assertEqual , assertTrue , etc.), while Pytest allows developers to use the regular Python assert statement for verifying expectations and values. There are many other differences; for example, Pytest facilitates the creation of parameterized tests and the reuse of fixtures. Due to the advantages of Pytest, many Python projects have migrated to this framework. A prior study discovered that 27% of top-100 most popular Python projects migrated or were migrating to Pytest ( 4 ) . To gain more insights into the relevance of this problem, we replicated this study in the same set of popular projects, found that the migration rate has increased to 37%. Migrating from unittest to Pytest may involve at least the following major steps: (1) removing test from class and moving to regular functions; (2) replacing assertions with Pytest asserts; and (3) moving setup/teardown operations to Pytest fixtures. Steps 1 and 2 is relatively simple to apply because the migration between unittest and Pytest is almost direct. For example, the test test_vsplit in Figure 1 only replaces the unittest assertions assertIsInstance and assertEqual by the Pytest assert statement. On the other hand, migrating the remaining steps is harder to accomplish because there is no direct mapping between unittest and Pytest. For example, the unittest setUp method is split into four Pytest fixture functions in Figure 1 : prompt_buffer