TAM-Eval : Evaluating LLMs for Automated Unit Test Maintenance
Elena Bruches 4 1 , Vadim Alperovich 2 1 , Dari Baturova 4 , Roman Derunets 4 , Daniil Grebenkin 4 , Georgy Mkrtchyan 2, Oleg Sedukhin 4 , Mikhail Klementev 4 , Ivan Bondarenko 3 , Nikolay Bushkov 2 , Stanislav Moiseev 2 · 2026
Read TAM-Eval : Evaluating LLMs for Automated Unit Test Maintenance by Elena Bruches 4 1 , Vadim Alperovich 2 1 , Dari Baturova 4 , Roman Derunets 4 , Daniil Grebenkin 4 , Georgy Mkrtchyan 2, Oleg Sedukhin 4 , Mikhail Klementev 4 , Ivan Bondarenko 3 , Nikolay Bushkov 2 , Stanislav Moiseev 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18241.txt.
Introduction
Elena Bruches and a large team present TAM-Eval, a benchmark for AI that maintains unit tests. Their 2026 preprint notes a gap in software research. Language models show promise across software engineering, yet test work has mostly meant writing new tests in isolation. Real projects also need old tests maintained as the code shifts. A test suite that is not maintained rots fast and starts to mislead. TAM-Eval measures how well models handle that upkeep.
Maintenance includes fixing broken tests and updating them for new behavior. It is unglamorous work that eats real engineering time. A shared benchmark can steer models toward it. The work extends evaluation from creation to care.
Catalog record
| CREATOR | Elena Bruches 4 1 , Vadim Alperovich 2 1 , Dari Baturova 4 , Roman Derunets 4 , Daniil Grebenkin 4 , Georgy Mkrtchyan 2, Oleg Sedukhin 4 , Mikhail Klementev 4 , Ivan Bondarenko 3 , Nikolay Bushkov 2 , Stanislav Moiseev 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18241
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TAM-Eval : Evaluating LLMs for Automated Unit Test Maintenance Elena Bruches 4 1 , Vadim Alperovich 2 1 , Dari Baturova 4 , Roman Derunets 4 , Daniil Grebenkin 4 , Georgy Mkrtchyan 2, Oleg Sedukhin 4 , Mikhail Klementev 4 , Ivan Bondarenko 3 , Nikolay Bushkov 2 , Stanislav Moiseev 2 Abstract While Large Language Models (LLMs) have shown promise in software engineering, their application to unit testing remains largely confined to isolated test generation or oracle prediction, neglecting the broader challenge of test suite maintenance. We introduce TAM-Eval ( T est A utomated M aintenance Evaluation), a framework and benchmark designed to evaluate model performance across three core test maintenance scenarios: creation, repair, and updating of test suites. Unlike prior work limited to function-level tasks, TAM-Eval operates at the test file level, while maintaining access to full repository context during isolated evaluation, better reflecting real-world maintenance workflows. Our benchmark comprises 1,539 automatically extracted and validated scenarios from Python, Java, and Go projects. TAM-Eval supports system-agnostic evaluation of LLMs, using a reference-free protocol based on test suite pass rate, code coverage, and mutation testing. Empirical results indicate that state-of-the-art LLMs have limited capabilities in realistic test maintenance processes and yield only marginal improvements in test effectiveness. We release TAM-Eval as an open-source framework to support future research in automated software testing. Our data and code are available at https://github.com/trndcenter/TAM-Eval . I Introduction Unit testing plays a pivotal role in ensuring robustness and reliability by validating individual components of software [ 18 ] , with industry reports estimating that testing consumes up to 25% of total project budgets [ 14 ] . Recent advances in large language models (LLMs) have significantly impacted software engineering tasks, including code generation [ 20 ] , test generation [ 39 ] , documentation [ 11 ] , refactoring [ 9 ] , and bug fixing [ 24 ] . Despite strong performance in function-level test generation, existing LLM-based tools largely produce isolated tests or assertions and do not address the broader lifecycle of real-world test suites. In practice, however, software quality depends heavily on test maintenance — the continuous generation, repair, and updating of tests as code evolves. Failing to maintain tests leads to outdated suites, broken pipelines, and undetected regressions. Yet, this setting remains underexplored in current research. Technical Challenges. Evaluating automated test maintenance requires context-aware reasoning over both production code and existing tests, as well as dynamic execution environments that support correctness checking and incremental evaluation across dimensions such as execution correctness, coverage and mutation score. Fig. 1: Overview of TAM-Eval, a framework and benchmark for evaluating LLMs on unit test maintenance. The pipeline filters GitHub repositories, constructs tasks for test creation, repair, and update with execution-based validation, performs LLM-driven test generation with iterative feedback, and computes Pass Rate, \Delta Test Coverage, and \Delta Mutation Score. Our Contributions. We present TAM-Eval ( T est A utomated M aintenance E valuation), a benchmark and evaluation framework for comprehensive unit test maintenance assessment. An overview of its pipeline is shown in Fig. 1 Its key contributions are as follows: • Test Maintenance Scenarios: The novelty of TAM-Eval is that it addresses the complete test maintenance cycle — generation , repair , and update of test suites at the granularity of entire test files, reflecting realistic developer workflows. • Curated Real-world Dataset: The benchmark includes 1,539 validated scenarios from actively maintained open-source repositories, which were selected via a multi-stage filtering pipeline to ensure quality with measures to minimize data contamination. • Unified and Feedback-Aware Evaluation: All tasks are prompted through a unified prompt template with minimal instructions and iterative execution feedback, testing the model’s ability to generalize across maintenance types and simulate practical workflows. • Extensible, Open Evaluation Suite: TAM-Eval supports reference-free evaluation via test pass rate, coverage, and mutation testing. It is language-agnostic, easy to extend, and fully open-sourced to facilitate reproducibility and community benchmarking. • Empirical Analysis of LLMs: We evaluate advanced SOTA (state-of-the-art) LLMs, revealing that even the best-performing models exhibit large variability across tasks and languages, with mutation coverage improvements rarely exceeding 12 percentage points, indicating substantial room for advancement. This highlights the need for developing more context-aware methods and distilling supervision from verifiers such as compilers, mutation, and coverage signals. Outline. Section II reviews relevant work. Section III presents our benchmark design. Section IV details the evaluation framework. Section V describes the experimental setup. Section VI reports our experimental results. II Background II-A Unit tests generation The attempts to automate the unit testing process started in the last century [ 25 ] and focused on code coverage, such as Randoop [ 31 ] or EvoSuite [ 12 ] . With the rise of LLMs, various prompting techniques [ 13 ] were used to generate unit tests. Whereas earlier models had difficulties with self-repair and test improvement [ 17 , 28 ] , such methods have received a second wind with the advent of reasoning approaches [ 35 , 40 ] . For example, ChatTester [ 41 ] leverages ChatGPT itself to improve the quality of its generated tests. [ 13 ] Libro [ 22 ] focuses on post-processing steps that help discern when LLMs are effective and rank the produced tests according to their validity. Rapid development in code generation and particularly in test generation has led to the need to create high-quality assessment systems for different test-related tasks like test repairing or test updating. II-B Unit tests repairing One of the important tasks associated with unit tests is the task of test repair, a process that helps validate the code and improve its performance. The frameworks and methods like TestART [ 16 ] , ChatUniTest [ 6 ] , already mentioned ChatTester [ 41 ] and others [ 15 , 32 ] use the APR (Automated Program Repair) idea, but the proposed pipelines also contain the repair stage, which can be rule-based, LLM-based, or combined. The rule-based repair strategy can include steps like syntax analysis and searching test annotations for test extraction, copying imports from focal file (tested file) to test file [ 6 ] , mutations of different entities to increase test correctness and robustness [ 16 ] . However, some research shows that in spite of the fact that repair can be a part of the test generation pipeline, these tasks require a different set of LLM skills [ 27 ] . Nevertheless, the use of different LLM strategies and prompt engineering allows the creation of architectures like TestGen-LLM [ 1 ] , which was fine-tuned to increase test’s coverage and reliability. II-C Unit tests updating Unit tests need to be periodically updated and maintained: they may not initially cover all the necessary functionality or be unstable to mutations, for which purpose systems are developed that can improve existing tests, such as Meta’s TestGen-LLM tool [ 1 ] . The solution of this issue may be associated with using commits that modify the tests. Application of the corresponding concept can be found in the dataset with vulnerabilities of open-source Android projects, collected on the basis of commits with vulnerabilities and fix-commits [ 5 ] . The GitBug-Actions benchmark also used a buggy commit and a subsequent