Evaluating LLM’s Effectiveness in Detecting and Correcting Test Smells: An Empirical Study PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE

Enio G. Santana Jr, Jander Pereira Santos Junior, Erlon P. Almeida, Iftekhar Ahmed, Paulo Anselmo da Mota Silveira Neto, and Eduardo Santana de Almeida · 2025

Read Evaluating LLM’s Effectiveness in Detecting and Correcting Test Smells: An Empirical Study PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE by Enio G. Santana Jr, Jander Pereira Santos Junior, Erlon P. Almeida, Iftekhar Ahmed, Paulo Anselmo da Mota Silveira Neto, and Eduardo Santana de Almeida (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07594.txt.

Introduction

Santana and colleagues test in twenty twenty-five whether large language models can find and fix test smells, signs of sloppy test code. Smelly tests are hard to maintain and less trustworthy. Existing tools mostly flag them without fixing them, so the team tried three top models on Python and Java test suites, using standard detectors to seed the task.

One model led on detection accuracy in both languages, while another trailed. All three could rewrite smelly tests, but quality varied and fixes sometimes added new smells. Only the leading model also raised test coverage after rewriting. The authors see real promise for automated cleanup, with care needed by language and smell type.

Catalog record

CREATOREnio G. Santana Jr, Jander Pereira Santos Junior, Erlon P. Almeida, Iftekhar Ahmed, Paulo Anselmo da Mota Silveira Neto, and Eduardo Santana de Almeida
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07594

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating LLM’s Effectiveness in Detecting and Correcting Test Smells: An Empirical Study PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE Enio G. Santana Jr, Jander Pereira Santos Junior, Erlon P. Almeida, Iftekhar Ahmed, Paulo Anselmo da Mota Silveira Neto, and Eduardo Santana de Almeida Abstract Test smells indicate poor development practices in test code, reducing maintainability and reliability. While developers often struggle to prevent or refactor these issues, existing tools focus primarily on detection rather than automated refactoring. Large Language Models (LLMs) have shown strong potential in code understanding and transformation, but their ability to both identify and refactor test smells remains underexplored. We evaluated GPT-4-Turbo, LLaMA 3 70B, and Gemini-1.5 Pro on Python and Java test suites, using PyNose and TsDetect for initial smell detection, followed by LLM-driven refactoring. Gemini achieved the highest detection accuracy (74.35% Python, 80.32% Java), while LLaMA was lowest. All models could refactor smells, but effectiveness varied, sometimes introducing new smells. Gemini also improved test coverage, unlike GPT-4 and LLaMA, which often reduced it. These results highlight LLMs’ potential for automated test smell refactoring, with Gemini as the strongest performer, though challenges remain across languages and smell types. I Introduction The development and maintenance of test suites are essential for ensuring software quality and reliability [ 1 , 2 ] . Well-structured test suites help verify expected behaviors, detect defects early, and support maintainability, ultimately reducing the cost of software evolution. However, test smells—sub-optimal coding patterns in test code [ 3 ] - pose significant challenges. These smells make test code harder to read, understand, and modify [ 4 ] , adding unnecessary complexity and obscuring the intent behind test cases. As a result, they increase maintenance effort and can lead to errors during future development and refactoring. To address this, automated detection tools have been developed to identify test smells and assist developers in maintaining high-quality test code [ 4 ] . By integrating these tools into development workflows (e.g., during code commits), developers can detect and fix smells early, preventing their accumulation over time [ 5 ] . Such tools are especially valuable when developers work under tight deadlines or lack testing expertise. Meanwhile, advances in Large Language Models (LLMs), such as GPT-4 [ 6 ] , LLaMA [ 7 ] , and Gemini [ 8 ] , along with NLP-based tools like CodeBERT and CodeXGLUE [ 9 , 10 ] , have shown great promise in code understanding, generation, and transformation. These models have successfully tackled tasks like code summarization, translation, and repair, suggesting that LLMs could be leveraged to detect and refactor test smells—an area that remains underexplored in research. This led us to answer our first research question, which is: RQ1: Are LLMs capable of detecting test smells in test code? To answer this question, we conducted an empirical study evaluating the capabilities of GPT-4, Gemini, and LLaMA in detecting test smells in Python and Java 1 1 1 These languages were selected due to their popularity and widespread adoption across various domains [ 11 ] . . Our study comprised over 1,400 identification tasks across 15 types of test smells in two programming languages. While identifying test smells is an important first step, detection alone is not sufficient to support developers in maintaining high-quality test suites. Developers often lack the time or expertise to address identified smells without additional guidance, and as a result, tools that only detect smells without offering solutions are less likely to be adopted in practice. To be truly useful, tools need to not only flag issues but also assist in automatically refactoring or suggesting concrete fixes. In this context, LLMs hold strong potential, as they have already shown impressive capabilities in automated code refactoring [ 12 , 13 ] and code generation [ 14 ] in non-test code scenarios. Given these successes, we posit LLMs should be able to extend their capabilities to the domain of test code and test smell refactoring, helping developers resolve test smells efficiently and encouraging broader tool adoption in real-world development workflows. This led us to answer our second research question, which is: RQ2: How effective are LLMs in refactoring identified test smells? To answer this question, we analyzed 450 refactoring tasks across 15 types of test smells in two programming languages using GPT-4, Gemini, and LLaMA. Our results indicate that all models generally performed better in Java than in Python. While Gemini exhibited substantial gains in statement coverage for Java projects, GPT-4 delivered the most consistent performance overall, with the least negative impact on missing statement coverage, especially in Python projects. Overall, this paper makes the following contributions: • We conduct a systematic evaluation of the effectiveness of LLMs, GPT-4, LLaMA, and Gemini, in identify and refactoring test smells in Python and Java test suites. • We integrate traditional detection tools (PyNose and TsDetect) with LLM-based refactoring, enabling a hybrid workflow that combines static analysis and AI-driven code transformation. • We provide a comprehensive analysis of the persistence of test smells after refactoring and assess the impact of automated refactoring on test coverage, offering a multifaceted view of refactoring effectiveness. • We identify limitations in LLM-based refactoring, including the introduction of new test smells, highlighting current challenges and opportunities for improving the robustness of AI-assisted test code transformation. • To foster replication and future research, we make all experimental artifacts, datasets, and analysis scripts available on our project website [ 15 ] . II Related Work This section describes related work on identifying and refactoring test smells, as well as studies that highlight the performance of using LLMs models in Software Tetsing. II-A LLMs in Software Testing Due to recent developments in LLM, researchers have started investigating the applicability of LLM in Software Engineering. Xinyi et al. [ 16 ] conducted a systematic review of 395 papers on LLM applications across the software development lifecycle (SDLC), showing that while 58.37% of studies focused on development tasks, only 10.30% addressed quality assurance, with common challenges including overfitting, scalability, and ambiguous code generation. Complementing this, Wang et al. [ 17 ] reviewed 102 studies specifically on LLMs in software testing, identifying test case generation and program repair as primary use cases and highlighting the importance of prompt engineering and model fine-tuning. Schäfer et al. [ 18 ] introduced TestPilot, an LLM-based JavaScript unit test generator that achieved 70.2% median statement coverage, 19 percentage points higher than existing techniques, while reducing test redundancy. The study concluded that LLMs can effectively generate unit tests without additional training, emphasizing the role of prompt design, model size, and training data in performance. Yaraghi et al. [ 19 ] proposed TaRGET, a fine-tuned LLM for automated test repair, reaching 66.1% exact-match accuracy on 45,373 broken tests. Wang et al. [ 16 ] investigated the use of LLMs, including Codex, Codegen, and ChatGPT, to generate edge-case programs for fuzzing deep learning libraries like PyTorch and TensorFlow. Their approach, FuzzGPT, uses historical bug-triggering programs to prime LLMs, outperforming traditional fuzzers by achieving higher code coverage and detecting more bugs. FuzzGPT identified 76 bugs, with 61 confirmed (49 of which were previously unknown), showcasing its ability to uncover edge-case scenarios not present in training data. Xie et al. [ 20

Read the complete text