AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models

Hengrui Xing, Cong Tian, Liang Zhao, Zhi Ma, Wensheng Wang, Nan Zhang, Chao Huang, Zhenhua Duan · 2025

Read AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models by Hengrui Xing, Cong Tian, Liang Zhao, Zhi Ma, Wensheng Wang, Nan Zhang, Chao Huang, Zhenhua Duan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05102.txt.

Introduction

Hengrui Xing and co-authors wrote a 2025 paper on AutoTestForge, a system that tests language models with little hand work. Behavioural tests probe what a model can and cannot do, but writing them by hand is slow and narrow. Their system lets large models draft test templates and fill them in.

A voting scheme across several models checks each test label for quality. Tests span three angles: skill type, fairness, and robustness. On feeling and similarity tasks the system found far more model errors than past data sets and tools. Varied draft styles all worked steadily. The authors pitch it as a fuller health check for language software.

Catalog record

CREATORHengrui Xing, Cong Tian, Liang Zhao, Zhi Ma, Wensheng Wang, Nan Zhang, Chao Huang, Zhenhua Duan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models Hengrui Xing, Cong Tian, Liang Zhao, Zhi Ma, Wensheng Wang, Nan Zhang, Chao Huang, Zhenhua Duan Abstract . In recent years, the application of behavioral testing in Natural Language Processing (NLP) model evaluation has experienced a remarkable and substantial growth. However, the existing methods continue to be restricted by the requirements for manual labor and the limited scope of capability assessment. To address these limitations, we introduce AutoTestForge, an automated and multidimensional testing framework for NLP models in this paper. Within AutoTestForge, through the utilization of Large Language Models (LLMs) to automatically generate test templates and instantiate them, manual involvement is significantly reduced. Additionally, a mechanism for the validation of test case labels based on differential testing is implemented which makes use of a multi-model voting system to guarantee the quality of test cases. The framework also extends the test suite across three dimensions, taxonomy, fairness, and robustness, offering a comprehensive evaluation of the capabilities of NLP models. This expansion enables a more in-depth and thorough assessment of the models, providing valuable insights into their strengths and weaknesses. A comprehensive evaluation across sentiment analysis (SA) and semantic textual similarity (STS) tasks demonstrates that AutoTestForge consistently outperforms existing datasets and testing tools, achieving higher error detection rates (an average of 30.89 % 30.89\% for SA and 34.58 % 34.58\% for STS). Moreover, different generation strategies exhibit stable effectiveness, with error detection rates ranging from 29.03 % − 36.82 % 29.03\%-36.82\% . DOI: XXXXXXX.XXXXXXX Conference: ; ; CCS: Security and privacy Software security engineering CCS: Software and its engineering Software testing and debugging 1. Introduction Figure 1. Overview of AutoTestForge Natural language processing (NLP) models have demonstrated excellent performance in analyzing and handling natural language, attracting widespread attention and significantly influencing various aspects of our lives ( Khurana et al., 2023 ; Sefara et al., 2022 ) . However, it has been discovered that these models may be vulnerable to adversarial attacks, or affected by biases and noise present in the training data, potentially leading to unexpected outputs. This can result in misunderstandings ( Okrent, 2016 ) , economic losses ( Strickland, 2019 ) , threats to personal safety ( Xiang, 2023 ) , and even political conflicts ( Grinberg et al., 2019 ) . Consequently, it is essential to conduct comprehensive test evaluation on NLP systems, which should cover a wide range of possible risks, including adversarial attacks ( Xiao et al., 2023b ) , inherent biases ( Xiao et al., 2023a ) , and other relevant concerns ( Tan et al., 2021 ) , to identify and mitigate potential safety hazards. The standard practice for model evaluation is to divide data into training, validation, and test sets. Specifically, the training set serves as the basis for model learning, the validation set is utilized for fine-tuning the model, and the test set is employed for the final assessment of model performance, as detailed in ( Zhou, 2021 ) . Within the research community, it is a common practice to submit model performance results to public evaluation platforms (leaderboards) and engage in comparisons. This practice enables the tracking of technological advancements in the field, as demonstrated in ( Wang et al., 2018 ; Kiela et al., 2021 ) . However, this evaluation method is fraught with limitations. Test data frequently emanates from sources closely resembling those of the training data, and as a result, may harbor identical data biases. This situation has the potential to lead to an overestimation of model performance, as elucidated in ( Recht et al., 2019 ; Belinkov and Bisk, 2018 ) . Furthermore, relying on single performance metrics presents another challenge. It renders the identification of specific model deficiencies arduous and fails to offer clear-cut guidance for model enhancement, as indicated by ( Wu et al., 2019 ; Ribeiro et al., 2020 ) . In order to conduct more rigorous evaluations of NLP systems, researchers have put forward a variety of innovative evaluation methods. Among these methods, fairness evaluation aims to determine whether the model’s performance across different demographic groups, such as race and gender, is biased ( Xiao et al., 2023a ) . Robustness evaluation is concerned with assessing the stability of a model’s prediction results when faced with adversarial attacks and input perturbations ( Xiao et al., 2023b ) . Additionally, vocabulary ability evaluation focuses on the model’s overall performance in terms of its lexical coverage and the depth of its semantic understanding ( Pilehvar and Camacho-Collados, 2019 ) . Most of the existing methods examine the model from a wide variety of perspectives. They uncover issues that are not easily detected when relying solely on aggregate indicators. However, these methods typically focus on specific tasks or capabilities, falling short of comprehensively evaluating the model. To address this issue, the Checklist proposed by Ribeiro et al. ( Ribeiro et al., 2020 ) divides the input space into different language phenomena and test types, and designs corresponding test cases for each phenomenon and type. While this tool is effective in comprehensively evaluating NLP models, it still has some significant drawbacks. Specifically, designing high quality test suites demands substantial expertise and creativity, which mainly relies on manual completion ( Bhatt et al., 2021 ) . An annotator typically requires approximately one hour to generate 5-7 templates covering only 1-2 capabilities, with a substantial time investment of 0.5-2 working days necessary for producing a comprehensive set of multi-dimensional test templates. To improve efficiency, recent studies ( Ribeiro and Lundberg, 2022 ; Rastogi, 2023 ; Yang et al., 2022 ) have explored the possibility of automated test generation using generative models in conjunction with human intervention. Nevertheless, the process of creating templates or designing test topics still requires substantial human involvement, which significantly constrains the efficiency and scalability of test creation. Further, individual test cases often evaluate isolated model capabilities, overlooking the fact that real-world tasks typically demand multiple competencies simultaneously. Failure to account for the intricate connections between various capabilities may lead to overestimating the model’s actual performance. In summary, the existing NLP evaluation techniques still face challenges in reducing human dependency and conducting multidimensional capability assessment. In light of these challenges, this paper puts forward AutoTestForge, an automated and multi-dimensional testing framework designed for NLP models. This framework serves to diminish the reliance on domain experts while comprehensively assessing the performance of NLP systems across a broad array of capabilities. As depicted in Figure 1 , the process commences with the identification of the target task for evaluation. Subsequently, large language models (LLMs) are harnessed to generate detailed sentence structure descriptions and corresponding templates. These templates are then expanded via filling-expansion language models, giving rise to a rich variety of test cases. To guarantee the precision of the generated test cases, differential testing is adopted to authenticate the labels of test cases. Thereafter, a voting mechanism is implemented to single out the most dependable ones, and those judged to be inaccurate are rectified. Following this stage, the generated test suite undergoes further refinement to augment vocabulary coverage, fairness, and robustness. This iterative refinement process meticulously incorporates all the test cases yielded through the preceding steps, culminating

Read the complete text