Towards Multi-Platform Mutation Testing of Task-based Chatbots

Diego Clerissi, Elena Masserini, Daniela Micucci, Leonardo Mariani · 2025

Read Towards Multi-Platform Mutation Testing of Task-based Chatbots by Diego Clerissi, Elena Masserini, Daniela Micucci, Leonardo Mariani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01389.txt.

Introduction

Task-based chatbots book flights and answer bank questions. They handle other set jobs too. But testing every conversation path they support is hard. Diego Clerissi, Elena Masserini, Daniela Micucci, and Leonardo Mariani extend MUTABOT. That is a mutation testing approach for such bots. Mutation testing injects small faults on purpose, to see whether a test suite catches them.

The extension now covers two platforms, Dialogflow and Rasa. It makes faulty chatbot versions that mimic real conversational defects. Experiments show mutation testing can expose weaknesses in test suites. Those suites were made by Botium, a leading test generator. The team argues a suite that misses injected faults may also miss real bugs. Mutation scores thus give builders a sharper measure of true test quality.

Catalog record

CREATORDiego Clerissi, Elena Masserini, Daniela Micucci, Leonardo Mariani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01389

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Multi-Platform Mutation Testing of Task-based Chatbots Diego Clerissi, Elena Masserini, Daniela Micucci, Leonardo Mariani Abstract Chatbots, also known as conversational agents, have become ubiquitous, offering services for a multitude of domains. Unlike general-purpose chatbots, task-based chatbots are software designed to prioritize the completion of tasks of the domain they handle (e.g., flight booking). Given the growing popularity of chatbots, testing techniques that can generate full conversations as test cases have emerged. Still, thoroughly testing all the possible conversational scenarios implemented by a task-based chatbot is challenging, resulting in incorrect behaviors that may remain unnoticed. To address this challenge, we proposed Mutabot , a mutation testing approach for injecting faults in conversations and producing faulty chatbots that emulate defects that may affect the conversational aspects. In this paper, we present our extension of Mutabot to multiple platforms (Dialogflow and Rasa), and present experiments that show how mutation testing can be used to reveal weaknesses in test suites generated by the Botium state-of-the-art test generator. I Introduction As technology advances and services are increasingly accessible, chatbots have become ubiquitous in everyday activities, being able to support users across a wide range of domains [ 1 ] . Unlike general-purpose chatbots (e.g., ChatGPT [ 2 ] ), those developed to perform specific tasks (e.g., booking a hotel room, creating a Google Calendar event, or providing weather updates) are commonly referred to as task-based chatbots [ 3 , 1 ] . Task-based chatbots can be implemented using a variety of platforms, including Google Dialogflow [ 4 ] , Rasa [ 5 ] , and Amazon Lex [ 6 ] . Despite their growing adoption, ensuring the reliability of task-based chatbots remains a largely open challenge [ 7 , 8 ] , as it requires the generation of conversational scenarios that exercise relevant behaviors, as well as the definition of oracles able to accurately assess the correctness of the responses. Botium [ 9 ] is a state-of-the-practice testing framework for conversational agents, supporting automated test generation and execution. In Botium, a test case corresponds to a sequence of user-bot interactions, testing a chatbot’s functionalities from a conversational aspect. Botium’s capabilities have been leveraged by follow-up approaches, such as Charm [ 10 ] and CTG [ 11 ] . Still, these techniques exhibit weaknesses in terms of input space and oracle precision, resulting in limited coverage of conversations and bug detection. Mutation testing [ 12 ] has thus been recently adapted to the context of chatbots [ 13 , 14 ] . In this context, artificial faults (namely mutants ) are introduced to take into account the peculiarities of conversational aspects and development platforms, to evaluate the effectiveness of existing chatbot testing techniques in detecting these faults (i.e., killing the mutants , according to the standard terminology [ 12 ] ). For example, Dialogflow chatbots include both JSON files, which specify user utterances (i.e., user-provided inputs) and chatbot responses, and Javascript code, which triggers custom functions. Instead, Rasa implements conversations both with multiple YAML files, which define the context domain, training data, and flow rules, and with custom actions defined as Python functions. We recently presented Mutabot [ 13 ] , a mutation testing tool designed to support multi-platform mutations for task-based chatbots, originally developed for Dialogflow. In this paper, we describe how we adapted Mutabot to the Rasa platform, reporting some preliminary findings on the effectiveness of tests generated by Botium against mutants generated with Mutabot . Results suggest that more research is needed to generate fault-revealing conversations. The paper is structured as follows. Section II introduces the tool and the design advancements. Section III discusses our findings following the experiments in Rasa. Then, Section IV discusses the related work. Finally, Section V provides some conclusions and outlines future work. II Mutation Testing of Chatbots with Mutabot Task‑based chatbot platforms rely on a set of key concepts that describe how conversations are structured and managed. These concepts are captured in the meta‑model shown in Figure 1 , adapted from the one originally proposed by Cañizares et al. , which provides a platform‑agnostic perspective on chatbot structure [ 15 ] . Fig. 1: Chatbot structure meta-model adapted from Cañizares et al. [ 15 ] . In particular, the meta‑model identifies the following core elements: the intents (i.e., the goals a user wants to achieve by interacting with the chatbot), entities (i.e., the data types that identify the parameters used in the conversations), parameters (i.e., also called slots in Rasa, the variables that store the input values), actions (i.e., the operations a chatbot performs to fulfill a user’s intents), and flows (i.e., the user-bot conversational scenarios). In Rasa, flows are represented as sequences of intents and actions described through stories (i.e., flexible conversation examples) and rules (i.e., constrained predefined paths). Unlike traditional mutation techniques that typically manipulate code-level constructs (e.g., conditional nodes, array indexes), Mutabot is designed to tackle the complexity and peculiarities of conversational agents implemented according to the aforementioned meta-model. For example, a mutation operator can be designed to change the conversational flow of a chatbot. This operator has different implementations, depending on the target platform. In Dialogflow, flows can be manipulated by changing context objects in JSON files, while in Rasa flows can be manipulated by modifying the structure of the YAML files that describe conversational stories. The faults injected by Mutabot represent realistic errors that may arise from imperfectly designed chatbots, such as parameter names accidentally swapped in a response, data not correctly propagated between intents, or required entities not correctly specified in the training data. Figure 1 highlights in green the additional elements we have introduced into the meta‑model to support Rasa, showing the names of the supported operators in the callout texts. The used meta-model includes the changes proposed by Urrico et al. [ 13 ] . To this end, we have currently implemented the following eleven operators: (1) removeIntentFromNLU , (2) removeEntity , (3) removeRule , (4) removeStory , (5) removeIntentFromStory , (6) removeIntentFromRule , (7) removeInteractionFromRule , (8) removeInteractionFromStory , (9) changeSessionExpTimeInt , (10) changeSessionExpTimeFloat , and (11) toggleCarryOverSlots . The first four operators affect the chatbot structure, operating on different files, by removing conversational elements, for example, the removal of an intent from the training data (operator 1) or the removal of an entity (operator 2). The remaining seven operators affect the flow of a conversation. For instance, operators 9 and 10 extend or shorten the lifespan of a user-bot session by a numeric value, while toggleCarryOverSlots (operator 11) enables the reset mode for data shared among intents when a session ends. Whenever an operator can be applied multiple times to a chatbot (e.g., operator removeIntentFromNLU applied to a chatbot that includes several intents), Mutabot iteratively applies it to all possible cases and produces distinct mutants (e.g., one mutant for each removed intent). We speculate that some other not yet implemented mutations affecting the conversational features defined in the meta-model may not apply to all cases, or may require adaptations. For example, the intent priority, which defines the order of competing intent activations, is an explicit numeric property associated with each intent in Dialogflow, while in Rasa this property can be manipulated only indirectly by modifying the

Read the complete text