Test Case Generation for Dialogflow Task-Based Chatbots
Rocco Gianni Rapisarda, Davide Ginelli, Diego Clerissi, Leonardo Mariani · 2025
Read Test Case Generation for Dialogflow Task-Based Chatbots by Rocco Gianni Rapisarda, Davide Ginelli, Diego Clerissi, Leonardo Mariani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05561.txt.
Introduction
Rocco Gianni Rapisarda, Davide Ginelli, Diego Clerissi, and Leonardo Mariani present an automatic test maker for task-based chatbots, in a 2025 paper. Chatbots now handle jobs from booking to support inside web and phone apps, and like all software they ship with bugs. Testing them well is hard, and few tools build test cases for chatbots built on Google's Dialogflow.
Their tool, called CTG, writes test cases on its own for such bots. The team ran it against two rival tools, BOTIUM and CHARM, across seven chatbots. The tests from CTG proved tougher and more effective at exposing faults than those from either rival. The authors present the tool as a step toward quality checks that match how widely these bots are now used.
Catalog record
| CREATOR | Rocco Gianni Rapisarda, Davide Ginelli, Diego Clerissi, Leonardo Mariani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Test Case Generation for Dialogflow Task-Based Chatbots Rocco Gianni Rapisarda, Davide Ginelli, Diego Clerissi, Leonardo Mariani Abstract Chatbots are software typically embedded in Web and Mobile applications designed to assist the user in a plethora of activities, from chit-chatting to task completion. They enable diverse forms of interactions, like text and voice commands. As any software, even chatbots are susceptible to bugs, and their pervasiveness in our lives, as well as the underlying technological advancements, call for tailored quality assurance techniques. However, test case generation techniques for conversational chatbots are still limited. In this paper, we present Chatbot Test Generator ( CTG ), an automated testing technique designed for task-based chatbots. We conducted an experiment comparing CTG with state-of-the-art Botium and Charm tools with seven chatbots, observing that the test cases generated by CTG outperformed the competitors, in terms of robustness and effectiveness. I Introduction Task-based chatbots are chatbots designed to deliver functionalities through conversations. Notable examples are chatbots for assisting users, booking services, and completing transactions [ 1 , 2 ] . In recent years, task-based chatbots (hereafter simply chatbots) have gained popularity thanks to their integration into Web and Mobile applications, and advances in underlying technologies have made them ubiquitous in a multitude of domains, such as e-commerce, booking, tech support, healthcare, and more [ 3 , 4 , 5 , 1 , 2 ] . Further, chatbots have become significant in the context of business companies, as they allow for a reduction in personnel expenses by providing a potentially active 24/7 service. As chatbots continue to permeate our daily lives supported by a large number of design platforms, such as Dialogflow [ 6 ] , Amazon Lex [ 7 ] , IBM Watson Assistant [ 8 ] , and Rasa [ 9 ] , ensuring their reliability has become a critical concern. Unlike regular test case generation approaches that validate software through interactions taking the form of, for instance, API calls, HTTP requests, or clicks on visual elements, chatbots require the generation of actual conversations, that is, natural language sentences that correspond to actual user requests. Further, responses also take the form of natural language sentences that must be interpreted to determine their correctness. It is thus imperative to design ad-hoc techniques that can generate test cases that thoroughly exercise software whose interface is conversational [ 10 , 11 ] . The initial effort in test case generation for chatbots was mainly focused on exercising speech recognition capabilities [ 12 , 13 , 14 , 15 , 16 ] , and other non-functional characteristics [ 17 ] . The generation of test cases (i.e., conversations) to thoroughly validate the functionalities implemented by the chatbot under test required extensive manual intervention [ 18 , 19 , 9 , 20 , 21 , 22 , 23 , 24 ] . So far, two approaches have targeted automatic test case generation for chatbots: Botium [ 25 ] , which is a state-of-the-practice tool originally developed by Botium GmbH (now Cyara [ 26 ] ), and Charm [ 27 ] , which extends Botium with the capability to generate more diverse test cases. However, both approaches are not able to thoroughly cover the conversational input space of a chatbot and often generate incorrect test cases , due to the challenge of predicting the responses that must be produced by tested chatbots. To address the need for an automated testing solution capable of generating robust and effective test cases that can be used, this paper presents Chatbot Test Generator ( CTG ), an automated testing tool designed for Dialogflow task-based chatbots. CTG exploits the tests generated by Botium as seed tests that are systematically augmented to produce test cases that cover alternative conversational paths . Moreover, CTG is the first dynamic test generation approach for chatbots, that is, it executes the test cases and records the chatbot responses to obtain test cases that can be faithfully re-executed as regression tests, compared to statically generated tests that may include incorrect oracles. Finally, CTG generates test cases with setup and teardown operations that prepare and clean up the environment, preventing any accidental interference with the generated test cases. We conducted an experiment comparing CTG with Botium and Charm , state-of-the-art tools, observing that the test cases generated by CTG outperformed those generated by competing approaches, in terms of both robustness and effectiveness. In particular, CTG generated 95% correct test cases, compared to Botium and Charm that generated 61% and 82% correct tests, respectively. Moreover, the tests generated by CTG killed the highest number of mutants in five out of the seven chatbots considered in our mutation testing experiment. This result suggests that the tests obtained with CTG can be used to create better regression test suites than using state-of-the-art approaches. This paper provides the following contributions: (i) it proposes CTG , the first dynamic test case generation tool for Dialogflow task-based chatbots, (ii) it presents a systematic strategy to generate reliable conversations that cover the possible conversational paths, including the usage of entity values, alternative utterances, and set up and tear down of the testing environment, and, (iii) it presents empirical results that show how CTG outperforms the Botium and Charm state-of-the-art tools when generating tests for seven task-based chatbots. The paper is organized as follows. Section II introduces the key elements of task-based chatbots. Section III describes CTG . Section IV presents our experiment. Section V discusses related work. Then, Section VI provides final remarks. II Dialogflow Task-Based Chatbots Chatbots, also known as virtual assistants or conversational agents, are software programs designed to interact with human users through textual, vocal, images, or other communication channels [ 2 ] . Chatbots can be categorized based on their operational focus (i.e., informative, conversational, and task-based), their context (i.e., domain-specific or domain-independent), and the technology and modules they are composed of (e.g., natural language processors, speech recognition systems, machine learning models) [ 1 , 2 ] . Task-based chatbots are chatbots designed to efficiently complete specific tasks through user conversations, prioritizing functionality over casual chat. They aim to fulfill a user’s intent, such as booking a service or guiding users through procedures. Dialogflow [ 6 ] is a Natural Language Understanding (NLU) platform, part of the Google Cloud Platform, and one of the most popular platforms for multi-language chatbot design [ 28 ] . The platform offers a user-friendly Web interface, advanced machine learning and voice recognition modules for NLU, and native support with Google Cloud services. Further, it integrates with existing services (e.g., Slack, Telegram, Messenger) and custom external services, via Cloud Functions for Firebase. Finally, it exposes APIs that enable connections with existing testing tools, such as Botium [ 25 ] . In terms of structure, a Dialogflow chatbot is composed of several components encapsulating the main concepts of a conversation [ 29 , 30 ] : • Intents : they represent the possible goals of a user that the chatbot considers as tasks to achieve; users express their intent using utterances (e.g., the intent to order a pizza can be expressed by the sentence I would like to order a pizza ); • Actions : they are the actions that a chatbot may