Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback
Moghis Fereidouni, Md Sajid Ahmed, Adib Mosharrof, A.B. Siddique · 2025
Read Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback by Moghis Fereidouni, Md Sajid Ahmed, Adib Mosharrof, A.B. Siddique (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13298.txt.
Introduction
Moghis Fereidouni, Md Sajid Ahmed, Adib Mosharrof, and Siddique tackle chat systems that must finish tasks in a 2025 paper. Task-oriented dialog systems help users book, order, and query through chat over many turns. Large language models do well on single turns but often fail on long tasks. Their errors often come when they call outside tools through application programming interfaces.
The team built a framework called RealTOD to fix this. It chains prompts to make a schema-fit example for each new task. It also checks each tool call and gives fine-grained feedback to fix errors. On two test sets, RealTOD raised full call accuracy well past prior systems. Human judges also rated its chats as more fluent and more complete.
Catalog record
| CREATOR | Moghis Fereidouni, Md Sajid Ahmed, Adib Mosharrof, A.B. Siddique |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13298
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback Moghis Fereidouni, Md Sajid Ahmed, Adib Mosharrof, A.B. Siddique Abstract Task-oriented dialog (TOD) systems facilitate users in accomplishing complex, multi-turn tasks through natural language. While traditional approaches rely on extensive fine-tuning and annotated data for each domain, instruction-tuned large language models (LLMs) offer a more flexible alternative. However, LLMs struggle to reliably handle multi-turn task completion, particularly with accurately generating API calls and adapting to new domains without explicit demonstrations. To address these challenges, we propose RealTOD, a novel framework that enhances TOD systems through prompt chaining and fine-grained feedback mechanisms. Prompt chaining enables zero-shot domain adaptation via a two-stage prompting strategy, eliminating the need for human-curated demonstrations. Meanwhile, the fine-grained feedback mechanism improves task completion by verifying API calls against domain schemas and providing precise corrective feedback when errors are detected. We conduct extensive experiments on the SGD and BiTOD benchmarks using four LLMs. RealTOD improves API accuracy, surpassing AutoTOD by 37.74% on SGD and SimpleTOD by 11.26% on BiTOD. Human evaluations further confirm that LLMs integrated with RealTOD achieve superior task completion, fluency, and informativeness compared to existing methods. 1 1 1 Source code will be released upon acceptance. 1 Introduction Figure 1: Overview of RealTOD: With only a single example dialog from one domain, RealTOD scales to infinitely many domains through prompt chaining, removing the need for human-curated dialogs for each domain. A fine-grained feedback loop from the API parser further improves API call accuracy. Task-oriented dialog (TOD) systems enable users to accomplish multi-turn tasks, such as booking flights, making restaurant reservations, and managing appointments, through multi-turn, natural language interactions He et al. 2022 . These systems must understand user intent, retrieve relevant information from external systems via API calls, and generate coherent responses to guide users toward task completion. Traditional TOD systems rely on extensive domain-specific fine-tuning and manually annotated datasets, which limit their scalability to new domains and increase deployment costs Xu et al. 2024 ; Mi et al. 2022 . As a result, developing TOD systems that generalize across diverse domains without extensive supervision remains an open challenge. Recent advances in instruction-tuned large language models (LLMs) Chung et al. 2024 ; Shu et al. 2024 have significantly improved performance across a wide range of natural language processing (NLP) tasks, including text classification Sun et al. 2023 ; Wang et al. 2023 ; Zhang et al. 2024b , summarization Pu et al. 2023 ; Zhang et al. 2024a ; Van Veen et al. 2023 , and response generation Radford et al. 2019 ; Brown et al. 2020 . These models, trained on diverse instruction-following datasets, can generate fluent and contextually relevant responses in dialog settings Thoppilan et al. 2022 ; Dubey et al. 2024 ; Achiam et al. 2023 , making them promising candidates for TOD systems. However, integrating LLMs into TOD remains challenging. Unlike single-step NLP tasks, TOD systems require multi-turn interactions to collect task-specific information, retrieve external data, and execute user requests while adhering to domain constraints Chung et al. 2023 . While LLMs demonstrate strong generalization in open-ended response generation Naveed et al. 2023 , they struggle with task completion – measured through API call accuracy Shinn et al. 2023 ; Jain et al. 2024 ; Song et al. 2025 . Common issues include hallucinated search results, incorrect API method names, invalid slot-value pairs, and missing required parameters, all of which can lead to execution failures and incomplete task fulfillment. Addressing these limitations is critical to unlock the full potential of LLM-based TOD systems. To overcome these challenges, we propose RealTOD, a novel framework that enhances LLM-based TOD systems through prompt chaining and fine-grained feedback. Prompt chaining enables zero-shot domain adaptation by using a two-stage prompting process: first, an example dialog from a source domain is transformed into a target domain dialog by aligning it with the domain schema while maintaining task consistency. This generated example serves as a demonstration, allowing LLMs to generalize to new domains without human-curated examples. Meanwhile, fine-grained feedback improves API execution reliability by systematically verifying API calls against domain schemas, detecting errors (e.g., incorrect method names, invalid slot assignments, missing parameters), and providing precise corrective signals. This iterative feedback mechanism enables real-time error correction, ultimately enhancing task completion rates. We evaluate RealTOD on two benchmark datasets – SGD Rastogi et al. 2020 and BiTOD Lin et al. 2021b – using four LLMs: two proprietary models (GPT-4o and Claude) and two open-source models (DeepSeek and LLaMA). We measure task completion using full API Call Accuracy, which assesses how often the generated API calls exactly match the ground truth. Even minor errors – such as incorrect method names, missing parameters, or invalid slot-value pairs – can lead to execution failures, resulting in failed tasks. The quality of natural language responses is evaluated using BERTScore Zhang et al. 2019 , which measures semantic similarity between generated and reference responses using contextual embeddings, offering a more reliable assessment than traditional n-gram-based metrics. Our results show that RealTOD significantly improves API accuracy across all models and datasets. On SGD, it surpasses AutoTOD Hosseini-Asl et al. 2020 by 37.74% in full API accuracy, while on BiTOD, it outperforms supervised fine-tuned SimpleTOD Xu et al. 2024 by 11.26%. Human evaluations further confirm that LLMs integrated with RealTOD generate more fluent, informative, and effective task completions than baseline models. Our ablation study demonstrates that both prompt chaining and fine-grained feedback contribute to improved multi-turn dialog quality and reliable task completion. 2 Related Works Fine-Tuned Task-Oriented Dialog Systems. TOD systems are typically classified into pipeline-based and end-to-end approaches. Pipeline-based methods Williams and Young 2007 ; Lee 2013 ; Lee et al. 2009 ; Peng et al. 2020 ; Chen et al. 2019 decompose the system into modular components—natural language understanding, dialog state tracking, policy learning, and natural language generation—allowing independent optimization of each module. In contrast, end-to-end approaches Hosseini-Asl et al. 2020 ; Madotto et al. 2018 ; Su et al. 2022 ; Mosharrof et al. 2023 ; Siddique et al. 2022 ; Lei et al. 2018 ; Lin et al. 2020 ; Imrattanatrai and Fukuda 2023 generate responses directly, bypassing these modules. A major drawback of these fine-tuned methods is their reliance on high-quality labeled data, which can be a significant limitation. LLM-Powered Systems. The rise of LLMs has led to the development of various intelligent systems, which can be broadly categorized into three classes. The first class includes Web Agents, which facilitate online interactions for information retrieval and task execution Yao et al. 2023 ; Kim et al. 2024 ; Ma et al. 2023 ; Fereidouni et al. 2024 ; Yao et al. 2022 ; Sridhar et al. 2023 ; Furuta et al. 2024 . The second class consists of Mobile Agents, which focus on optimizing LLM-based decision-making for performing diverse tasks on mobile applications Bai et al. 2024 ; Lee et al. 2023 ; Wen et al. 2024 ; Wen et al. 2023 ; Wang et al. 2024b ; Wang et al. 2024a . The third and most relevant class to our work is LLM-powered TOD Systems Chung et al. 2023 ; Mi et al. 2022 ; Gao et