ConsistentChat : Building Skeleton-Guided Consistent Dialogues for Large Language Models from Scratch

Jiawei Chen, Xinyan Guan, Qianhao Yuan, Guozhao Mo, Weixiang Zhou, Yaojie Lu, Hongyu Lin, Ben He, Le Sun, Xianpei Han · 2025

Read ConsistentChat : Building Skeleton-Guided Consistent Dialogues for Large Language Models from Scratch by Jiawei Chen, Xinyan Guan, Qianhao Yuan, Guozhao Mo, Weixiang Zhou, Yaojie Lu, Hongyu Lin, Ben He, Le Sun, Xianpei Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03558.txt.

Introduction

Jiawei Chen and co-authors build multi-turn dialogue data that stays on track, in a 2025 paper. Instruction data for language models is mostly single-turn, and stitched-together conversations drift off context, which lowers task completion in long chats. Their framework first models the intent arc of a whole conversation, assigning it to one of nine well-defined trajectories, so the exchange keeps a goal and a sensible flow.

On that skeleton, a sequence of user queries is generated to match the intent, and the full dialogue is then synthesised under that scaffold. The resulting data set, ConsistentChat, holds about fifteen thousand conversations and over two hundred and twenty-four thousand utterances. Models fine-tuned on it improved chat consistency by twenty to thirty percent on standard benchmarks and raised task success by up to fifteen percent. Both gains beat models trained on existing single and multi-turn sets.

Catalog record

CREATORJiawei Chen, Xinyan Guan, Qianhao Yuan, Guozhao Mo, Weixiang Zhou, Yaojie Lu, Hongyu Lin, Ben He, Le Sun, Xianpei Han
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ConsistentChat : Building Skeleton-Guided Consistent Dialogues for Large Language Models from Scratch Jiawei Chen, Xinyan Guan, Qianhao Yuan, Guozhao Mo, Weixiang Zhou, Yaojie Lu, Hongyu Lin, Ben He, Le Sun, Xianpei Han Abstract Current instruction data synthesis methods primarily focus on single-turn instructions and often neglect cross-turn coherence, resulting in context drift and reduced task completion rates in extended conversations. To address this limitation, we propose Skeleton-Guided Multi-Turn Dialogue Generation, a framework that constrains multi-turn instruction synthesis by explicitly modeling human conversational intent. It operates in two stages: (1) Intent Modeling, which captures the global structure of human dialogues by assigning each conversation to one of nine well-defined intent trajectories, ensuring a coherent and goal-oriented information flow; and (2) Skeleton Generation, which constructs a structurally grounded sequence of user queries aligned with the modeled intent, thereby serving as a scaffold that constrains and guides the downstream instruction synthesis process. Based on this process, we construct ConsistentChat , a multi-turn instruction dataset with approximately 15,000 multi-turn conversations and 224,392 utterances. Experiments on the Light , Topdial , and MT-Eval benchmarks show that models fine-tuned on ConsistentChat achieve a 20–30% improvement in chat consistency and up to a 15% increase in task success rate, significantly outperforming models trained on existing single-turn and multi-turn instruction datasets. 1 Introduction (a) Single-Turn Instruction Synthesis Manners. (b) Multi-Turn Instruction Synthesis Manners. Figure 1: Comparison of single-turn and multi-turn instruction synthesis manners. Instruction synthesis plays a fundamental role in enabling large language models (LLMs) to perform a wide array of real-world dialogue tasks Iyer et al. 2022 . High-quality instruction datasets are indispensable for supervised fine-tuning (SFT), allowing LLMs to better capture user intent and handle diverse conversational scenarios, such as customer support and educational tutoring Sanh et al. 2022 ; Wang et al. 2023b . As the demand for scalable and domain-adaptive models grows, the automatic construction of instruction data has become a challenge in the development of advanced dialogue systems. Currently, existing instruction data construction paradigms are primarily tailored to single-turn interactions. Works such as Self-Instruct Wang et al. 2023b , Evolve-Instruct Xu et al. 2023 , and REInstruct Chen et al. 2024 leverage prompt expansion and corpus mining to generate large-scale <instruction, response> pairs. However, these methods mainly focus on turn-level exchanges, overlooking the sequential dependencies and interactive dynamics intrinsic to natural conversations. This gap not only limits the capacity of models to capture real conversational develop, but also fails to reflect the actual requirements of real-world applications, where multi-turn interactions are the norm rather than the exception. Fundamentally, some approaches explored the synthesis of multi-turn instruction datasets, such as MemoChat Lu et al. 2023 , ChatAlpaca Bian et al. 2023 and GLAN Li et al. 2024 . These approaches largely rely on either transforming traditional NLP tasks into dialogues or simulating multi-turn conversations with LLMs. Such strategies are typically unconstrained: the generation process focuses on producing locally coherent, turn-level instruction with response pairs, failing to model the evolution of human intent or the overall dialogue trajectory. As a result, these datasets often suffer from poor chat consistency across turns, with frequent occurrences of topic drift throughout the conversation. This limitation poses a challenge for training models to maintain alignment and consistency in multi-turn interactions. Based on the above observations, we propose ConsistentChat , an instruction synthesis framework that can model human conversational intent and information flow to guide multi-turn dialogue generation. We analyze nine categories of dialogue intent Rapp et al. 2021 (e.g., Problem-Solving-Interaction and Educational-Interaction) and formalize them as dynamic information flows that constrain both topic progression and role interaction throughout the dialogue. Our approach first generates a globally coherent sequence of user queries conditioned on the underlying intent, and subsequently produces agent responses that maintain alignment with both current and global conversational objectives. We evaluate ConsistentChat on the Light, Topdial, and MT-Eval benchmarks. Preliminary analyses reveal that LLMs’ multi-turn dialogue capabilities degrade with increasing dialogue depth, and strong chat consistency in training data is crucial for SFT models’ alignment across turns. Leveraging our skeleton-guided framework, we build high-quality multi-turn instruction dialogues. Experimental results show that models fine-tuned on ConsistentChat instruction consistently outperform those trained on existing datasets, both in single-turn and multi-turn settings, establishing state-of-the-art results in dialogue chat consistency and alignment. Our findings highlight the importance of modeling human conversational intent and information flow for generating reliable instruction data for multi-turn dialogue agents. The main contributions of this paper are: 1) This is the first work investigating how modeling human conversational intent in instruction synthesis influences the chat consistency of fine-tuned dialogue models. 2) We propose ConsistentChat , a simple yet effective Skeleton-Guided framework for supervised fine-tuning, which can be applied in broad downstream dialogue scenarios. 3) Extensive experiments demonstrate that instruction data generated by ConsistentChat outperforms existing popular multi-turn datasets in terms of chat consistency, as well as both single-turn and multi-turn conversational capability. 2 Revealing Multi-Turn Degradation and Chat Consistency Effects In this section, we first investigate the performance of popular dialogue models in multi-turn conversations and explore how fine-tuning on datasets with varying chat consistency affects their performance. We conduct analyses addressing these two questions: • Question I : When current conversational models engage in multi-turn dialogues, how does the multi-turn conversational abilities evolve as the conversation progresses? • Question II : Whether the chat consistency of dialogue datasets affects the performance of fine-tuned models? We conducted experiments for the multi-turn conversational ability on MT-Eval Kwan et al. 2024 , using mainstream dialogue models to answer question I. We then selected instructions from ShareGPT Chiang et al. 2023 with different levels of consistency to supervise fine-tuning pre-trained model, using chat consistency metric on Light Urbanek et al. 2019 and TopDial Wang et al. 2023a for question II. More details are in Appendix A . 2.1 Conversational Abilities Diminishes as the Dialogue Deepens Setups. We selected several dialogue models (Qwen-2.5, LLaMA-3.1, Mistral-0.3, Vicuna-1.5), totaling 8 models, to evaluate the performance on Refinement task in MT-Eval benchmark. Results. Figure 2 reveals that as dialogue turns increase, the conversational ability diminishes correspondingly. Detailed results are in Appendix A.1 . Figure 2: Performance of multi-turn conversational ability declines as more instructions are added on Refinement task in MT-Eval. Each dialogue in task has 2 NLP tasks with each task comprising 6 increasingly complex instructions. The transition to the second NLP task occurs at the seventh turn as denoted by the gray dashed line. Judged by Qwen-2.5-72B-Instruct. 2.2 Dataset Consistency Affects Fine-tuned Model Performance Setups. We curated three distinct sets of multi-turn dialogues from ShareGPT Xu 2023 , each grouped by different levels of chat consistency: high consistency, low consistency, and randomly sampled conversations. Each dataset comprised approximately 10,000 dialogue instances. We subsequently conducted supervised fine-tuning of LLaMA-3.1-8B model on these datasets and evaluated the models for chat consistency assessment and multi-turn abilities. Results. Figure 3 shows that LLaMA-3.1-8B-High achieves the best results in both consistency metrics (detailed in Table 8 ) and conversational capabilities (detailed in Table 9 ), illustrating that chat consistency plays a crucial role in fine-tuning models’ performance. Figure 3: Performance of LLaMA-3.1-8B models (Low, Sample, and High consistency) across four evaluation settings: TopDial , Light and Single-Turn

Read the complete text