Factors affecting the in-context learning abilities of LLMs for dialogue state tracking
Pradyoth, Santosh, Ján, Šimon, Bolaji, Oldřich, Deepak, Jan · 2025
Read Factors affecting the in-context learning abilities of LLMs for dialogue state tracking by Pradyoth, Santosh, Ján, Šimon, Bolaji, Oldřich, Deepak, Jan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08753.txt.
Introduction
Dialogue state tracking means keeping track of what a user wants across a conversation, and the authors test whether in-context learning can do that job well. They pull example dialogues with a sentence-embedding nearest-neighbor method and slot the examples plus the test case into a template fed to a language model. Then they run a systematic study of what matters: how examples are picked and how the prompt is framed.
The work uses the MultiWoZ2.4 data set and centers on three models: OLMo-7B-instruct, Mistral-7B-Instruct-v0.3, and Llama3.2-3B-Instruct. The findings give practical guidance on in-context learning for dialogue state tracking, showing which choices in example selection and prompt context move performance and which do not.
Catalog record
| CREATOR | Pradyoth, Santosh, Ján, Šimon, Bolaji, Oldřich, Deepak, Jan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08753
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Factors affecting the in-context learning abilities of LLMs for dialogue state tracking Pradyoth, Santosh, Ján, Šimon, Bolaji, Oldřich, Deepak, Jan Abstract This study explores the application of in-context learning (ICL) to the dialogue state tracking (DST) problem and investigates the factors that influence its effectiveness. We use a sentence embedding based k-nearest neighbour method to retrieve the suitable demonstrations for ICL. The selected demonstrations, along with the test samples, are structured within a template as input to the LLM. We then conduct a systematic study to analyse the impact of factors related to demonstration selection and prompt context on DST performance. This work is conducted using the MultiWoZ2.4 dataset and focuses primarily on the OLMo-7B-instruct, Mistral-7B-Instruct-v0.3, and Llama3.2-3B-Instruct models. Our findings provide several useful insights on in-context learning abilities of LLMs for dialogue state tracking 1 1 1 Accepted to Interspeech’2025, ISCA . Hegde Kesiraju Švec Sedláček Yusuf Plchot K T Černocký Brno University of TechnologyCzechia Indian Institute of Information Technology DharwadIndia keywords in-context learning, dialog state tracking † † email: [email protected], [email protected] 1 Introduction Instruction-tuned large language models (LLMs) have demonstrated enhanced capabilities compared to traditional language models, enabling them to perform a broader range of tasks based on instructions [ 1 , 2 , 3 ] . A key capability of LLMs is in-context learning (ICL), which allows them to generalise to new tasks without requiring explicit fine-tuning, thus reducing the need for extensive task-specific training data and computational resources [ 4 , 5 , 6 ] . This adaptability makes them particularly attractive for complex applications, such as task-oriented dialogue systems, where the ability to quickly adapt to new domains and user intents is highly desirable [ 7 ] . Task-oriented dialogue (TOD) systems are conversational systems designed to achieve specific, predefined goals, such as booking a flight, ordering food, or scheduling an appointment. To achieve these goals, the dialogue manager within the system maintains a representation of the current dialogue state. The dialogue state comprises the domain of the conversation (e.g. restaurant, taxi) and the relevant information or values associated with that domain, referred to as slots (e.g. restaurant-name, taxi-destination). In multi-turn dialogues, accurately tracking the domain and the corresponding slot values is known as dialogue state tracking (DST). This is crucial for the overall performance of TOD systems, as it directly impacts the system’s ability to understand user requests, provide relevant information, and ultimately achieve the desired task completion [ 8 ] . The effectiveness of ICL is heavily influenced by the selection of appropriate demonstrations [ 9 , 10 , 11 , 12 , 13 ] . Several recent works have focused on improving demonstration selection strategies for LLMs [ 14 , 15 , 16 ] . Rubin et al. [ 17 ] explored a retrieval-based approach, training a model to score demonstrations based on LLM performance. Venkateswaran et al. [ 18 ] fine-tuned a language model using ICL and observed that while fine-tuning can improve performance, the quality and diversity of the demonstrations remain critical factors, noting that low diversity and semantic similarity in the selected demonstrations can negatively impact performance. Interestingly, Min et al. [ 19 ] found that the ground truth slot values in the demonstrations may not be as important as previously thought, as replacing them with random words did not significantly degrade performance, suggesting that the structure and context provided by the demonstrations may be more influential than the specific slot values themselves. In this work, we study a nearest-neighbour-based retrieval to select relevant demonstrations (dialogue turns) for ICL-based DST. These demonstrations are then formatted using a carefully designed prompt template to effectively guide the LLM towards accurate state tracking. Specifically, we propose a modular prompt structure for DST, consisting of distinct blocks for the conversation history, the domain(s), and the corresponding slot values. This modularity allows for easy adaptation and experimentation with different prompt configurations. This study focuses exclusively on in-domain examples. Due to the poor performance of zero-shot prompting in our format, we limit our investigation to demonstration-based DST. We conduct a systematic investigation and identify the factors influencing both the demonstration retriever and the prompt structure in the final DST performance. Our investigation reveals that • Using embeddings of only the user’s utterances within the retriever for demonstration selection yields better performance than including both user and agent utterances. This suggests that user input is a more reliable indicator of the underlying dialogue state. • Embedding models trained on multilingual data (LaBSE) or specifically for dialogue tasks (D2F) achieve comparable results in our setup, indicating that domain-specific fine-tuning of the embedding model may not be necessary for achieving good performance in our ICL-based DST system. • We also find that tags indicating speaker roles (user, agent), the number of demonstrations, and dialogue history in the demonstrations play an important role in the final DST performance. User turn (test sample) Demonstration retriever (embedding models) Prompt constructor (dialogue history, speaker tags) Inference with LLM (constrained decoding) Post-process (JSON repair) Predicted dialogue state Figure 1: Block diagram of the ICL scheme used in our experiments. Factors influencing the shaded parts are studied in this paper. The remainder of this paper is organised as follows. Section 2 details our methodology and describes the key components involved. Section 3 describes our experiments and presents the results, showing the performance of our system under different configurations. Section 4 provides an in-depth analysis of our results and highlights the main findings of our investigation. Finally, Section 5 concludes by summarising our contributions and outlining potential avenues for future work in the field of ICL-based DST. 2 Methodology This section provides a detailed step-by-step description of our proposed system. Figure 1 provides a high-level representation of the data flow and the interactions between different modules of the system. 2.1 Demonstration retriever The selection of appropriate demonstrations is crucial for effective in-context learning, as the quality and relevance of the demonstrations directly influence the LLM’s ability to generalise to new examples. While various strategies exist for selecting demonstrations, such as ensuring coverage across all domains or including multi-domain examples to enhance robustness, we observed that retrieving demonstrations from semantically similar dialogues yield better performance. Semantically similar dialogues are likely to contain relevant contextual information (domains) and slot keys that will be helpful for accurately tracking the dialogue state. Our system employs a retriever that extracts embeddings of each dialogue turn of the user along with its history, using a pre-trained sentence embedding. In this work, we compare two embedding models. The first is LaBSE that was trained on multilingual parallel data, which efficiently captures cross-lingual features that are language agnostic [ 20 ] . The second is Dialog2Flow (D2F), which is trained with a contrastive objective to produce semantically similar embeddings for turns with similar dialogue acts [ 21 ] . To improve efficiency and reduce computational overhead, embeddings of the training set are pre-computed. During inference time, the retriever selects the K closest training samples to the test sample based on cosine similarity . 2.2 Prompt construction The prompt serves as the primary interface of the LLM and guides its output. Our prompt template consists of three modular blocks: the conversation history, the domain(s)