Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs
Šimon, Bolaji, Ján, Pradyoth, Santosh, Oldřich, Jan · 2025
Read Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs by Šimon, Bolaji, Ján, Pradyoth, Santosh, Oldřich, Jan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08633.txt.
Introduction
Voice assistants that book tables or check orders must track what has been settled so far in a chat, and doing that straight from speech is harder than from typed text. Šimon Sedláček and co-authors bridge speech and language models in a 2025 paper. A small connector joins a speech reader to a large language model so the two share one inner space.
The team tests how to tune the parts, whether the agent's own turns help as history, and how fuzzy matching can clean up names in the output. That cleaning step alone lifted scores a lot. Their best fully open build set a new high score on a spoken booking test set. A larger partner model pushed the score higher still, which shows the bridge scales with the model behind it.
Catalog record
| CREATOR | Šimon, Bolaji, Ján, Pradyoth, Santosh, Oldřich, Jan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08633
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs Šimon, Bolaji, Ján, Pradyoth, Santosh, Oldřich, Jan Abstract In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test. Sedláček Yusuf Švec Hegde Kesiraju Plchot Černocký Brno University of TechnologyCzechia Indian Institute of Information Technology DharwadIndia keywords dialogue state tracking, task-oriented dialogue, speech LLMs † † email: {isedlacek,iyusuf,isvecjan,kesiraju,iplchot,cernocky}@fit.vut.cz, [email protected] 1 Introduction Task-oriented dialogues (ToD) are multiturn conversations between a user and an agent, where the former has a specific goal (e.g., booking a restaurant for 5 people on Friday night) that is achieved with the help of the agent. A key component of automated ToD systems is dialogue state tracking (DST), the task of tracking the user’s intent (e.g. “book-restaurant”) and identifying the slots (e.g. “restaurant-people”: “5”, “day”: “Friday”). Recently, aided by the DSTC-11 challenge [ 1 ] and the development of realistic SpokenWoZ [ 2 ] dataset, the scope of DST research is slowly advancing from being exclusively text-based to speech domain. A typical approach to DST from spoken conversations is via a cascade of several systems: automatic speech recognition (ASR) \rightarrow error correction module \rightarrow text-based DST [ 1 , 3 ] . Although end-to-end (E2E) DST training offers an attractive alternative due to the simplicity of training and the potential to avoid cascading errors, E2E DST models are difficult to train due to the scarcity of data compared to other speech processing tasks. The paradigm of interconnecting pre-trained speech encoders with large language models (LLMs) offers a solution to the issue and has shown performance that is competitive with cascade systems [ 4 ] . However, most prior work relies on fully or partly closed models for which either the model weights or training data are not openly available. Modality matching is one of the key challenges in aligning pre-trained speech and language models. Various approaches such as text-representation up-sampling [ 5 ] , speech-representation sub-sampling and fixed-length representations based on Q-formers have been studied in the past [ 6 , 7 ] . The application of such speech language models was not only studied on traditional tasks such as ASR, speech translation and text-to-speech [ 8 , 9 ] , but also on language understanding. question-answering tasks and dialogue state tracking [ 10 , 11 , 12 ] . Retaining the ability of pre-trained LLMs even after the alignment is one of the main challenges [ 4 ] , as simple connector based approaches [ 13 ] do not generalize well beyond the domain of the training data [ 14 ] . Our work complements the prior works, as we study the task of dialogue state tracking from spoken conversations with fully open speech and language models. In this paper, we propose an end-to-end DST model based on fully open components. We connect a WavLM speech [ 15 ] encoder with an OLMo [ 16 ] LM through a small trainable connector module, which we first train to align the representation spaces of the pre-trained modes and then fine-tune the resulting model directly for DST. We conducted experiments on the SpokenWoZ [ 2 ] and Speech-Aware MultiWoZ [ 1 ] datasets, showing that our proposed system with the fully open OLMo-1B model already yields state-of-the-art performance on SpokenWoZ and that we can obtain further significant improvements when we use the Gemma-2-9B-Instruct model. 2 Method overview Our method treats DST as the problem of mapping from user speech and user-agent dialogue history to a dictionary containing the correct transcription of the user’s speech, and the inferred dialogue states (domains, slots). Therefore, we adopt a model which we train to take speech as input and directly output a JSON string representing this dictionary at each turn of the dialogue. The model is composed of three parts: a pretrained speech encoder, a pretrained LLM and a small connector module that joins the speech encoder and the LLM. The encoder computes a representation of the speech input at the current turn of the conversation. Then, the speech representations are downsampled to better match the granularity of the LLM text input, and connector module maps them into the text embedding space of the LM, where they act as soft prompts. Finally, the LM auto-regressively generates the tokens that comprise the JSON string. The LM input is prefixed with the connector output to provide speech conditioning and by the dialogue history in the form of the transcription of previous user inputs and agent responses. 2.1 Training Starting from pretrained encoder, we train the model in the two stages illustrated in Figure 1 : an ASR pretraining stage and a joint ASR-DST finetuning. 2.1.1 ASR pre-training Since the speech encoder and LM are pretrained independently of each other, we first bridge their representation spaces before fine-tuning for DST. In this first stage, we freeze the LM and finetune the encoder and connector for ASR, conditioning the LLM solely using the connector outputs 1 1 1 ASR prompt format: ‘%speech_embeds% {“transcription”: %labels%}’ . Thus, we are able to leverage large scale ASR datasets (which are orders of magnitude larger than typical DST datasets) to obtain robust alignment of the encoder and the LM. Moreover, ASR training is necessary since the dialogue history for the final DST model requires transcribing the user input. 2.1.2 Joint ASR-DST finetuning In the second training stage, we keep the speech encoder frozen, and introduce LoRA [ 17 ] adapters into the LM, which we finetune along with the connector for the target DST task. For each turn in a conversation, we append the ground-truth dialogue history to the connector output speech embeddings, and train the model to minimize the negative log-likelihood of the JSON string encoding the dialogue state for the current turn – the transcription, domains, and slots. Note that it is imperative that we still train jointly for ASR and DST since the model will not have access to the full ground-truth dialogue history at inference time, and will have to rely on its own ASR capabilities to get the user side of the dialogue. 2.2 Inference and post-processing Inference proceeds in a turn-by-turn fashion. For each turn, the JSON string outputted by the model is converted into the corresponding dictionary whose elements are read as the dialogue state and the dictionary field corresponding to the user ASR transcription is appended to the previous dialogue history for use in subsequent turns of the dialogue. Input speech Speech Encoder Connector LLM ASR