Spoken Conversational Agents with Large Language Models
Chao-Han Huck Yang, Andreas Stolcke, Larry P Heck, NVIDIA Research, Uniphore, Georgia Institute of Technology [email protected] [email protected] [email protected] · 2025
Read Spoken Conversational Agents with Large Language Models by Chao-Han Huck Yang, Andreas Stolcke, Larry P Heck, NVIDIA Research, Uniphore, Georgia Institute of Technology [email protected] [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-02593.txt.
Introduction
Chao-Han Huck Yang, Andreas Stolcke, and Larry P Heck survey spoken conversational agents built on large language models. Voice based models have drawn wide interest from researchers and the public alike. Leading closed models now handle speech recognition, translation, and spoken understanding far better than older open systems.
Their 2025 overview maps where these voice agents stand and what they can do. Speech tasks once handled by separate tools are merging into single conversational systems. A clear map of the field helps builders choose models, spot gaps, and design assistants people can truly talk with. Voice is how most of us talk, so gains here reach folks who type with pain or not at all.
Catalog record
| CREATOR | Chao-Han Huck Yang, Andreas Stolcke, Larry P Heck, NVIDIA Research, Uniphore, Georgia Institute of Technology [email protected] [email protected] [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-02593
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spoken Conversational Agents with Large Language Models Chao-Han Huck Yang, Andreas Stolcke, Larry P Heck, NVIDIA Research, Uniphore, Georgia Institute of Technology [email protected] [email protected] [email protected] Abstract 1 Introduction Recent advancements in large language models (LLMs) with voice interfaces have garnered significant attention from both the research community and broader society. Closed-source models, such as GPT-4o and Gemini-1.5-pro, have demonstrated superior performance across classical speech tasks, including (i) speech recognition, (ii) translation, and (iii) spoken language understanding, significantly surpassing previous open-source benchmarks. Despite these strides, there remains a lack of comprehensive studies on the design and mechanisms underpinning the integration of speech modalities into LLMs for true multi-modal understanding. Additionally, the interaction between speech models and LLMs, particularly in the context of layered, self-play cognitive agents Shah et al. 2018a ; Shah et al. 2018b , has only recently begun to be explored. This tutorial aims to provide a thorough review of the historical trajectory of probabilistic language modeling Jurafsky and Martin 2024 for speech processing, offering insights that motivate the development of multi-agents system in conversational models. We will delve into advanced topics such as cross-modal adaptation, introducing on the theoretical foundations Yang et al. 2021 required to align non-text modalities with textual representations. These concepts will be discussed alongside open-source, reproducible benchmarks Chen et al. 2023a , providing a practical grounding for participants. In addition, we will explore more recent trends toward end-to-end multi-modal speech-language models, emphasizing designs that utilize generative autoregressive approaches with joint speech-text tokenization. By examining both cascaded and end-to-end perspectives, this tutorial will equip participants with a comprehensive understanding of current strategies and open challenges in speech-augmented LLMs as shown in Figure 1 . Figure 1: Examples of spoken conversational agent with different LLMs to understand different linguistic information from (a) semantics, (b) paralinguistics, and to (c) more phonetic signals. 2 Tutorial Outline This three-hour tutorial will focus on the rapidly evolving field of speech-language modeling in the era of voice-interfacing LLMs and agent systems. Each core theme will be covered in a 35 -minute segment, followed by a 10 -minute Q&A and a 10 -minute break. For each section, we will provide an overview of the relevant topics, followed by an in-depth exploration of key studies that shape the current landscape. The tutorial will conclude with a discussion of the open challenges and emerging research opportunities in this exciting and transformative area of joint speech-language modeling. 2.1 Language Modeling for Speech Processing Background, History, and Beyond • Probabilistic LMs for Speech Signals: We will begin by discussing the early foundational work in Bayesian and n-gram-based language models, which were instrumental in building probabilistic frameworks for speech recognition tasks. These models laid the groundwork for current systems by modeling word sequences and handling uncertainties in speech inputs. • Contextual End-to-End Speech Models: Next, we will explore the rise of end-to-end speech models that integrate contextual information directly into the modeling process. These models capture not only linguistic content but also paralinguistic features such as prosody, emotion, and speaker characteristics, allowing for more robust and nuanced speech understanding and generation. • Post-ASR LLM Correction and Fairness: Finally, we will examine the role of LLMs in post-ASR error correction, where language models are used to refine and correct the outputs of traditional ASR systems. Special attention will be given to the fairness and bias issues that arise in speech processing, particularly with regard to speaker variability (e.g., accent, dialect, and sociolect). We will discuss how recent advances are addressing these challenges, and the ethical implications of ensuring equitable performance across diverse speech populations. 2.2 Large Language Models for Audio, Speech, and Conversational Signals • Theoretical Basics of LLMs Adaptation : We will review some theoretical foundations of LLM adaptation and how these frameworks connect to LLMs with speech input. Topics include population risk measurement Yang et al. 2021 and model transferability estimation Chen et al. 2023b from speech models to LLM adaptation, motivating different design pipelines Radhakrishnan et al. 2023 ; Hu et al. 2024 ; Chen et al. 2024 ; Yang et al. 2023 of spoken agents. • Speech-Text Pre-training / Post-alignment : Building on this, we will examine joint text-speech pre-training Chiu et al. 2022 ; Barrault et al. 2023 ; Chen et al. 2022 methods, which have pushed the boundaries of multi-modal understanding by combining speech and text learning objectives. The application of LLMs for voice quality estimation will also be discussed, showing how these models can assess and adapt to different speaker characteristics in real-time. • Multi-task Evaluation for Voice-LLMs: Lastly, we will cover the latest advances in joint generative translation models, which integrate both speech and text modalities for seamless, high-quality translation across languages. This section will provide a comprehensive look at how state-of-the-art voice-interfaced LLMs Reid et al. 2024 ; Chu et al. 2023 ; Radford et al. 2023 are the processing and understanding of speech and conversational signals. 2.3 Spoken Dialogue Systems • Historical Foundations The last decade saw significant progress in the creation and deployment of large-scale voice-powered AI virtual assistant technology, starting with Siri (SRI/Apple), and then followed by Cortana (Microsoft), Google Assistant, Alexa (Amazon), and Viv/Bixby (Samsung). This section of the tutorial will cover the technological innovations underlying this era of AI virtual assistants including intent detection and slot filling with RNNs Mesnil et al. 2014 , dialogue management with reinforcement learning Shah et al. 2016 , leveraging web-scale query-clicks Hakkani-Tür et al. 2011 ; Tur et al. 2011 , exploiting the semantic web Heck and Hakkani-Tür 2012 , and incorporating knowledge graphs in spoken language understanding Huang et al. 2015 • Current Trends The current work in AI virtual assistants builds upon the voice-only systems of the last decade by leveraging LLMs to significantly improve the coverage and robustness of the spoken language understanding and dialogue state tracking components, in addition to substantial advancements in spoken language generation. This tutorial section provides an overview of existing LLMs and methodologies for adapting them to downstream tasks Ni et al. 2021 ; Qin et al. 2023 ; Yi et al. 2024 . It highlights recent advancements in multi-turn dialogue systems, encompassing both LLM-based open-domain dialogue (ODD) and task-oriented dialogue (TOD) systems, as well as relevant datasets and evaluation metrics. Additionally, it addresses emerging research challenges associated with the development of LLMs and the growing demands on multi-turn dialogue systems. • Future Directions Future work will build on the recent progress in LLMs and unify task-oriented and open-domain dialogue systems. This section of the tutorial will cover this technology shift and highlight some examples of promising future directions. Some examples include building new foundation LLMs pretrained with conversational data Jawale et al. 2024 , e2e training of dialogues Liu et al. 2017 ; Liu et al. 2018 , and dialogue self-play Shah et al. 2018a ; Shah et al. 2018b . Another significant future direction is situated dialogue systems: grounding LLM-based dialogue systems with content (web pages Heck et al. 2013 , lists Bapna et al. 2017 , forms Heck et al. 2024 , tables Sundar and Heck 2023 ; Sundar et al. 2024b ; Sundar et al. 2024a , papers with figures, equations, tables Sundar et al. 2024c