Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR

Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke · 2026

Read Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR by Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26246.txt.

Introduction

Kumar and co-authors study how past turns in a chat can help a language model transcribe speech. After multi-turn training, context mainly helps the model catch names and other context-bound facts. Raw past audio is costly, since its embedding grows with each turn. Their method, Abstract Compression, swaps each past turn of audio for a fixed set of learned tokens, while keeping past transcripts. Tests in and out of domain recover part of the gain at lower cost.

Catalog record

CREATORShashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37038
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26246

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR Shashi Kumar, Esaú Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Bañeras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas Stolcke Abstract Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs. 1 Introduction Automatic speech recognition (ASR) is increasingly used in settings that are inherently conversational, such as voice assistants, customer-support calls, meetings, spoken search, and multimodal agents. In these scenarios, the correct interpretation of an utterance often depends on previous turns: earlier context may introduce named entities, establish speaker-specific pronunciations, or provide discourse cues that help resolve ambiguity. Yet despite this natural dependence on prior turns, most ASR systems still process each utterance independently ( Kim and Metze 2018 ; Kim et al. 2019 ; Hori et al. 2021 ; Lee et al. 2024 ) . This limitation is particularly relevant for contextual entities such as names, locations, and domain-specific terminology. These words are often rare and error-prone. As a result, ASR systems frequently fail exactly where conversational context should be most useful ( Pundak et al. 2018 ; Williams et al. 2018 ; Jain et al. 2020 ; Tong et al. 2023 ; Zhou et al. 2024 ; Li et al. 2024 ; Liu et al. 2024 ) . Figure 1: Overview of Abstract Compression for context-aware ASR. Prior conversational turns are represented by both transcript and audio. In our implementation, the audio from each prior turn is distilled into a fixed number of latent tokens, while transcript is retained explicitly. These compressed context representations are provided to the LLM alongside the current turn’s full-resolution audio tokens. This preserves part of the conversational context while reducing the cost of raw-context conditioning. Multimodal large language models (LLMs) provide a natural framework for revisiting this problem ( Tang et al. 2024 ; Wu et al. 2023 ; Ma et al. 2024 ; Kumar et al. 2025 ; Abouelenin et al. 2025 ; Carofilis et al. 2026 ) . By mapping audio into the token space of a text-generative model, recent LLM-based ASR systems can condition on audio, text, and structured prompts within a unified autoregressive architecture. In principle, this makes it possible to transcribe the current utterance while conditioning on the preceding conversation, allowing prior turns to provide both linguistic evidence (what was said) and acoustic evidence (how it was said). However, exploiting conversational multimodal context is not straightforward. In LLM-based ASR, prior-turn audio is represented by long sequences of audio tokens, so the total prompt length grows rapidly with the number of previous turns. This may lead to high key-value (KV) cache cost, increased latency, and memory bottlenecks during inference. This raises two linked questions: Does conversational multimodal context actually improve LLM-based ASR? And can those gains be retained under a fixed and substantially smaller context budget? In this work, we study this trade-off in multimodal LLM-based ASR. We first examine whether conversational multimodal context improves LLM-based ASR. In our setting, simply prepending raw context at inference time degraded performance. After supervised fine-tuning on multi-turn inputs, however, the model does benefit from context, with the clearest gains appearing on contextual entities, as reflected in Bias-WER (defined in Section 6.2 ). This suggests that conversational context is useful, but representing raw audio is expensive. To reduce the cost of raw-context conditioning, we introduce Abstract Compression , a method that compresses the audio from prior turns into a small set of latent tokens, while retaining prior-turn transcripts in their original textual form (see Figure 1 ). This design targets the dominant source of context-token cost in LLM-based ASR, since audio is represented by long token sequences whereas transcript is comparatively compact. Across in-domain and out-of-domain evaluation, this compressed representation recovers part of the gains of raw-context conditioning while representing each prior-turn audio with a fixed number of latent audio tokens. Overall, this work makes three contributions. First, we show that multimodal context from prior turns can improve LLM-based ASR, with the largest gains appearing on contextual entities after supervised multi-turn training. Second, we introduce Abstract Compression, together with a two-stage training strategy, to represent prior-turn context with a smaller token footprint. Our experiments focus on compressing prior-turn audio, which accounts for most of the contextual cost in our setup, while preserving prior-turn transcripts explicitly. Third, we present ablation studies that clarify which aspects of the compression setup most strongly influence performance. 2 Related Work Contextual and conversational ASR. Prior work has studied how to incorporate context into ASR. Early approaches focused on contextual biasing, where external information such as contact names, locations, or domain-specific phrases is injected into decoding through weighted finite-state transducers, shallow fusion, rescoring, or related mechanisms. These methods are especially effective for rare words and named entities, which are often underrepresented in standard training data yet critical for downstream usability ( Williams et al. 2018 ; Pundak et al. 2018 ; Jain et al. 2020 ; Tong et al. 2023 ; Zhou et al. 2024 ; Li et al. 2024 ) . Conversation-level language modeling using LSTMs has been shown to benefit accuracy in hybrid neural ASR systems Xiong et al. 2018 . More recent end-to-end ASR systems incorporate contextual signals directly into the model architecture through bias encoders, attention over phrase lists, and context-aware transducer or encoder-decoder formulations ( Pundak et al. 2018 ; Jain et al. 2020 ; Tong et al. 2023 ) . A related line of work studies conversational or discourse-aware ASR, where preceding utterances are used to improve consistency across turns ( Kim and Metze 2018 ; Kim et al. 2019 ; Hori et al. 2021 ; Lee et al. 2024 ) . Our work is aligned with this motivation, but differs in two respects: we study context in a multimodal LLM-based ASR setting, and we focus not only on whether conversational context helps, but also on how to represent it efficiently when the context includes audio embeddings. LLM-based and multimodal ASR. Recent work has explored adapting large language models for speech recognition by coupling pretrained LLMs with speech encoders and projection modules, or by training multimodal foundation models that natively process audio and text in a unified autoregressive architecture ( Tang et al. 2024 ; Wu et al. 2023 ; Ma et al. 2024 ; Kumar et al. 2025 ; Abouelenin et al. 2025 ; Carofilis et al. 2026 ; Lakomkin et al. 2024 ) . These models are appealing because they combine the

Read the complete text