Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs

Dingdong Wang, Junan Li, Mingyu Cui, Dongchao Yang, Xueyuan Chen, Helen Meng · 2025

Read Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs by Dingdong Wang, Junan Li, Mingyu Cui, Dongchao Yang, Xueyuan Chen, Helen Meng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17863.txt.

Introduction

Dingdong Wang and co-authors settle a design choice for speech language models: feed the model chopped sound tokens or smooth sound features, in a 2025 paper. Both routes have strong results behind them, but no fair race had been run. The team compared self taught versions of each under matched settings across six understanding tasks and two sizes of language model.

Smooth features won most tasks overall. Layer by layer probes showed the two routes learn unlike patterns inside both the speech encoder and the language model, and they differ in toughness as well. The authors map the trade offs for future speech systems.

Catalog record

CREATORDingdong Wang, Junan Li, Mingyu Cui, Dongchao Yang, Xueyuan Chen, Helen Meng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17863

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs Dingdong Wang, Junan Li, Mingyu Cui, Dongchao Yang, Xueyuan Chen, Helen Meng Abstract With the rise of Speech Large Language Models (SpeechLLMs), two dominant approaches have emerged for speech processing: discrete tokens and continuous features. Each approach has demonstrated strong capabilities in audio-related processing tasks. However, the performance gap between these two paradigms has not been thoroughly explored. To address this gap, we present a fair comparison of self-supervised learning (SSL)-based discrete and continuous features under the same experimental settings. We evaluate their performance across six spoken language understanding-related tasks using both small and large-scale LLMs (Qwen1.5-0.5B and Llama3.1-8B). We further conduct in-depth analyses, including efficient comparison, SSL layer analysis, LLM layer analysis, and robustness comparison. Our findings reveal that continuous features generally outperform discrete tokens in various tasks. Each speech processing method exhibits distinct characteristics and patterns in how it learns and processes speech information. We hope our results will provide valuable insights to advance spoken language understanding in SpeechLLMs. 1 Introduction Learning speech representations that are robust and effective is a key challenge in modern speech processing systems. Recent advances in Speech Large Language Models (SpeechLLMs), especially systems such as GPT-4o, have captured significant attention in the speech domain ( Ji et al. 2024 ; Arora et al. 2025 ; Cui et al. 2025 ; Peng et al. 2025 ; Yin et al. 2024 ; Caffagni et al. 2024 ) . With the advancement of SpeechLLMs, two main types of speech representations are employed as inputs: continuous features and discrete speech tokens. Using speech continuous features is an intuitive approach ( Chu et al. 2023 ; Chen et al. 2023 ; Gong et al. 2023a ; Wang et al. 2023a ; Tang et al. 2023 ; Gong et al. 2023b ; Wang et al. 2025 ) for integrating speech signals with large language models (LLMs). This type of representation is typically obtained from certain layers of a self-supervised learning (SSL) model or a speech encoder module. Specifically, in SLAM-ASR ( Ma et al. 2024 ) , continuous features are obtained from the final layer of the HuBERT model. In models like the Qwen-Audio series ( Chu et al. 2023 ; Chu et al. 2024 ) and SALMONN ( Tang et al. 2023 ) , raw speech is transformed into high-dimensional embeddings using the Whisper encoder ( Radford et al. 2023 ) and adapted for LLMs through an adapter module. These continuous features preserve the audio signal’s richness and have shown strong performance in speech understanding ( Ji et al. 2024 ) . However, since autoregressive LLMs like GPT ( OpenAI 2023 ) and LLaMA ( Dubey et al. 2024 ) are naturally designed to work with discrete text tokens, this has inspired researchers to explore representing speech as sequences of discrete tokens. Recent interest in speech discrete tokens has grown rapidly, driven by their compatibility with large language models that operate on text tokens. Unlike continuous features, which require a speech adapter for modality alignment, discrete tokens can be directly integrated into the LLM’s vocabulary. Several works have used discrete tokens in SpeechLLMs ( Zhang et al. 2023a ; Rubenstein et al. 2023 ; Wang et al. 2023b ; Wang et al. 2024 ; Mitsui et al. 2024 ; Nguyen et al. 2024 ; Défossez et al. 2024 ) . For instance, both AudioPaLM ( Rubenstein et al. 2023 ) and SpeechGPT ( Zhang et al. 2023a ) use K-means clustering to discretize speech embeddings. More recent works, such as Mini-Omni series ( Xie and Wu 2024a ; Xie and Wu 2024b ) and GLM-4-Voice ( Zeng et al. 2024 ) , they explore tokenizer modules based on compression methods, leveraging encoder-decoder architectures with residual vector quantization (RVQ) ( Zeghidour et al. 2021 ) to extract discrete tokens. Continuous-feature-based and discrete-token-based SpeechLLMs have each demonstrated promising performance across various speech-related tasks ( Ji et al. 2024 ; Arora et al. 2025 ) . However, despite their success, there has been no systematic and fair comparison between these two types of representation within the spoken language understanding (SLU) area. Intuitively, discrete tokens compress information through quantization, sacrificing precision for compactness, while continuous representations retain fine-grained acoustic and temporal details, which may affect the performance of discrete tokens when compared to continuous features. However, the extent and nature of the potential performance differences remain unclear. Moreover, in multimodal settings, it is still largely unknown whether LLMs interpret these two forms of speech input differently, and how such differences affect various downstream SLU-related task outcomes. Clarifying these distinctions is crucial for understanding each paradigm’s limitations and guiding future advancements in architecture, tokenizer design, and training methodologies. To address this gap, we present a comprehensive benchmark comparing continuous features and discrete tokens under consistent experimental conditions. Our experiments utilize self-supervised learning (SSL) models for speech feature extraction, leveraging K-means clustering for discretization and using original embeddings as continuous features. This approach provides a straightforward way to compare these two paradigms. Specifically, we select HuBERT-Large ( Hsu et al. 2021 ) and WavLM-Large ( Chen et al. 2022 ) as SSL models, and Qwen1.5-0.5B ( Bai et al. 2023 ) and Llama3.1-8B ( Dubey et al. 2024 ) as LLM decoders. We aim to investigate the performance differences between discrete tokens and continuous features in spoken language understanding, evaluating both representations across a range of tasks, including automatic speech recognition, phoneme recognition, speech translation intent classification, keyword spotting, and emotion recognition. Beyond benchmark comparison, we further conduct detailed analysis from multiple perspectives, including efficiency, SSL and LLM layer behaviour, and robustness. Across a range of tasks, our results show that continuous features generally outperform discrete tokens, with the magnitude of this gap varying by task. Interestingly, discrete tokens perform better on phonetic-level tasks, indicating their strength in capturing subword-level structure. Our SSL and LLM layer analyses also reveal distinct learning patterns for each representation across the model hierarchy. Each representation paradigm also demonstrates unique strengths: discrete tokens excel in data compactness and training efficiency, while continuous features offer superior robustness in diverse conditions. Our key contributions are as follows: • We present the first comprehensive benchmark comparing speech continuous features and discrete tokens for spoken language understanding (SLU) under consistent experimental conditions. • We conduct an in-depth analysis of both representations, examining efficiency, robustness, and learning dynamics across SSL and LLM layers. • We identify the complementary strengths of each representation, offering practical guidance for future SpeechLLMs development. 2 Pipeline Design Figure 1: Architectures of two approaches for integrating speech into Large Language Models (LLMs). (Left) discrete token-based encoding. (Right) continuous feature processing. To systematically compare the performance of continuous and discrete tokens in SLU related tasks, we adopt two widely used ( Chu et al. 2024 ; Xu et al. 2025 ; Gong et al. 2023a ; Wang et al. 2023a ; Zhang et al. 2023a ; Shon et al. 2024 ; Dekel and Fernandez 2024 ) speech processing pipelines, as illustrated in Fig. 1 : (a) a discrete token pipeline and (b) a continuous features pipeline. 2.1 Speech Discretization In our approach, we utilize SSL models

Read the complete text