HybridCodec: Modeling Discrete and Continuous Representations For Efficient Speech Language Models
Artem, Francesco, Samir, Mirco · 2026
Read HybridCodec: Modeling Discrete and Continuous Representations For Efficient Speech Language Models by Artem, Francesco, Samir, Mirco (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27627.txt.
Introduction
Speech systems often turn audio into discrete tokens, short symbols that a language model can handle. Squeezing speech into symbols throws away detail, though, and quality suffers. Ploujnikov, Verdini, Sadok, and Ravanelli propose HybridCodec in 2026. It pairs compressed discrete tokens with continuous residuals. Residuals are leftover smooth signals that carry what the symbols missed.
The design joins a hybrid codec with a hybrid Transformer. The Transformer predicts the discrete stream step by step. A separate path predicts the residuals without step-by-step generation. Tests show the approach keeps far more of a speaker's character than discrete-only methods. The method also needs fewer step-by-step predictions. Its frame rate is extremely low, about six frames per second. The combined model beat a discrete-only baseline on synthesis quality and clarity. It also lowered error rates on recognition tasks.
Catalog record
| CREATOR | Artem, Francesco, Samir, Mirco |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27627
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HybridCodec: Modeling Discrete and Continuous Representations For Efficient Speech Language Models Artem, Francesco, Samir, Mirco Abstract Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilities into Large Language Models (LLMs). However, numerous studies report performance degradation on various downstream tasks due to information loss during discretization. To address this, we propose a novel approach combining temporally compressed discrete tokens with dimensionality-reduced continuous residuals. Our framework consists of a hybridized discrete-continuous focal modulation codec and a hybrid Transformer. This architecture performs autoregressive inference in the discrete domain, coupled with non-autoregressive prediction and continuous residual upsampling. Experimental results show that our approach significantly improves the retention of speaker characteristics compared to discrete-only methods, while simultaneously reducing the number of required autoregressive steps. Ploujnikov Verdini Sadok Ravanelli keywords speech recognition, speech synthesis, text-to-speech, audio representations, neural audio codecs. † † address: 1 Mila, Quebec AI Institute, Canada; 2 Concordia University, Canada; 3 Sapienza University of Rome, Italy; 4 Inria, Université Grenoble Alpes CNRS, LJK, France † † email: [email protected], [email protected], [email protected], [email protected] 1 Introduction The human mind processes the world through a complex interplay of discrete categories and continuous spectra [ 26 , 15 ] . Human language perfectly illustrates this duality. It imposes a clear discrete hierarchy (sequences of phonemes forming words and sentences captured in alphabets or logographic systems) onto a rich modulation of continuous characteristics, such as pitch, tone, emotion, and prosody. The advent of the Transformer architecture [ 36 ] established discrete token sequences as the de facto medium for modern artificial intelligence. This paradigm, which drives autoregressive generation and Large Language Models (LLMs) [ 5 , 11 , 2 ] , was subsequently adapted for the audio domain. Pioneering architectures like the Vector Quantized Variational Autoencoder (VQ-VAE) [ 35 ] demonstrated that continuous information can be effectively compressed into a discrete latent space, motivating the development of neural audio codecs (NACs) [ 22 , 9 , 40 , 16 ] . Fundamentally, a NAC comprises an encoder, a vector quantizer, and a decoder that map continuous audio into low-bitrate discrete tokens and back to the waveform. Unlike traditional codecs such as MP3 that rely on algorithmic signal processing and psychoacoustics, NACs learn a finite or variable data-driven vocabulary of sounds. This allows them to achieve extreme compression rates while preserving rich semantic and acoustic features, effectively bridging the gap between raw signal processing and natural language modeling by enabling LLMs to process speech as natively as text. Models like AudioLM (semantic and acoustic modeling) [ 4 ] , VALL-E (zero-shot voice cloning) [ 37 ] , and SpeechGPT (cross-model speech-text LLM) [ 44 ] have successfully leveraged these discrete audio tokens to drive significant breakthroughs in zero-shot speech synthesis and end-to-end multimodal dialogue. Despite their advantages, fully discrete representations introduce an inherent quantization penalty. As evidenced by Benchmarks (e.g., SUPERB [ 42 ] , DASB [ 23 ] ) and recent comparative surveys [ 22 , 38 , 14 ] highlight a fundamental trade-off: while discrete tokens facilitate stable convergence and seamless LLM integration, the quantization process irreversibly discards fine-grained acoustic details. Fundamentally, this loss stems from the classic rate-distortion trade-off [ 7 , 32 ] . At low bit-rates, NACs prioritize semantic content intelligibility over acoustic richness, lacking the bandwidth to encode micro-prosody and speaker timbre [ 22 ] . To mitigate this limitation, we propose a novel hybrid paradigm in which the codec supports optional refinements through high-frame-rate continuous residuals, and an LM can start with a lossy, low-resolution approximation and then compute a one-step continuous refinement, vastly reducing the total number of forward passes required in inference. Our main contributions are as follows: (1) HybridCodec , a novel NAC framework extending FocalCodec [ 19 , 10 ] , which jointly extracts time-reduced discrete tokens and models the remaining information as dimensionality-reduced continuous residuals; (2) HybridLM , a decoder-only Transformer [ 36 ] designed to process these hybrid representations. It unifies efficient, low-frame rate autoregressive (AR) prediction for discrete tokens with a single-step non-autoregressive (NAR) prediction and continuous residual upsampling; (3) A unified framework that leverages the HybridLM architecture to effectively handle major downstream speech tasks, including ASR and TTS, within a single framework. Figure 1 : Overview of the proposed architecture: HybridCodec (left) provides dual-path discrete-continuous compression, and HybridLM (right) unifies these representations through interleaved autoregressive and non-autoregressive decoding. This hybrid paradigm restores the fine-grained information lost in discrete LMs. Experimental results on LibriTTS [ 43 ] dataset showed that our approach significantly outperforms discrete baselines, especially at extremely low frame rates such as 6.25 Hz, while substantially reducing AR steps. 2 Related Work Recent work has been responding to the discrete-continuous performance gap with further task-specific analysis and a variety of adaptations. Studies in ASR [ 41 ] confirm the gap, showing that such an information bottleneck directly degrades downstream performance by stripping the signal of its prosodic nuance and speaker identity. To overcome this limitation, recent literature has explored re-integrating continuous features through diffusion mechanisms, continuous autoregressive modeling, or masked modeling [ 39 , 34 , 30 ] . However, these approaches remain highly task-specific, sacrificing the unified, generalizable framework that discrete LLMs provide. This limits their ability to handle diverse speech applications (like generation and recognition) within a single model. Discrete-continuous hybridization has been successfully explored in other domains, such as RL and robotics [ 18 , 12 , 45 ] , text diffusion [ 27 ] , and others. This raises a critical question: is it possible to design a unified language model that leverages the efficiency of discrete tokens while restoring the rich acoustic nuances of continuous speech? To the best of our knowledge, our approach is the first to unify discrete and continuous refinement within a single Transformer architecture. By leveraging these two domains, we achieve high-fidelity speech synthesis at ultra-low frame rates. 3 Model Architecture 3.1 Preliminaries: The FocalCodec Architecture FocalCodec [ 19 ] employs an asymmetric VQ-VAE architecture centered around a compressor-quantizer-decompressor bottleneck. It uses the first six layers of a pretrained WavLM as a base encoder to extract jointly acoustic and semantic features. Its core pipeline relies on focal modulation : a Focal Encoder (FE) (compressor) downsamples these continuous features into a compact latent space in linear time by aggregating multi-scale global and local contexts (noted as \mathbf{x}_{\text{base}} ). The representations are then discretized using Binary Spherical Quantization (BSQ) [ 46 ] , a lookup-free approach that enforces bounded quantization errors and maximizes codebook utilization. Then, a Focal Decoder (FD) (decompressor) mirrors the downscaling process to upsample the discrete tokens, and explicitly reconstruct the original continuous WavLM representations. Finally, a lightweight Vocos decoder [ 33 ] synthesizes the audio waveform directly from these restored continuous features. 3.2 HybridCodec: Extracting Hybrid Representations The HybridCodec, shown in Figure 1 (left), extends FocalCodec [ 19 ] by adding a secondary pathway. This branch, consisting of an additional focal encoder and decoder, captures and compresses the continuous residual information lost during discretization. Encoding: Dual-Path Feature Extraction. The encoding process maps the base representations, \mathbf{x}_{\text{base}}\in\mathbb{R}^{T\times d_{\text{}}} , into a dual discrete-continuous latent space. First, the discrete pathway (highlighted in red in Fig. 1 )