q2d2: a geometry-aware audio codec leveraging two-dimensional quantization

Eliya Nachmani, Tal Shuster · 2025

Read q2d2: a geometry-aware audio codec leveraging two-dimensional quantization by Eliya Nachmani, Tal Shuster (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01537.txt.

Introduction

Eliya Nachmani and Tal Shuster build a new audio codec for speech and sound. Neural codecs now rebuild sound with high quality. Most squeeze sound with stacked vector codes. Those codes limit the shape of the latent space. Correlations between traits are then hard to keep.

Their codec, q2d2, uses two-dimensional quantization instead. That form keeps richer geometry in the latent space. Richer form should yield better sound at the same bit rate. Better codecs cut bandwidth for calls, streams, and stored voice data. Two-dimensional codes also ease streaming, since partial sound can be rebuilt early while later bits are still on their way.

Catalog record

CREATOREliya Nachmani, Tal Shuster
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01537

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

q2d2: a geometry-aware audio codec leveraging two-dimensional quantization Eliya Nachmani, Tal Shuster Abstract Recent neural audio codecs have achieved impressive reconstruction quality, typically relying on quantization methods such as Residual Vector Quantization (RVQ), Vector Quantization (VQ) and Finite Scalar Quantization (FSQ). However, these quantization techniques limit the geometric structure of the latent space, make it harder to capture correlations between features leading to inefficiency in representation learning, codebook utilization and token rate. In this paper we introduce Two-Dimensional Quantization (Q2D2), a quantization scheme in which feature pairs are projected onto structured 2D grids—such as hexagonal, rhombic, or rectangular tiling—and quantized to the nearest grid values, yielding an implicit codebook defined by the product of grid levels, with codebook sizes comparable to conventional methods. Despite its simple geometric formulation, Q2D2 improves audio compression efficiency, with low token rates and high codebook utilization while maintaining state of the art reconstruction quality. Specifically, Q2D2 achieves competitive to superior performance in various objective and subjective reconstruction metrics, across extensive experiments in speech domain compared to state of the art models. Comprehensive ablation studies further confirm the effectiveness of our design choices. 1 introduction In recent years, Large Language Models (LLMs) ( Brown et al. 2020 ) have demonstrated remarkable progress in audio generation tasks, ranging from multi-speaker speech synthesis ( Wang et al. 2023 ; Kharitonov et al. 2023 ; Jiang et al. 2023 ; Ji et al. 2024a ) to music generation ( Agostinelli et al. 2023 ) and general-purpose audio synthesis ( Kreuk et al. 2022 ) . At the same time, growing attention has been devoted to incorporating speech as a modality within large multimodal systems, as seen in models such as SpeechGPT ( Zhang et al. 2023a ) , AnyGPT ( Zhan et al. 2024 ) , GPT-4o, GPT-5, and Moshi ( Défossez et al. 2024 ) . A key enabler of these advances has been the use of discrete acoustic representations produced by neural codecs ( Zeghidour et al. 2021 ; Défossez et al. 2022 ; Kumar et al. 2023 ; Ji et al. 2024b ) . By converting high-rate speech signals into compact sequences of discrete tokens, acoustic codec models provide the crucial link between continuous audio and token-based language models, thereby enabling the direct application of LLM architectures to audio. Most end-to-end discrete codec models ( Défossez et al. 2022 ; Wu et al. 2023 ) adopt a three-stage structure consisting of an encoder, a RVQ module ( Lee et al. 2022 ) , and a decoder. The encoder performs downsampling of the audio signal in the time domain to obtain compressed audio frames. Each compressed audio frame is then quantized by a series of quantizers, with each quantizer operating on the residual of the previous one. The number of quantizers determines the overall bitrate. The decoder, on the other hand, performs upsampling in the time domain to reconstruct the audio signal from the quantizer outputs. Existing acoustic codec models ( Kumar et al. 2023 ; Défossez et al. 2022 ; Siuzdak 2023 ) demonstrate impressive reconstruction quality, and generative models based on discrete codecs are now capable of synthesizing speech at near-human levels. In response ( Ji et al. 2024b ) proposed a much simpler design: instead of stacked RVQ, it uses a single VQ layer ( Gray 1984 ) over features, showing that efficient tokenization can be achieved without deep quantizer hierarchies. Additional models have contributed to the expansion of the codec landscape. Some models ( Pan et al. 2024 ; Yang et al. 2023 ; Zhang et al. 2023b ) enhanced robustness, controllability, and synthesis quality through architectural and training innovations, while other models ( Li et al. 2024 ; Liu et al. 2024 ; Xin et al. 2024 ) aimed for universality and scalability, either by unifying audio and speech tasks under a single tokenizer or by increasing codec capacity. Complementary efforts refined training strategies, with stronger discriminators advancing adversarial learning ( Ahn et al. 2024b ; Ahn et al. 2024a ) . Despite these successes, existing quantization schemes based on Vector Quantized-Variational AutoEncoder (VQ-VAE) and RVQ are challenging to optimize, and leads to well-documented problem of underutilized codebooks ( Łańcucki et al. 2020 ; Takida et al. 2022a ; Dhariwal et al. 2020 ; Huh et al. 2023a ) as the codebook size is increased, many codewords will be unused. Subsequent works aimed to improve this with various tricks such as reinitializing the entire codebook or some codewords ( Dhariwal et al. 2020 ; Łańcucki et al. 2020 ) , stochastic formulations ( Takida et al. 2022a ) , etc. At the other extreme, FSQ architecture ( Mentzer et al. 2024 ) offered a strikingly simple alternative: each latent channel is quantized independently onto a fixed set of scalar levels, forming an implicit codebook given by the product of these sets. FSQ avoids collapse by design and guarantees high codebook utilization. However, because it quantizes each feature dimension separately, it results one-dimensional isolated quantization per channel, thereby being less effective in capturing correlations between feature dimensions. Hexagon Grid (a) x y \hat{z} Rectangle grid (b) x y \hat{z} Rhombic grid (c) x y \hat{z} Figure 1: Visualization of quantization grids used in Q2D2: Hexagonal Grid (a): a hexagonal tiling with 9 quantization levels in x and y axis. Rectangle Grid (b): a rectangle tiling with 7 quantization levels in x and y axis. Rhombic Grid (c): a rhombic tiling with 7 quantization levels in x axis, and 6 levels yielding to 11 quantization levels in y axis. Our motivation is to retain the simplicity and utilization benefits of FSQ, while enriching the representational capacity of discrete audio codes. In particular, we ask: can we capture correlations between channels without reintroducing the instability and inefficiency of high-dimensional vector quantization? Our approach is to move beyond simple 1D scalar grids by introducing structured 2D geometric tilings, and extending further to 3D geometric tilings in future work (Appendix E ). In this paper, we introduce Q2D2 , a geometry-aware quantization scheme capable of reconstructing speech with low token rate. Instead of quantizing each latent channel independently, Q2D2 groups channels into pairs and maps them onto structured two-dimensional grids. Each pair is snapped to the nearest grid point from a fixed tiling (e.g., hexagonal, rhombic, or rectangle), producing an implicit codebook defined by the product of all pairwise grids. To interface with neural encoders, Q2D2 introduces lightweight linear projections into and out of the quantization space. Quantization itself is implemented with Straight-Through gradient Estimators (STE), and per-pair grid construction, ensuring differentiability, stability, and flexibility. This design achieve high utilization and robustness while introducing geometric structure that captures correlations between features and expands the expressive capacity of discrete codes. Our contributions can be summarized as follows: 1. Conceptual Contributions. We introduce Q2D2 a novel approach of compressing the quantizer layers of acoustic codes models to a geometry-aware quantizer that groups channels into pairs and jointly quantizes them on utilizing complex geometric structures for the first time, enhancing semantic information of the codec and captures correlations between feature dimensions. Q2D2 supports per-pair quantization, levels selection, dimension selection, projection layers, and straight-through estimators, enabling efficient end-to-end training. 2. Methodological

Read the complete text