SpectroStream: A Versatile Neural Codec for General Audio
Yunpeng Li, Kehang Han, Brian McWilliams, Zalán Borsos, Marco Tagliasacchi · 2025
Read SpectroStream: A Versatile Neural Codec for General Audio by Yunpeng Li, Kehang Han, Brian McWilliams, Zalán Borsos, Marco Tagliasacchi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05207.txt.
Introduction
Yunpeng Li and four co-authors introduce in 2025 SpectroStream, a neural codec that compresses general audio while keeping it sounding whole. It succeeds their earlier SoundStream codec, which handled only single-channel sound up to a lower fidelity ceiling.
SpectroStream works in the time-frequency domain, a view that pays off at high sample rates, and fuses channels with a delay so stereo stays coherent. It rebuilds full-band forty-eight kilohertz stereo music at just four to sixteen kilobits per second, telephone-grade data rates for concert-grade sound. Such codecs underpin streaming, calls, and the audio side of generative models.
Catalog record
| CREATOR | Yunpeng Li, Kehang Han, Brian McWilliams, Zalán Borsos, Marco Tagliasacchi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-05207
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpectroStream: A Versatile Neural Codec for General Audio Yunpeng Li, Kehang Han, Brian McWilliams, Zalán Borsos, Marco Tagliasacchi Abstract We propose SpectroStream, a full-band multi-channel neural audio codec. Successor to the well-established SoundStream, SpectroStream extends its capability beyond 24 kHz monophonic audio and enables high-quality reconstruction of 48 kHz stereo music at bit rates of 4–16 kbps. This is accomplished with a new neural architecture that leverages audio representation in the time-frequency domain, which leads to better audio quality especially at higher sample rate. The model also uses a delayed-fusion strategy to handle multi-channel audio, which is crucial in balancing per-channel acoustic quality and cross-channel phase consistency. † † address: Google DeepMind {yunpeng,kehanghan,bmcw,zborsos,mtagliasacchi}@google.com 1 Introduction A high-fidelity audio codec with a compact representation is crucial for audio compression and generation. The development of SoundStream [ 1 ] was a key development that unlocked the potential of autogressive audio generation with standard language-modeling objectives [ 2 ] . Since then, many new codecs have been proposed that pushed the frontier in fidelity and compactness [ 3 , 4 , 5 , 6 , 7 ] . However, most focus on speech [ 3 , 5 , 6 ] or limit their attention to single-channel audio [ 4 , 7 ] . This makes them either unsuitable or suboptimal for more general audio such as music, which is typically presented in a stereo format. In this paper we propose SpectroStream , a neural codec that can encode full-band 48 kHz general audio with joint modeling of multiple audio channels. Unlike SoundStream [ 1 ] , SpectroStream models the audio in the time-frequency domain with a 2D-convolutional encoder-decoder architecture. For multi-channel audio, it employs a delayed fusion and early splitting strategy in the encoder and the decoder, respectively, allowing audio channels to be processed independently in some parts of the model but jointly in other parts (see Figure 1 ). This leads to more efficient encoding and decoding of multi-channel audio, as well as improved consistency across channels. Moreover, SpectroStream uses causal convolutions with a small look-ahead in its encoder and decoder. This design choice, supported by the KWS streaming framework [ 8 ] , allows it to perform real-time streaming inference with only a small architectural latency. This can be readily achieved on a single desktop CPU, without needing specialized accelerators. 1.1 Related Work SoundStream [ 1 ] is a convolutional encoder-decoder model based on SEANet [ 9 ] architecture. The latent representation in the bottleneck is quantized using a residual vector quantizer (RVQ). Aside from being an efficient compressed representation for general audio, SoundStream has shown remarkable success as a representation for generating speech, general audio [ 2 , 10 ] and high-fidelity polyphonic music [ 11 , 12 ] . SoundStream is trained using a combination of a waveform reconstruction loss and an adversarial loss. Encodec [ 3 ] and Descript Audio Codec (DAC) [ 4 ] propose methods similar to SoundStream but with modifications to the architectures and training objectives. Encodec uses a multi-scale STFT discriminator and an additional relative feature matching loss on the generator. DAC combines a similar discriminator with an improved RVQGAN architecture and adds a multi-scale reconstruction loss on the mel spectrogram. Mimi [ 5 ] takes a different approach, as it was designed for generation tasks from the outset. Mimi combines the RVQ structure of SoundStream-like methods, and it additionally distills non-causal semantic information into the first level of the RVQ code. This modification is motivated by the insight that many generative models of speech combine both SoundStream-like acoustic tokens with more semantic representations to provide the model with improved linguistic capabilities. 2 Model Architecture Figure 1: Encoder (left) and decoder (right) architecture. The model in our experiment has base convolution depths of C_{e}=32 for the encoder and C_{d}=64 for the decoder. SpectroStream consists of an encoder, a decoder, and a quantizer. Additionally, a discriminator is used solely for adversarial learning, which we will describe in Section 3 . The input audio is first converted into spectrograms by applying short-time Fourier transform (STFT) to each of its stereo channels. The real and imaginary components of the complex output are treated as the channel dimension of the input tensor to the encoder network. The encoder’s output embedding vectors are quantized into integer tokens using residual vector quantization (RVQ) [ 1 ] to create a compact discrete representation of the input audio. At decoding time, the tokens are dequantized back into embeddings and inverse STFT is applied to the decoder output at the end to recreate the time-domain waveform. The discriminator works the same way as the encoder except that it also receives the modulus of the STFT spectrogram as input, along with the real and imaginary values. Therefore, the discriminator input tensor has three channels (for each input audio channel) as opposed to just two for the encoder. The encoder, the decoder, and the discriminator are all 2D convolutional networks that process the signal as time-frequency spectrograms. The encoder and the discriminator use strided convolutions to down-sample their inputs, while the decoder mirrors the encoder and employs transposed convolutions for up-sampling. The encoder employs delayed fusion : The early layers of the encoder process each input audio channel separately, with shared layer parameters but no inter-channel interaction. The intermediate tensors for the audio channels are fused just before the final few layers, which then process them jointly to produce joint embeddings for all audio channels. The delayed fusion scheme is again mirrored in the decoder (as early splitting) and emulated in the discriminator. Finding the right fusion point is crucial for good compression and reconstruction: Fusing too early results in poor acoustic quality, as such early interactions between channels can distract the model from fidelity and realism objectives; whereas fusing too late makes the reconstructed audio lose phase consistency due to insufficient joint processing, and sound unfocused and directionless. Figure 2: Discriminator architecture. The number of frequency bins F is half of the STFT frame length (keeping the DC and omitting the Nyquist bin), which is equal to the STFT frame step size as we use a 2x overlapping factor. We use C_{0}=32 for all discriminators in our experiments. The full encoder and decoder architecture for stereo audio is given in Figure 1 , and the discriminator is shown in Figure 2 . We use weight normalization [ 13 ] in the convolution layers of the encoder and the decoder, and layer normalization [ 14 ] in the discriminator. Following the original formulation of [ 14 ] , layer normalization is applied after convolution (before activation) and computed over all non-batch tensor dimensions. The encoder and the decoder use causal convolutions to ensure minimal latency for real-time inference. The discriminator, on the other hand, uses regular convolutions with symmetric padding, since it is not needed at inference time and so its look-ahead latency is not a consideration. All convolutions are pre-activated – that is, preceded by an activation function – with the exception of the very first layer in the network. We use ELU [ 15 ] in the encoder/decoder, and Leaky ReLU [ 16 ] with a slope of 0.2 in the discriminator. For