SAME: A Semantically-Aligned Music Autoencoder
Julian D. Parker, Zach Evans, CJ Carr, Zachary Zukowski, Matthew Rice, Jordi Pons · 2026
Read SAME: A Semantically-Aligned Music Autoencoder by Julian D. Parker, Zach Evans, CJ Carr, Zachary Zukowski, Matthew Rice, Jordi Pons (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18613.txt.
Introduction
Julian Parker and six co-authors present SAME, a 2026 meaning-aligned music autoencoder reaching 4096-fold time compression for stereo music and general audio. A transformer base mixes semantic rules, phase-aware rebuild losses, and better discriminator designs to keep rebuild quality and downstream generative power. Two versions ship in open weights: a large SAME-L and a small CPU-friendly SAME-S. The design cuts compute cost through compression and well-tuned transformer parts.
Catalog record
| CREATOR | Julian D. Parker, Zach Evans, CJ Carr, Zachary Zukowski, Matthew Rice, Jordi Pons |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32645 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18613
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SAME: A Semantically-Aligned Music Autoencoder Julian D. Parker, Zach Evans, CJ Carr, Zachary Zukowski, Matthew Rice, Jordi Pons Abstract Latent representations are at the heart of the majority of modern generative models. In the audio domain they are typically produced by a neural-audio-codec autoencoder. In this work we introduce SAME (Semantically-Aligned Music autoEncoder), an autoencoder for stereo music and general audio that reaches a 4096 × \times temporal compression ratio while maintaining reconstruction quality and downstream generative performance. We achieve this by combining a tranformer-based backbone with set of semantic regularisation approaches, phase-aware reconstruction losses and improved discriminator designs. The architecture delivers substantial computational cost benefits, through both its high compression ratio and its reliance on well-optimised transformer primitives. Two variants (a large SAME-L and a CPU-deployable SAME-S) are released in open-weights form. 1 Introduction Most modern generative media models operate on latent distributions (continuous or discrete) rather than on raw data. The paradigm was established in the image domain [ 25 ] using continuous latents from a variational autoencoder (VAE). In the audio domain these models are called Neural Audio Codecs (NACs). The dominant NAC paradigm, established by SoundStream [ 36 ] and refined by EnCodec [ 5 ] and DAC [ 16 ] , is a VQ-VAE [ 28 ] variant with convolutional encoder/decoder networks and a vector-quantised bottleneck, trained with STFT-based reconstruction and adversarial losses. The quantized tokens can then be modelled by an autoregressive generative model [ 2 , 4 ] . Diffusion and flow-matching generative models require continuous latents, so a different approach is needed. A popular example is the continuous NAC in Stable Audio Open [ 9 ] , which shares the discrete-NAC architecture and training recipe but replaces the quantized bottleneck with a VAE bottleneck. Recent continuous NACs for music and general audio have largely followed this recipe, adjusting the training objective for audio quality [ 30 , 1 ] . Several also use diffusion-model variants for decoding [ 22 , 24 , 23 ] . In the speech domain, NAC innovation has accelerated with transformer-based encoder/decoder backbones [ 21 , 31 ] , query-based resampling [ 33 ] , and alignment with semantic representations [ 38 , 6 ] or ground-truth features [ 7 ] . We present SAME (Semantically-Aligned Music autoEncoder), which synthesises and extends many of these innovations. SAME consists of: 1. A query-based transformer resampling block (the Transformer Resampling Block, TRB), enabling fast inference and scaling to large parameter counts. 2. A bottleneck regularised for generative tractability and alignment with specific semantic concepts, improving generative performance. 3. Improved multi-resolution STFT (MRSTFT) reconstruction losses and an improved discriminator design, improving audio quality. We train two variants. SAME-L is an 852M-parameter model that outperforms baselines on audio quality while being significantly faster at inference. SAME-S is a distilled 108M-parameter variant with extremely fast inference, intended for CPU use on edge devices. Weights for both are released. 1 1 1 https://stability-ai.github.io/SAME 2 Architecture SAME follows an encoder-bottleneck-decoder structure (Fig. 1 ), in which a parameter-free patching pretransform and a Transformer Resampling Block (TRB) jointly achieve the target compression ratio. \sim \sim Patch Encoder TRB Soft-Norm Bottleneck Decoder TRB Unpatch \sim \sim 2P\!\times\!\frac{T}{P} d\!\times\!\frac{T}{PS} d\!\times\!\frac{T}{PS} 2P\!\times\!\frac{T}{P} Auxillary Losses Reconstruction Losses Adversarial Losses Figure 1: SAME architecture and training losses. Total compression: PS\!\times . Dashed boxes indicate loss components. 2.1 Patching Pretransform Stereo audio waveforms of shape (B,2,T) are partitioned into non-overlapping patches of P samples per channel and reshaped to (B,2P,T/P) , so each embedding is a 2P -dimensional vector concatenating the P left-channel and P right-channel samples. With P{=}256 this gives 256\times temporal downsampling with no learned parameters. Decoding applies the inverse reshape. Gradients flow through the transform, so the encoder and decoder train end-to-end against the original waveform. 2.2 Transformer Resampling Blocks The Transformer Resampling Block (TRB) performs temporal resampling through self-attention rather than strided convolution or pooling, an approach shown to work well in image and speech domains [ 35 , 33 ] that builds on earlier cross-attention-based resampling [ 14 ] . A TRB operates in either encoder or decoder mode, depending on the direction of resampling. Both modes share the same structure: a sequence of input embeddings is interleaved with learnable output embeddings, processed by a stack of transformer layers, and the output embeddings are extracted as the resampled representation. A stride parameter S controls the resampling ratio. In encoder mode, the TRB downsamples by S . A weight-normalised linear projection maps patch embeddings to the transformer’s internal dimension. The result is partitioned into N=\lceil T/S\rceil non-overlapping segments of S embeddings each. A single learnable output embedding (initialised near zero and perturbed with low-amplitude Gaussian noise) is appended to each segment, forming subsequences of length S{+}1 . The interleaved sequence is processed by D transformer layers. The output embedding is then extracted from each subsequence, yielding an S -fold reduction in temporal resolution. A linear projection then maps to the desired latent dimension. Fig. 2 illustrates encoder-mode interleaving with S{=}2 . In decoder mode, the TRB upsamples by S . Each input embedding is paired with S learnable output embeddings, each perturbed with Gaussian noise, forming length- (S{+}1) subsequences in which the roles are reversed: the input provides context and the transformer populates the S outputs. After the stack, the outputs are extracted and the latent embeddings are discarded, yielding an S -fold increase in temporal resolution. A weight-normalised linear projection maps from the transformer dimension back to the patch embedding dimension. x_{0} x_{1} q x_{2} x_{3} q x_{4} x_{5} q x_{6} x_{7} q Seg. 1 Seg. 2 Seg. 3 Seg. 4 Transformer \mathcal{T}_{1} \mathcal{T}_{2} \mathcal{T}_{3} \mathcal{T}_{4} \cdots \mathcal{T}_{D} x_{0} x_{1} y_{0} x_{2} x_{3} y_{1} x_{4} x_{5} y_{2} x_{6} x_{7} y_{3} extract y embeddings (discard x ) Figure 2: Embedding interleaving in encoder-mode TRB (stride S{=}2 ). Each transformer layer is a pre-norm residual block. Self-attention uses differential attention [ 34 ] with per-head QK-normalisation and rotary position embeddings (RoPE) [ 27 ] . Both normalisation sites use Dynamic Tanh (DyT) [ 41 ] , a learnable \tanh(\alpha\cdot x) plus affine transformation that replaces LayerNorm/RMSNorm; DyT avoids the per-batch-element-statistics issues caused by silence or low-level noise in the input [ 21 ] . The feed-forward network is a gated linear unit (GLU) with SiLU activation. In the decoder, the final K of D layers use a sinusoidal activation f(x){=}\sin(\pi x) instead, providing a periodic basis suited to reconstructing waveform-level detail [ 16 ] . All branch outputs are zero-initialised so each layer starts as an identity. An audio autoencoder must handle variable-length audio and is usally trained on very short sequences (5s or less), so standard attention (causal or not) is unsuitable. We use one of two strategies (Fig. 3 ). Sliding-window attention lets each embedding attend to a fixed number of neighbours on each side, giving linear complexity in sequence length and bounding the receptive field for length generalisation. This is the preferred option. However, sliding-window attention is not supported by current CPU-inference libraries (e.g. LiteRT [ 11 ] ), making chunked attention necessary for CPU deployment. Chunked attention folds the sequence into fixed-size chunks processed independently. Hard chunk boundaries can produce audible artefacts at transitions. We