Fast Text-to-Audio Generation with Adversarial Post-Training
2025
Read Fast Text-to-Audio Generation with Adversarial Post-Training (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08175.txt.
Introduction
Zachary Novack and co-authors present a fast way to turn text prompts into sound, in work from 2025. Text-to-audio models make rich clips but run too slowly for many creative uses. The usual speed fix, shrinking a big model into a small one, is costly to run.
Their post-training method, ARC, pits the model against a critic in a contest setup and adds a contrast goal that rewards staying on prompt. It needs no teacher model. Paired with other speed tweaks on an open audio model, the result makes about twelve seconds of stereo sound in a blink on a top chip. A phone-class device made about seven seconds at a similar pace.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08175
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fast Text-to-Audio Generation with Adversarial Post-Training Abstract Text-to-audio systems, while increasingly performant, are slow at inference time, thus making their latency unpractical for many creative applications. We present Adversarial Relativistic-Contrastive (ARC) post-training, the first adversarial acceleration algorithm for diffusion/flow models not based on distillation. While past adversarial post-training methods have struggled to compare against their expensive distillation counterparts, ARC post-training is a simple procedure that (1) extends a recent relativistic adversarial formulation to diffusion/flow post-training and (2) combines it with a novel contrastive discriminator objective to encourage better prompt adherence. We pair ARC post-training with a number optimizations to Stable Audio Open and build a model capable of generating ≈ \approx 12s of 44.1kHz stereo audio in ≈ \approx 75ms on an H100, and ≈ \approx 7s on a mobile edge-device, the fastest text-to-audio model to our knowledge. Zachary Novack 1,2∗† , Zach Evans 2∗ , Zack Zukowski 2 , Josiah Taylor 2 , CJ Carr 2 , Julian Parker 2 Adnan Al-Sinan 3 , Gian Marco Iodice 3 , Julian McAuley 1 , Taylor Berg-Kirkpatrick 1 , Jordi Pons 2 1 UC – San Diego, 2 Stability AI, 3 Arm, ∗ Shared authorship, † Work done while an intern at Stability AI 1 Introduction Despite the recent progress building generative text-to-audio models, such systems require seconds to minutes per generation [ 1 , 2 , 3 , 4 ] , which limits their usability for most creative use cases. Our goal is to accelerate gaussian flow-based models, i.e. , diffusion models [ 5 ] or the more recent rectified flows [ 6 ] . Since gaussian flow models can be costly at inference time due to their iterative (step-based) sampling [ 7 ] , considerable effort has gone into accelerating them [ 8 , 9 , 10 , 11 ] . Most of these works rely on (step) distillation, where the teacher model provides direct supervision to train a distilled few-step generator. As such, the few-step generator learns to map multiple inference steps into a single step, or a small number of steps, by distilling the teacher’s trajectories. However, most distillation approaches come with practical drawbacks: like online methods [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 9 ] , which are costly to train as they require 2-3 full models held in memory at the same time, or offline methods [ 20 , 21 , 22 ] , which require significant resources to generate and store trajectory-output pairs offline to later train on. Furthermore, most distillation setups [ 12 , 13 , 14 , 15 , 17 , 18 ] distill the teacher with Classifier-Free Guidance (CFG), which is a double-edged sword, as the generator inherits both the high quality/prompt adherence as well as the low diversity and over-saturation artifacts of CFG [ 23 ] . To avoid such drawbacks, some studied acceleration through post-training (without distillation) [ 24 , 25 ] . These works are primarily adversarial, as opposed to distillation methods that use adversarial auxiliary losses [ 9 , 18 , 17 , 10 , 11 ] , and use real data rather than teacher-generated samples, thus freeing the costly requirement of using trajectory-output pairs. The motivation here is that the adversarial loss encourages realism , making each estimate better than the standard gaussian flow estimate. Such improved estimates enable post-trained models to use fewer (1-8) sampling steps [ 24 , 25 ] . Recent image works, like UFOGen [ 24 ] and APT [ 25 ] , explored adversarial post-training, yet UFOGen reported limited gains [ 24 ] and APT required a distillation-based initialization [ 25 ] . How to improve adversarial post-training and apply it to audio remains an open question. Fig. 1: ARC’s adversarial relativistic loss . Pairs of generated and real samples (with the same text prompts) are passed into the discriminator (with additive noise), where the generator and discriminator are trained to minimize and maximize (respectively) the difference between fake and real outputs. Fig. 2: ARC’s contrastive loss . The discriminator is also trained to maximize the difference between audios with correct and incorrect (shuffled) prompts. Within the audio-domain, only a few works exist on accelerating gaussian flow models. Most rely on consistency models [ 26 , 27 , 28 , 29 ] and have no adversarial component, despite the presence of adversarial losses in their image-domain counterparts [ 17 ] . The only work to use any adversarial component is Presto [ 18 ] , although it is mainly based on distillation [ 9 ] . Finally, most existing accelerated text-to-audio models focus on band-limited [ 26 ] , mono [ 27 , 29 , 18 ] audio. We show that rectified flows can be accelerated with adversarial post-training, introducing the first fully-adversarial text-to-audio acceleration framework that requires neither distillation nor CFG. We propose A dversarial R elativistic- C ontrastive post-training ( ARC ) that (1) extends a recently proposed relativistic adversarial formulation [ 30 , 31 ] to text-conditional audio generation and (2) combines this with a novel contrastive loss that encourages the discriminator to focus on prompt adherence. To our knowledge, we are the first to attempt such a post-training recipe on audio generation, the first to use adversarial relativistic losses for gaussian flow acceleration across modalities, and the first to use a contrastive discriminator formulation. We pair ARC with an array of architecture updates to Stable Audio Open (SAO) [ 4 ] to achieve accelerated performance, generating \approx 12s of 44.1kHz stereo audio in \approx 75ms on a H100 GPU, being 100x faster than the original SAO model. Across experiments, we find that ARC is competitive with existing state-of-the-art acceleration methods, and notably preserves generative diversity far beyond previous methods. We further optimize it for on-device CPU-based inference, achieving a text-to-audio model that can run locally on edge-devices ( i.e. , smartphones) with an inference time of \approx 7s. To our knowledge, we are the first to operate at this inference speed, enabling efficient on-device execution, and facilitating the applicability of text-to-audio in creative domains. We make our code public, and a demo website is available. 2 METHOD 2.1 Rectified flows pre-training Given a prompt \bm{c} and a (stereo audio) representation \mathbf{x}_{0} , the goal of (text-to-audio) rectified flows [ 20 , 6 ] is to learn a model that transfers between the data distribution p_{0} and some prior distribution p_{1} ( e.g. , isotropic gaussian noise) given \bm{c} , thus allowing to generate samples from p_{0} by sampling from p_{1} given \bm{c} according to the learned model. The forward corruption process (from data to noise) is given by: \mathbf{x}_{t}=(1-t)\mathbf{x}_{0}+t\bm{\epsilon},\quad\bm{\epsilon}\sim\mathcal{N}(0,\bm{I}), (1) where t is the scalar time parameter ranging from 0 (data) to 1 (noise) that is p_{\text{gen}}(t) distributed. For brevity, we refer to the noising process in Eq. 1 as sampling from the forward probability distribution q(\mathbf{x}_{t}\mid\mathbf{x}_{0}) . To reverse this process and generate samples (from noise to data), we can solve the ordinary differential equation (ODE): \mathrm{d}\mathbf{x}_{t}=-{v}_{{\theta}}(\mathbf{x}_{t},t,\bm{c})\mathrm{d}t, (2) where {v}_{{\theta}}({\mathbf{x}_{t}},t,\bm{c}) is a model trained to predict the velocity of the flow \mathbf{v}=\bm{\epsilon}-\mathbf{x}_{0} . This model can thus be trained by adding noise to real samples according