SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Serving

Ayush, Sofia, Nikhil, Arnav, Abhinav, Muchen, Fan · 2026

Read SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Serving by Ayush, Sofia, Nikhil, Arnav, Abhinav, Muchen, Fan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07865.txt.

Introduction

Barik and colleagues speed up text-to-audio diffusion, which now needs many model calls and so runs slowly. Rather than start from pure noise, their training-free SoundWeaver builds a prompt-shaped sound prior from bits of cached clips and warms up diffusion partway. A composition graph picks cross-clip joins that fit the prompt and keep sound smooth, while a bandit gate picks how deep to skip for each request. Tests show clear speed gains from a small cache, with sound quality also improved.

Catalog record

CREATORAyush, Sofia, Nikhil, Arnav, Abhinav, Muchen, Fan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22944
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.07865

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Serving Ayush, Sofia, Nikhil, Arnav, Abhinav, Muchen, Fan Abstract Text-to-audio diffusion models produce high-fidelity audio but require tens of function evaluations (NFEs), incurring multi-second latency and limited throughput. We present SoundWeaver, the first training-free, model-agnostic serving system that accelerates text-to-audio diffusion by warm-starting from semantically similar cached audio. SoundWeaver introduces three components: a Reference Selector that retrieves and temporally aligns cached candidates via semantic and duration-aware gating; a Skip Gater that dynamically determines the percentage of NFEs to skip; and a lightweight Cache Manager that maintains cache utility through quality-aware eviction and refinement. On real-world audio traces, SoundWeaver achieves 1.8–3.0 × \times latency reduction with a cache of only ∼ {\sim} 1K entries while preserving or improving perceptual quality. Barik Stoica Sarda Kethana Khanduja Xu Lai keywords Text-to-Audio, Retrieval-Augmented Generation, Diffusion, Systems for Audio † † address: 1 University of Illinois Urbana-Champaign, USA 2 Assured Intelligence, USA † † email: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] 1 Introduction Diffusion models have rapidly become the foundation of text-to-audio (T2A) generation for applications such as music composition and sound-effect synthesis [ 18 , 17 ] . These models iteratively denoise Gaussian noise into high-fidelity waveforms or spectrograms over tens of steps while conditioning on natural language prompts. Despite their quality advantages, inference remains inherently computationally intensive: generating audio clips can require multiple seconds on modern GPUs [ 16 ] . At production scale, where services process millions of requests daily [ 29 ] , this translates into high user-perceived latency and substantial infrastructure cost. To mitigate this efficiency bottleneck, prior algorithmic advances have primarily focused on reducing the number of function evaluations (NFE). Algorithmic advances include improved samplers [ 19 , 20 ] , adaptive timestep scheduling [ 3 ] , early-exit strategies [ 23 ] , and distillation that compresses multi-step diffusion into few-step samplers [ 27 , 28 ] . Concurrently, systems-oriented efforts have targeted multi-GPU parallelism, and hardware–software co-design [ 7 , 21 ] . We explore a complementary, underexamined opportunity: exploiting inherent semantic similarity in audio distributions to reduce NFEs. As shown in Figure 1 , analysis of realistic traces (e.g., AudioCaps [ 13 ] ) reveals that most user-uploaded audios have close semantic neighbors. In diffusion models, early NFEs establish coarse, low-frequency structure, while later steps refine high-frequency perceptual detail [ 4 , 24 ] . Semantically similar samples, therefore, share structural components, allowing a cached neighbor to act as a strong prior—warm-starting from an intermediate step [ 32 ] and skipping the NFEs that construct coarse structure. Figure 1 : Distribution of CLAP scores for nearest-neighbor retrievals across AudioCaps prompts. Building on this insight, we present SoundWeaver \xspace , which works through audio-guided warm-starting, which involves maintaining a reference cache and, upon a new request, selecting a semantically aligned candidate to warm-start generation from an intermediate state, skipping initial NFEs. Enabling such approximate caching introduces non-trivial challenges, which we address with the following contributions: [topsep=2pt, partopsep=0pt, leftmargin=1.5em] • A novel Reference Selector that performs semantic and duration-aware retrieval and alignment from the cache (§ 2.1 ). • A Skip Gater that balances quality–latency trade-offs and adapts to the user-prompt distribution to determine the percentage of NFEs to skip during diffusion (§ 2.2 ). • A lightweight Cache Manager that performs quality-aware eviction and refinement, ensuring high reuse utility while bounding memory and compute overhead. (§ 2.3 ). By transforming audio similarity into computation savings, SoundWeaver \xspace substantially improves serving latency. Our evaluations on realistic workloads show that SoundWeaver \xspace achieves a 1.8–3.0× end-to-end latency speedup, while preserving—and often even improving—perceptual quality with a cache of only \sim 1K audio entries (§ 3 ). 2 Methods T2A diffusion models learn a reverse process p_{\theta}(x_{t-1}|x_{t},c) that iteratively denoises a latent from pure noise x_{T}\sim\mathcal{N}(0,I) toward a clean audio latent x_{0} over T steps, conditioned on text prompt c . SoundWeaver warm-starts this reverse process using a semantically aligned cached reference \hat{x}_{0} . Rather than initializing from pure noise, we sample an intermediate latent via the forward process q(x_{t^{*}}|\hat{x}_{0}) at skip timestep t^{*}<T , and run the reverse process from x_{t^{*}} , reducing the required NFEs from T to T-t^{*} . The semantic structure of \hat{x}_{0} acts as a prior steering generation toward c , with greater semantic alignment permitting more aggressive skipping. However, realizing consistent gains requires addressing gaps of prior works such as duration alignment between \hat{x}_{0} and the target-request duration, preserving output diversity, and adaptively selecting t^{*} . Overview. SoundWeaver \xspace consists of three key components, which will be defined in the following sections: 1 the Reference Selector (§ 2.1 ), 2 the Skip Gater (§ 2.2 ), and 3 the Cache Manager (§ 2.3 ). Given a text prompt, the Reference Selector retrieves top- K semantically similar audio samples from the cache, filters them using a quality gate, and stochastically selects a candidate before running it through a phase vocoder which temporally aligns the candidate to the requested duration. The aligned audio is passed to the Skip Gater , which determines the optimal skip percentage to warm-start diffusion at an intermediate step. Meanwhile, the Cache Manager asynchronously maintains the cache while ensuring runtime quality. Figure 2 : SoundWeaver \xspace overview and request execution flow. 2.1 Reference Selector Given a user prompt, the Reference Selector identifies a cached audio that maximizes NFE skipping while preserving generation quality. This decision requires jointly considering (i) semantic alignment with the new request (ii) output diversity and (iii) length compatibility with the requested duration. Quality-Aware Reference Retrieval. SoundWeaver \xspace introduces a gating mechanism that enforces quality and duration constraints on top- K retrieved candidates while preserving diversity. For each candidate i , we compute two CLAP scores: s_{\text{pos}}(i) , the CLAP score between the candidate’s audio embedding and the user prompt, and s_{\text{neg}}(i) , the similarity to a fixed negative prompt (e.g., “low quality”). Candidates are sampled proportionally to similarity: p_{i}\propto\exp(s_{\text{pos}}(i)/\tau) , where \tau controls diversity. To prevent degradation from weak candidates, we apply a quality gate admitting only samples satisfying q_{i}\geq\theta_{q} , where: \displaystyle q_{i}=\min\left[\frac{s_{\text{pos}}(i)}{\max_{j}\,s_{\text{pos}}(j)},\ \frac{1-s_{\text{neg}}(i)}{\max_{j}\,(1-s_{\text{neg}}(j))}\right] (1) We refer to the first term as a_{i} (normalized positive similarity) and the second as b_{i} (normalized negative dissimilarity) on the right side of the equation. \theta_{q} is a predefined quality threshold value. For scalability, candidate embeddings are indexed using FAISS [ 12 ] , which applies coarse K -means clustering followed by hierarchical approximate nearest-neighbor search. To further improve cache coverage, we introduce a pyramid indexing scheme that materializes CLAP embeddings at multiple temporal granularities down to a configurable minimum granularity \delta (e.g., \frac{1}{4} of the clip duration). Concretely, a long audio clip is partitioned into multi-scale segments, each associated with its own embedding, enabling retrieval to match the most semantically aligned portion of a cached audio rather than the entire clip. Importantly, this design does not increase audio storage overhead, as we retain only a single full-resolution copy of each audio and index lightweight segment-level embeddings. Duration-Aware Adaptation. Durations of T2A generation requests can vary substantially, and diffusion latents are inherently duration-dependent, preventing direct reuse of cached clips with mismatched lengths. Unlike text or image generation, audio supports continuous time-stretching that can adjust

Read the complete text