FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search

Qihui Yang, Randal Leistikow, Yongyi Zang · 2025

Read FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search by Qihui Yang, Randal Leistikow, Yongyi Zang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21667.txt.

Introduction

Qihui Yang, Randal Leistikow, and Yongyi Zang generate virtual instruments that keep one consistent voice across every pitch and playing strength. Sampled and modeled instruments live or die by that consistency. Note-by-note generators tend to let the tone wander as notes change. That betrays the illusion of a single instrument.

Their FlowSynth reworks flow matching so the model's velocity field is itself a distribution. The model can admit uncertainty instead of committing to one average path. That honesty pays off at generation time. The system samples several trajectories and weighs them by the model's own confidence. It keeps the output that best preserves timbre across notes. Against the leading TokenSynth baseline, FlowSynth wins on single-note quality. It also wins on cross-note consistency, edging toward instruments fit for live performance.

Catalog record

CREATORQihui Yang, Randal Leistikow, Yongyi Zang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21667

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FlowSynth: Instrument Generation Through Distributional Flow Matching and Test-Time Search Qihui Yang, Randal Leistikow, Yongyi Zang Abstract Virtual instrument generation requires maintaining consistent timbre across different pitches and velocities, a challenge that existing note-level models struggle to address. We present FlowSynth , which combines distributional flow matching (DFM) with test-time optimization for high-quality instrument synthesis. Unlike standard flow matching that learns deterministic mappings, DFM parameterizes the velocity field as a Gaussian distribution and optimizes via negative log-likelihood, enabling the model to express uncertainty in its predictions. This probabilistic formulation allows principled test-time search: we sample multiple trajectories weighted by model confidence and select outputs that maximize timbre consistency. FlowSynth outperforms the current state-of-the-art TokenSynth baseline in both single-note quality and cross-note consistency. Our approach demonstrates that modeling predictive uncertainty in flow matching, combined with music-specific consistency objectives, provides an effective path to professional-quality virtual instruments suitable for real-time performance. 1]Smule Labs 2]University of California, San Diego \contribution [†]Work done during internship at Smule 1 Introduction Modern music production demands precise control over timbre, pitch, and dynamics at the note level. While recent advances in flow matching ( Lipman et al. 2023 ) have enabled high-quality music generation through models ( Prajwal et al. 2024 ; Ning et al. 2025 ) , maintaining consistent timbre across an instrument’s range remains a fundamental challenge for note-level generation. This gap limits the adoption of AI-generated instruments in professional music production, where musicians expect reliable timbral behavior comparable to physical instruments or high-quality sample libraries. Virtual instrument generation, the task of synthesizing individual notes with specified pitch, velocity, and timbre for real-time playability, has evolved significantly from early neural approaches. NSynth ( Engel et al. 2017 ) pioneered learned instrument synthesis, while DDSP ( Engel et al. 2020 ) introduced differentiable signal processing for more controllable generation. The current state-of-the-art, TokenSynth ( Kim et al. 2025 ) , employs transformer architectures with CLAP embeddings ( Wu et al. 2024 ) to achieve reasonable timbre consistency across 88 piano keys. However, even these advanced models exhibit timbral drift: a ”warm piano” with a fundamental frequency of C3 (131 Hz) may sound metallic at C6 (1046 Hz), despite identical text conditioning. This inconsistency partly stems from the deterministic nature of current generation pipelines, which lack mechanisms to explore multiple solutions and select for consistency, a limitation noted in studies on timbre disentanglement ( Luo et al. 2019 ) . Meanwhile, the broader generative modeling community has explored probabilistic approaches to address similar challenges. Variational Rectified Flow Matching ( Guo and Schwing 2025 ) introduces latent variables for capturing multi-modal velocity fields in graph generation, while energy-weighted flow matching ( Woo and Ahn 2024 ) uses importance weighting for challenging distributions. In parallel, test-time compute scaling has demonstrated remarkable gains in language models ( Snell et al. 2024 ) , with best-of-N sampling and its variants proving effective across domains ( Mudgal et al. 2024 ; Gui et al. 2024 ) . However, these advances have not been adapted to address the specific challenge of cross-note timbre consistency in virtual instruments. We introduce FlowSynth, which addresses timbre consistency through distributional flow matching (DFM), a novel formulation that learns distributions over velocity fields rather than point estimates. While Variational Rectified Flow Matching ( Guo and Schwing 2025 ) also models velocity field uncertainty through latent variables optimized with an evidence lower bound (ELBO) objective assuming unit Gaussian priors, DFM takes a more direct approach: we parameterize the velocity field itself as a Gaussian distribution with learned mean and variance, optimized via negative log-likelihood (NLL). NLL optimization without unit Gaussian prior assumptions enables our model to learn problem-specific uncertainty patterns. The NLL objective naturally encourages higher-variance predictions in regions of genuine ambiguity (where multiple valid velocities exist), while maintaining confident predictions elsewhere. This learned uncertainty map proves crucial for test-time search, allowing us to explore more extensively in ambiguous regions while preserving high-confidence predictions, ultimately enabling the principled trajectory sampling that drives our consistency improvements. At test time, we leverage this learned uncertainty through a temperature-controlled sampling strategy that scales with computational budget. Drawing inspiration from extreme value theory, we design a sublinear temperature schedule that balances exploration with stability. For each note generation after the first, we sample multiple trajectories weighted by the model’s confidence and evaluate them against a timbre consistency objective—transforming generation from a single-shot process to an optimization problem where additional compute directly improves instrument quality. FlowSynth builds on established audio generation architectures, employing a Diffusion Transformer (DiT) ( Peebles and Xie 2023 ) with adaptive layer normalization for conditioning, similar to recent models like AudioX ( Tian et al. 2025 ) . We leverage CLAP encoders ( Wu et al. 2024 ) for text-audio alignment and adopt the VAE from DiffRhythm ( Ning et al. 2025 ) for efficient latent space modeling. This architectural foundation, combined with our distributional formulation, delivers substantial improvements: even without test-time search, FlowSynth surpasses TokenSynth in audio quality and timbre consistency, with comparable prompt adherence when using unconditional search. The true strength of DFM emerges with test-time compute scaling: increasing the search budget by 8\times improves timbre consistency by 13 % and prompt adherence by 246 %, demonstrating that our uncertainty-aware approach effectively converts compute into quality gains. Our contributions are: (1) Distributional flow matching (DFM), a formulation using negative log-likelihood optimization that captures predictive uncertainty for improved generation; (2) A test-time optimization framework combining confidence-weighted sampling with music-specific consistency objectives; (3) FlowSynth, achieving state-of-the-art results in virtual instrument generation with demonstrated compute scaling benefits. By combining uncertainty-aware generation with domain-specific optimization, we provide a practical path toward professional-quality virtual instruments that maintain consistent timbre across their entire range. Figure 1 : Proposed distributional flow matching (DFM) architecture. (Left) Training phase: The diffusion transformer (DiT) learns to predict the instantaneous velocity field by projecting latent representations into mean ( \mu ) and variance ( \sigma ) parameters of a Gaussian distribution. Negative log-likelihood (NLL) loss is minimized between the ground-truth latent at current timestep and this learned distribution. (Right) Inference phase: Multiple velocity field samples are drawn from the learned Gaussian distribution at each generation timestep t . These samples undergo iterative one-step generation until the final timestep T , where a search metric evaluates all candidate latents to select the optimal output for the current timestep. 2 Distributional Flow Matching Standard flow matching learns deterministic mappings from source to target distributions, providing no mechanism to express predictive uncertainty or explore alternative generation paths. We propose Distributional Flow Matching (DFM), which reformulates velocity field prediction as learning a probability distribution over possible velocities, enabling uncertainty-aware generation and principled test-time exploration. 2.1 Probabilistic Velocity Field Formulation In conventional flow matching, a neural network f_{\theta} predicts a deterministic velocity field v_{\theta}(x_{t},t) that transports samples along an optimal transport path. We extend this framework by modeling the velocity field as a conditional probability distribution: p(v|x_{t},t,c)=\mathcal{N}(v;\mu_{\theta}(x_{t},t,c),\sigma^{2}_{\theta}(x_{t},t,c)I), (1) where x_{t} denotes the state at time t , c represents optional conditioning information, and I is the identity matrix. The neural network jointly predicts the distribution parameters: [\mu_{\theta}(x_{t},t,c),\log\sigma^{2}_{\theta}(x_{t},t,c)]=f_{\theta}(x_{t},t,c). (2) We parameterize the log-variance to ensure numerical stability and strictly positive variance values through the exponential

Read the complete text