Designing Neural Synthesizers for Low-Latency Interaction This is the Author’s Accepted Manuscript (AAM), accepted for publication in the Journal of the Audio Engineering Society .
Franco Caspe 1 , Jordie Shier 1 , Mark Sandler 1 , Charalampos Saitis 1 , Andrew McPherson 2 · 2025
Read Designing Neural Synthesizers for Low-Latency Interaction This is the Author’s Accepted Manuscript (AAM), accepted for publication in the Journal of the Audio Engineering Society . by Franco Caspe 1 , Jordie Shier 1 , Mark Sandler 1 , Charalampos Saitis 1 , Andrew McPherson 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11562.txt.
Introduction
Franco Caspe and co-authors tackle delay in neural music synthesizers in a 2025 paper. Such models let players shape rich, high quality sound in real time, but high delay makes close musical play feel wrong.
The authors trace where delay and jitter come from in these models. They apply the lessons to a known audio autoencoder used for timbre transfer. Step by step design changes then cut the delay down. The result, called BRAVE, keeps pitch and loudness tracking strong while matching the older model on timbre change. It runs in a special low delay framework, and the team built a proof of concept plugin for live instrument signals.
Catalog record
| CREATOR | Franco Caspe 1 , Jordie Shier 1 , Mark Sandler 1 , Charalampos Saitis 1 , Andrew McPherson 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Designing Neural Synthesizers for Low-Latency Interaction This is the Author’s Accepted Manuscript (AAM), accepted for publication in the Journal of the Audio Engineering Society . Franco Caspe 1 , Jordie Shier 1 , Mark Sandler 1 , Charalampos Saitis 1 , Andrew McPherson 2 Abstract Neural Audio Synthesis (NAS) models offer interactive musical control over high-quality, expressive audio generators. While these models can operate in real-time, they often suffer from high latency, making them unsuitable for intimate musical interaction. The impact of architectural choices in deep learning models on audio latency remains largely unexplored in the NAS literature. In this work, we investigate the sources of latency and jitter typically found in interactive NAS models. We then apply this analysis to the task of timbre transfer using RAVE, a convolutional variational autoencoder for audio waveforms introduced by Caillon et al. in 2021. Finally, we present an iterative design approach for optimizing latency. This culminates with a model we call BRAVE (Bravely Realtime Audio Variational autoEncoder), which is low-latency and exhibits better pitch and loudness replication while showing timbre modification capabilities similar to RAVE. We implement it in a specialized inference framework for low-latency, real-time inference and present a proof-of-concept audio plugin compatible with audio signals from musical instruments. We expect the challenges and guidelines described in this document to support NAS researchers in designing models for low-latency inference from the ground up, enriching the landscape of possibilities for musicians. I INTRODUCTION In recent years, interest in interactive Neural Audio Synthesis (NAS) algorithms has increased within the music research community. NAS algorithms utilize deep neural networks, trained on audio datasets, and recent advancements have paved the way for high-quality synthesis [ 1 ] . The flexibility of these architectures has enabled a variety of control modalities, ranging from MIDI input [ 2 ] to audio-based control [ 3 , 4 ] , and has inspired entirely new interfaces [ 5 ] . The use of variational autoencoders [ 6 ] for neural audio synthesis has been particularly fruitful, with the RAVE model [ 4 ] facilitating a range of real-time interactive musical applications and inspiring the development of novel interaction modalities [ 5 , 7 ] . Many NAS algorithms support real-time inference, but they are generally unsuitable for live instrumental performance because they often introduce latencies of several hundred milliseconds [ 8 ] . This is an important factor in music performance; systems that offer low latency can lead to richer and better-perceived experiences [ 9 , 10 , 11 ] . However, latency in NAS models has yet to be formally investigated, which hinders the development of low-latency musical NAS models. In this work, we first present an analysis of primary latency sources in interactive, audio-to-audio NAS algorithms. These are: 1) buffering delay; 2) cumulative delay; 3) representation delay; 4) data-dependent latency; and 5) positional uncertainty (jitter). As a case study, we analyse RAVE’s architecture for these sources of latency, highlighting how this is unsuitable for low-latency interaction. To support this analysis we design an open-source toolkit that we use to empirically evaluate latency during inference. Building on our initial analysis, we present a design approach where we iteratively modify the RAVE architecture to achieve our target latency and jitter goals for musical interaction, as well as computing requirements for real-time inference on a consumer CPU. Our iterative design culminates with a model we call BRAVE (Bravely Realtime Audio Variational autoEncoder), which we release in a proof-of-concept audio plugin for musicians to experiment with. See the accompanying website for training source code, and audio examples 1 1 1 https://fcaspe.github.io/brave . We evaluate the model variants on an audio timbre transfer task, i.e., modifying the timbre of an audio input while preserving its musical content, and show that BRAVE improves upon RAVE in preserving musical content (in terms of pitch and loudness) and presents similar capabilities for reproducing the timbre of the training dataset. The results of our analysis and evaluation point to the importance of the model’s receptive field in enabling rich musical affordances such as timbre transfer. The ability of the decoder to consider time-varying trajectories of latent representations, supported by a suitable receptive field, is critical in the emergence of useful musical interactions. These findings suggest practical methods for future development of low-latency NAS systems beyond RAVE. In summary, in this work, we put latency at the center of NAS design. We provide a design framework that leverages analytical and empirical observations grounded in our musical goals, which motivate and inform architectural decisions, ensuring that both performance and responsiveness are prioritized. We hope that this perspective, along with the results of this research, will not only support the improvement of existing NAS models but also support the development of novel systems that enrich the landscape of DMIs for musicians. II BACKGROUND II-A Real-time Constraints for Musical Interaction Low action-to-sound latency and timing stability are crucial in DMIs for supporting control intimacy, time-keeping, and developing performance skills and personal style [ 12 ] . Wessel and Wright [ 13 ] suggest an upper bound on latency of 10ms and a jitter (variation of latency) of \pm 1 ms for musical interaction. Furthermore, Jack et al. [ 11 ] conducted an empirical study that supported this recommendation for percussive instruments, demonstrating that 10ms of latency was acceptable to performers in the absence of jitter; however, 20ms of latency or 10 \pm 3 ms of latency significantly degraded the experience of performance. Lester and Boley [ 14 ] performed subjective tests with practicing musicians to investigate the effect of latency under different monitoring scenarios. Their results highlight the context-dependent nature of latency perception with sensitivity amongst musicians varying based on the instrument and type of monitoring device. In general, however, their results reinforce the 10ms bound for minimal impact on musical performance. In this work, we aim to reach strict latency constraints that approximate those of 10 \pm 1 ms reported in previous work focused on instrumental interaction. II-B Interactive Neural Audio Synthesis NAS is a technique that employs Deep Neural Networks (DNNs) as learned audio synthesizers. Neural synthesis has its origins in offline generation; however, increased computation capabilities have enabled real-time inference, which has opened the doors for NAS use in live performance systems with control schemes that span from MIDI inputs [ 2 , 15 ] to Music Information Retrieval features [ 16 ] , latent spaces [ 17 , 18 ] , and other audio signals [ 4 , 19 ] . Despite this, latency is typically only addressed after models are trained [ 8 ] . A common solution in neural audio plug-in development is to report latency to the audio workstation for it to be compensated by re-aligning other audio tracks, which is incompatible with our live use case. Recently, a set of NAS models such as RAVE and the Differentiable DSP (DDSP) Decoder [ 3 ] have gained popularity due to their ability to transform audio in real-time according to learned timbral and dynamic characteristics of a dataset. These models can supplement musical instruments with new timbral possibilities, however, such models are not optimized for real-time interaction with the audio of musical instruments, due