Neural Proxies for Sound Synthesizers: Learning Perceptually Informed Preset Representations
Paolo Combes, Stefan Weinzierl, Klaus Obermayer · 2025
Read Neural Proxies for Sound Synthesizers: Learning Perceptually Informed Preset Representations by Paolo Combes, Stefan Weinzierl, Klaus Obermayer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07635.txt.
Introduction
Paolo Combes and 2 co-authors train neural proxies for sound synthesizers in a 2025 paper. Automatic synthesizer programming helps musicians find presets that match a target sound, but software synthesizers can be black boxes that do not allow gradient-based training. Their method approximates an arbitrary synthesizer with a neural network that maps presets into an audio embedding space from a pre-trained model.
The proxy produces compact, effective preset representations, so an audio embedding loss can guide synthesizer programming systems. The team evaluates which pre-trained audio models give the best representations and compares feedforward, recurrent, and transformer architectures for the proxy. Tests use synthetic and hand-crafted presets from three popular software synthesizers, judged on sound matching. A good proxy turns any beloved synth into a trainable instrument for AI-assisted sound design.
Catalog record
| CREATOR | Paolo Combes, Stefan Weinzierl, Klaus Obermayer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07635
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Proxies for Sound Synthesizers: Learning Perceptually Informed Preset Representations Paolo Combes, Stefan Weinzierl, Klaus Obermayer Abstract Deep learning appears as an appealing solution for Automatic Synthesizer Programming (ASP), which aims to assist musicians and sound designers in programming sound synthesizers. However, integrating software synthesizers into training pipelines is challenging due to their potential non-differentiability. This work tackles this challenge by introducing a method to approximate arbitrary synthesizers. Specifically, we train a neural network to map synthesizer presets onto an audio embedding space derived from a pretrained model. This facilitates the definition of a neural proxy that produces compact yet effective representations, thereby enabling the integration of audio embedding loss into neural-based ASP systems for black-box synthesizers. We evaluate the representations derived by various pretrained audio models in the context of neural-based nASP and assess the effectiveness of several neural network architectures – including feedforward, recurrent, and transformer-based models – in defining neural proxies. We evaluate the proposed method using both synthetic and hand-crafted presets from three popular software synthesizers and assess its performance in a synthesizer sound matching downstream task. While the benefits of the learned representation are nuanced by resource requirements, encouraging results were obtained for all synthesizers, paving the way for future research into the application of synthesizer proxies for neural-based ASP systems. 0 Introduction 0.1 Background Sound synthesizers have revolutionized music by expanding the timbral palette available to musicians, enabling the creation of entirely new sounds. However, the vast creative possibilities offered by synthesizers introduce significant usability challenges. Synthesizer interfaces are typically designed around their underlying synthesis methods, with control parameters directly mapping to the internal workings of the synthesis engine [ 37 ] . As a result, these interfaces necessitate non-intuitive parameter adjustments for achieving conceptually simple sound manipulations [ 22 ] . As synthesis methods become more complex, synthesizer usability decreases, reducing accessibility for a wider range of musicians and sound designers. To address these usability challenges, researchers have developed a range of systems for Automatic Synthesizer Programming (ASP), which refers to any methods aiming to support the process of programming a sound synthesizer [ 38 ] . These systems include synthesizer sound matching methods [ 48 , 14 , 8 , 27 ] , query from vocal imitation [ 6 ] , high-level control using timbral attributes [ 23 ] or natural language [ 9 ] . Other methods include techniques for interpolation [ 26 ] , generation [ 34 ] , and retrieval [ 3 ] of presets – pre-programmed lists of synthesizer parameter values used to generate sound. Recent ASP systems typically rely on either genetic algorithms or deep learning approaches. While methods based on genetic algorithms can achieve high prediction quality, they are prohibitively time-consuming for interactive use in creative contexts [ 30 ] . In contrast, neural-based methods for Automatic Synthesizer Programming (nASP) offer an attractive solution, as they provide a balanced trade-off between prediction quality and computational efficiency. Additionally, their greater flexibility makes them more suitable for creative applications. 0.2 Challenges in nASP systems Two primary challenges arise when applying deep learning to synthesizers. First, integrating synthesizers into deep learning training pipelines is inherently challenging. Neural networks are typically trained using gradient-based optimization techniques. Although effective, this approach limits the integration of non-differentiable synthesizers with unknown internal structure – i.e., black-box synthesizers – within an end-to-end training pipeline. To maintain the differentiability required for gradient-based optimization, the synthesizer must be positioned after the loss computation within the training pipeline. As a result, the loss function can only depend on synthesizer parameter values, since incorporating the synthesizer into the network would disrupt the backpropagation chain. This poses a challenge since loss functions based on synthesizer parameter values – rather than synthesized audio – are suboptimal for two reasons. First, parameter differences are not necessarily proportional to perceptual differences, which diminishes nASP system performance in tasks requiring perceptual evaluation. Second, in supervised training, this requires the ground truth to be based on synthesizer parameter values, thereby restricting training data to in-domain sounds, i.e., sounds produced by the synthesizer itself, raising concerns about the generalization ability of nASP systems to arbitrary sounds. The second challenge arises from the data-driven approach required to train nASP systems. Ideally, in-domain training datasets should be derived from hand-crafted presets, i.e., presets designed by humans, which is both costly and time-consuming. As a result, the number of available hand-crafted presets is usually scarce. This scarcity hinders the use of larger and more powerful architectures, since they typically require more training data. Moreover, this makes the use of a specific synthesizer dependent on the available number of hand-crafted presets, which is a considerable drawback. Therefore, these two key challenges limit the performance of nASP methods across arbitrary synthesizers and must be addressed. 0.3 Main Contributions In this work, we propose a method to tackle the challenge posed by the lack of differentiability using neural proxies, which are neural networks trained to approximate black-box functions [ 19 ] . Neural proxies have been used to model various audio effects, including guitar amplifiers and effects chains [ 45 , 42 ] . Our approach aims to train a half-hybrid neural proxy [ 42 ] , where the proxy is used during training, but the software synthesizer is used during inference. More specifically, we propose the following: • A method for training a neural proxy for arbitrary synthesizers, referred to as the preset encoder. The preset encoder is trained to map synthesizer presets onto an audio embedding space, defined by a pretrained audio model serving as a fixed feature extractor. • A sound attribute ranking evaluation to assess representations derived from pretrained audio models for defining loss functions in nASP systems. • A benchmark comparing the impact of various neural network architectures and input preprocessing techniques on the preset encoder’s performance across three synthesizers of varying complexity. • An analysis of the generalization ability of preset encoders trained on a large-scale dataset of 10M synthetic presets to hand-crafted presets, examining how synthesizer programming complexity affects generalization. • An evaluation of the proposed method on a synthesizer sound matching downstream task, demonstrating its integration into nASP systems and highlighting its benefits. Once trained, the preset encoder enables the use of audio embedding loss functions – loss functions based on representations derived from pretrained audio models – for training nASP systems. By doing so, we aim to facilitate the development of neural proxies for training nASP systems, thereby improving the performance of deep learning models for arbitrary synthesizers. This, in turn, improves synthesizer usability and enables a broader range of music creators and sound designers to translate their ideas into sound more effectively. While prior work [ 2 ] has demonstrated the value of auditory loss functions derived from hand-crafted time-frequency representations produced by a neural proxy, we propose a complementary approach by integrating losses derived from pretrained audio models. Our initial findings align with previous research [ 2 , 31 , 44 ] , supporting the notion that incorporating synthesized audio-based losses provides advantages over using parameter value-based losses alone. In contrast to the aforementioned studies, which use large, hand-crafted representations, our results suggest that improved performance in sound matching tasks