SynthCloner: Synthesizer Preset Conversion via Factorized Codec with ADSR Envelope Control

Jeng-Yue Liu, Ting-Chao Hsu, Yen-Tung Yeh, Li Su, Yi-Hsuan Yang · 2025

Read SynthCloner: Synthesizer Preset Conversion via Factorized Codec with ADSR Envelope Control by Jeng-Yue Liu, Ting-Chao Hsu, Yen-Tung Yeh, Li Su, Yi-Hsuan Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24286.txt.

Introduction

Synth sounds are shaped by knobs that control tone color and the attack-decay envelope, which makes copying a synth style onto new audio hard. Jeng-Yue Liu and 4 co-authors built SynthCloner, a codec model that splits audio into three parts: envelope, timbre, and content. That split lets users steer each part on its own during transfer.

The team also built SynthCAT, a data set rendered for this task with hundreds of timbres, envelopes, and note sequences. SynthCloner beat rival methods on both measured scores and listener judgments. Independent control of envelope shape is the feature past spectral or implicit methods lacked.

Catalog record

CREATORJeng-Yue Liu, Ting-Chao Hsu, Yen-Tung Yeh, Li Su, Yi-Hsuan Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24286

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SynthCloner: Synthesizer Preset Conversion via Factorized Codec with ADSR Envelope Control Jeng-Yue Liu, Ting-Chao Hsu, Yen-Tung Yeh, Li Su, Yi-Hsuan Yang Abstract Electronic synthesizer sounds are controlled by presets, parameters settings that yield complex timbral characteristics and ADSR envelopes, making preset conversion particularly challenging. Recent approaches to timbre transfer often rely on spectral objectives or implicit style matching, offering limited control over envelope shaping. Moreover, public synthesizer datasets rarely provide diverse coverage of timbres and ADSR envelopes. To address these gaps, we present SynthCloner, a factorized codec model that disentangles audio into three attributes: ADSR envelope, timbre, and content. This separation enables expressive synthesizer preset conversion with independent control over these three attributes. Additionally, we introduce SynthCAT, a new synthesizer dataset with a task-specific rendering pipeline covering 250 timbres, 120 ADSR envelopes, and 100 MIDI sequences. Experiments show that SynthCloner outperforms baselines on both objective and subjective metrics, while enabling independent attribute control. The code, model checkpoint, and audio examples are available here 1 1 1 https://buffett0323.github.io/synthcloner/ . † † address: 1 National Taiwan University 2 Academia Sinica † † footnotetext: * Equal contribution. 1 Introduction Figure 1: Data rendering pipeline of SynthCAT. A sustained-note segment is cropped, mapped to the MIDI sequence through pitch shifting and duration alignment, and then shaped with the ADSR envelope on a per-note basis to produce the final audio. Electronic synthesizers have become essential tools in modern music production, offering vast flexibility in sound design. The sound of a synthesizer is defined by a preset, a parameter configuration of oscillators, filters, and envelopes, that jointly shape its sonic characteristics. Despite the creative possibilities this offers, crafting presets remains labor-intensive and requires expert knowledge [ 1 ] . To address this challenge, synthesizer preset conversion aims to convert a source preset to match a reference while preserving the content, thereby reducing manual parameter tuning and enabling efficient sound exploration. In this paper, we define the sonic character of a basic preset as comprising two aspects: timbre and ADSR envelope. Timbre denotes the time-invariant spectral characteristics shaped by oscillators and filters; for simplicity, we exclude LFO-induced variations and additional envelope routing that introduce time-varying effects [ 2 ] , while the ADSR envelope (Attack, Decay, Sustain, Release) governs the time-varying master amplitude trajectory. Together, these components produce the sound that range from percussive plucks to sustained leads. Consequently, effective preset conversion requires accurate modeling and control of both timbre and ADSR envelope. Existing methods for timbre transfer [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ] focus on acoustic settings and learn spectral or latent style mappings for reference-guided transfer, while guitar tone conversion [ 10 ] models the analog signal chain to reshape tone. Both paradigms primarily emphasize spectral similarity while leaving the input’s ADSR envelope unchanged—an assumption valid for acoustic instruments but misaligned with synthesizer design. A related direction is synthesizer sound matching [ 1 , 11 , 12 , 13 , 14 , 15 , 16 ] , which recovers parameters through differentiable or DSP-based models; however, preset conversion requires direct learning from audio for joint timbre and envelope control. Recent work has begun to consider envelopes more explicitly: Caspe et al. [ 17 ] predicted ADSR-like parameters to capture note onsets and offsets, while Masuda et al. [ 16 ] introduced a differentiable ADSR module. Yet these efforts focus on onset reproduction or differentiable control, rather than efficient preset conversion. A further challenge is the lack of publicly available datasets that jointly span timbral and envelope diversity. NSynth [ 18 ] offers large-scale note-level samples with broad timbre coverage but mostly sustained notes and limited envelope variation, while Synth1B1 [ 19 ] provides massive synthetic audio with parameter metadata but no explicit envelope control. To address this gap, we introduce SynthCAT, a dataset built from systematic combinations of 250 timbres, 120 ADSR envelopes, and 100 MIDI sequences, capturing the full diversity of combinations. Building on this resource, we present SynthCloner, a factorized codec model inspired by FACodec [ 20 ] . SynthCloner disentangles synthesizer audio into three attributes, content, timbre, and ADSR envelope, and integrates information perturbation [ 21 ] with supervised training [ 20 ] to enforce attribute separation. This enables faithful and controllable conversion across presets, explicitly modeling both spectral and temporal dynamics. The contributions of this paper are as follows: • We demonstrate empirically that explicit modeling of ADSR envelope control alongside timbre is essential for effective synthesizer preset conversion. • We present SynthCloner, the first model to our knowledge that performs synthesizer preset conversion through factorization of timbre and ADSR envelope attributes. • We introduce SynthCAT, a synthesizer dataset with 250 timbres, 120 ADSR envelopes, and 100 contents, providing a foundation for synthesizer preset conversion. 2 Data To construct SynthCAT, we use the Serum 2 2 2 https://xferrecords.com/products/serum-2 synthesizer driven by MIDI notes, as Serum is one of the most widely used VST instruments in music production. Our goal is to render the same content under varied timbre and ADSR envelope settings. Isolating envelopes is non-trivial since many presets couple them with osicaltors, filters, or effects [ 22 ] , leading to unwanted timbral changes. To mitigate this, we select presets from commercial packages 3 3 3 https://theproducerschool.com/ that generate sustained tones and render them as long notes. From each rendering, we extract a one-second one-shot sample from the flattest region of the waveform, quantified by the flatness score [ 23 , 24 ] : \text{Flatness}(x)=\frac{1}{1+\mathrm{Var}(\mu(x))},\quad\mu(x)=\tfrac{1}{N}\sum_{n=1}^{N}x[n], where x[n] is the waveform of length N , \mu(x) is its temporal mean, and \mathrm{Var}(\cdot) computes the variance across the segment. Segments with flatness above 0.95, indicating low variance in amplitude, are retained as distinct timbres. Each one-shot is then pitch-shifted and duration-aligned to MIDI sequences, and shaped with ADSR envelopes to render multi-note monophonic phrases. We generate 120 envelopes by uniformly sampling parameter ranges: Attack (10–100 ms), Decay (50–300 ms), Hold (0–200 ms), Sustain (0.0–0.80), and Release (30–300 ms). Finally, the dataset is built from the Cartesian product of 250 timbres, 120 envelopes, and 100 MIDI files from the mono-midi-transposition dataset [ 25 ] , yielding 3M monophonic audio samples (44.1 kHz, mono) with a total duration of about 2,500 hours. Figure 2: SynthCloner model architecture. To perform synthesizer preset conversion, replace \mathbf{x}_{\text{e}} and \mathbf{x}_{\text{t}} with the reference audio. 3 Proposed Approach 3.1 The SynthCloner Framework As illustrated in Fig. 2 , SynthCloner is inspired by the FACodec [ 20 ] framework but incorporates two key extensions: ADSR envelope modeling and attribute-specific information perturbation, detailed in Section 3.2 . The model decomposes an input audio \mathbf{x} into three disentangled latent representations through separate processing paths, each with specific perturbations [ 21 ] during training. First, the ADSR envelope path processes the perturbed input \mathbf{x}_{\text{e}} by converting it to log-RMS value [ 26 ] and passing it through the ADSR envelope extractor which is a temporal multi-scale Conv-BiLSTM network to capture envelope dynamics [ 27 ] , yielding the ADSR envelope embeddings \mathbf{z}_{\text{e}}\in\mathbb{R}^{D\times T} , where D denotes the embedding dimension and T the frame length. Second, the content path follows the FACodec [ 20 ] design, where the perturbed input \mathbf{x}_{\text{c}} passes through the

Read the complete text