Evaluating Latent Space Structure in Timbre VAEs: A Comparative Study of Unsupervised, Descriptor-Conditioned, and Perceptual Feature-Conditioned Models

Joseph M. Cameron, Alan F. Blackwell · 2026

Read Evaluating Latent Space Structure in Timbre VAEs: A Comparative Study of Unsupervised, Descriptor-Conditioned, and Perceptual Feature-Conditioned Models by Joseph M. Cameron, Alan F. Blackwell (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16713.txt.

Introduction

Cameron and Blackwell compare how three timbre autoencoders organize their latent spaces. One model is unsupervised; one takes the 19 word labels as conditions. The third takes continuous perceptual features from the AudioCommons models. All train on the team's labeled guitar dataset. Perceptual conditioning gave the most compact, pitch-stable space. One-hot word conditioning lagged on several structure metrics. The study supplies tools for judging timbre latent spaces. Controllable, readable audio generation is the end goal.

Catalog record

CREATORJoseph M. Cameron, Alan F. Blackwell
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18600
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16713

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Latent Space Structure in Timbre VAEs: A Comparative Study of Unsupervised, Descriptor-Conditioned, and Perceptual Feature-Conditioned Models Joseph M. Cameron, Alan F. Blackwell Abstract We present a comparative evaluation of latent space organization in three Variational Autoencoders (VAEs) for musical timbre generation: an unsupervised VAE, a descriptor-conditioned VAE, and a VAE conditioned on continuous perceptual features from the AudioCommons timbral models. Using a curated dataset of electric guitar sounds labeled with 19 semantic descriptors across four intensity levels, we assess each model’s latent structure with a suite of clustering and interpretability metrics. These include silhouette scores, timbre descriptor compactness, pitch-conditional separation, trajectory linearity, and cross-pitch consistency. Our findings show that conditioning on perceptual features yields a more compact, discriminative, and pitch-invariant latent space, outperforming both the unsupervised and discrete descriptor-conditioned models. This work highlights the limitations of one-hot semantic conditioning and provides methodological tools for evaluating timbre latent spaces, contributing to the development of more controllable and interpretable generative audio models. † † address: Department of Computer Science & Technology, University of Cambridge Cambridge, United Kingdom 1 Introduction Controlling musical timbre with generative models is a key challenge in machine learning for audio. Variational Autoencoders (VAEs) are widely used in this domain due to their ability to learn latent spaces that support interpolation and continuous control [ 11 , 8 ] . However, the structure and interpretability of these latent spaces, particularly in relation to timbral descriptors, remain poorly understood. This limits their applicability in real-time, human-centered audio interfaces [ 2 ] . Recent research has experimented with conditioning VAEs using both high-level semantic labels and low‑level acoustic features. For instance, Yutani et al. developed a wavetable synthesis model using a CVAE conditioned on semantic descriptors such as ‘bright’, ‘warm’, and ‘rich’ for intuitive timbre control [ 21 ] . Roche et al. introduced a VAE that incorporates a perceptual regularization loss informed by listener ratings across timbral dimensions like ‘metallic’ or ‘evolving’, enabling controllable timbre transformations [ 19 ] . Additionally, Natsiou et al. proposed a VAE trained on low-level acoustic features like spectral centroid and attack time to align latent representations with human perception of timbre [ 14 ] . Yet, no comprehensive study has systematically compared how such semantic versus perceptual conditioning strategies affect latent-space structure and interpretability for timbre generation and manipulation. Specifically, it remains unclear which VAE architectures best support consistent, discriminative, and controllable latent organization, qualities crucial for interactive audio applications. This paper presents a comparative analysis of three VAE models trained on a curated dataset of electric guitar sounds with annotated timbre descriptors: (1) an unsupervised VAE, (2) a descriptor-conditioned VAE using one-hot semantic labels, and (3) a perceptual feature-conditioned VAE using continuous AudioCommons timbral features [ 15 , 16 ] . We evaluate each model’s latent space using both standard clustering metrics and novel interpretability measures tailored to timbre, including within-pitch separation, magnitude progression linearity, and cross‑pitch consistency. Our results show that perceptual conditioning yields significantly more structured and pitch-invariant latent spaces, outperforming both the unsupervised and semantic descriptor conditioned variants. These findings provide evidence that perceptually grounded conditioning supports more usable latent representations for musical control tasks, and offer practical evaluation tools for future generative audio models. 2 Related Work Generative models have become increasingly central in computational audio synthesis [ 8 ] , particularly for representing and manipulating musical timbre [ 2 ] . Among these, Variational Autoencoders (VAEs) [ 11 ] have gained popularity due to their ability to learn continuous and interpretable latent spaces that support interpolation and continuous control. Recent work has explored conditioning VAEs to enhance control and improve disentanglement. Conditioning strategies include symbolic labels such as instrument class [ 1 ] , musical pitch [ 12 ] , or expressive descriptors [ 21 ] . However, semantic labels can be noisy, subjective, and/or coarse-grained, which may hinder their effectiveness for fine-grained timbre control [ 12 ] . To address this, other approaches have used continuous perceptual features derived from audio analysis toolkits such as Timbre Toolbox [ 18 ] or the AudioCommons project [ 15 , 16 ] . These features include timbral dimensions like brightness, roughness, or warmth, which have been shown to align well with human perceptual ratings [ 7 ] . Roche et al. proposed a VAE with perceptual regularization based on human annotations [ 19 ] , while Natsiou et al. trained a VAE to reconstruct and organize audio samples based on low-level acoustic descriptors [ 14 ] . Despite these developments, few studies have directly compared how different conditioning strategies impact latent space organization, interpretability, and cross-pitch consistency noted in pitch-timbre disentanglement [ 12 ] . Furthermore, existing evaluations often rely on standard clustering metrics [ 13 , 20 ] without addressing musical domain-specific needs like trajectory linearity or timbre magnitude progression. This paper fills that gap by combining standard metrics with tailored interpretability measures to systematically compare three VAE variants for musical timbre modeling. 3 Methodology To enable a valid and unbiased comparison of latent space structures, all three models in this study were trained under identical experimental conditions. Each model was trained on the same curated dataset, with consistent input representations, architecture capacity, training schedules, and optimization settings. This experimental design isolates the effect of the conditioning strategy on the resulting latent space structure, enabling direct attribution of observed differences to the conditioning method rather than to confounding factors such as data variability or training configuration. 3.1 Dataset We use a curated subset of the Semantic Timbre Dataset [ 4 , 3 ] , comprising 1,771 monophonic electric guitar spectrograms. Each sample is labeled with: • One of the Semantic Timbre Dataset’s 19 semantic timbre descriptors [ 3 ] . • One of four timbre descriptor magnitudes (25%, 50%, 75%, 100%), where timbre magnitude represents the magnitude of the Guitar Rig Pro 7 effect that corresponds to each relevant Semantic Timbre Dataset timbre descriptor [ 5 ] applied to monophonic electric guitar notes recorded by Pedroza et al. [ 17 ] . • One of 23 musical pitches ranging from E4 to D6. All of this audio was preprocessed into STFTs computed with a 1024-point FFT and 512 hop size at 22.05kHz. These STFTs were then used to train the VAEs to obtain different latent spaces for comparison. 3.2 The VAE Models Figure 1 : Semantic Timbre Generation VAE model architecture. The encoder maps input spectrograms to a 128D latent space. The decoder reconstructs log-magnitude spectrograms, which are converted back to audio. Three VAE models’ latent spaces were compared. Each model uses the same convolutional encoder-decoder VAE architecture with a 128-dimensional latent space, which can be seen in Figure 1 , and identical training hyperparameters. The models differ only in the conditioning strategy: • Unsupervised VAE : Learns representations solely from spectrograms with no auxiliary input. • Descriptor-Conditioned VAE (CVAE) : Conditions on a 42-dimensional one-hot vector comprising: 19 binary descriptor indicators (multiplied by descriptor magnitude), and 23 one-hot pitch labels. The condition vector is concatenated to both encoder and decoder inputs. • Perceptual Feature-Conditioned VAE (AudioCommonsCVAE) : Conditions on 7 continuous perceptual timbre features (brightness, depth, hardness, roughness, warmth, sharpness, boominess) from the AudioCommons models [ 15

Read the complete text