Learning Perceptually Relevant Temporal Envelope Morphing

Carnegie Mellon University · 2025

Read Learning Perceptually Relevant Temporal Envelope Morphing by Carnegie Mellon University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01588.txt.

Introduction

Satvik Dixit and three co-authors tackle how to blend the loudness shape of two sounds in a way that feels right to the ear. Past tools often just lay both shapes on top of each other, which sounds odd when the sounds have very unlike timing.

The team first ran listening tests to learn what a good blend should keep, then built large training sets that follow those rules. They trained a model that squeezes sound shapes into a compact code and blends them there. Tests on made up and real sounds beat older methods.

Catalog record

CREATORCarnegie Mellon University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01588

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Perceptually Relevant Temporal Envelope Morphing Carnegie Mellon University Abstract Temporal envelope morphing, the process of interpolating between the amplitude dynamics of two audio signals, is an emerging problem in generative audio systems that lacks sufficient perceptual grounding. Morphing of temporal envelopes in a perceptually intuitive manner should enable new methods for sound blending in creative media and for probing perceptual organization in psychoacoustics. However, existing audio morphing techniques often fail to produce intermediate temporal envelopes when input sounds have distinct temporal structures; many morphers effectively overlay both temporal structures, leading to perceptually unnatural results. In this paper, we introduce a novel workflow for learning envelope morphing with perceptual guidance: we first derive perceptually grounded morphing principles through human listening studies, then synthesize large-scale datasets encoding these principles, and finally train machine learning models to create perceptually intermediate morphs. Specifically, we present: (1) perceptual principles that guide envelope morphing, derived from our listening studies, (2) a supervised framework to learn these principles, (3) an autoencoder that learns to compress temporal envelope structures into latent representations, and (4) benchmarks for evaluating audio envelope morphs, using both synthetic and naturalistic data, and show that our approach outperforms existing methods in producing temporally intermediate morphs. All code, models, and datasets will be made publicly available upon publication. \NoHyper 1 Introduction Fig. 1: Method overview. In Stage 1, we repurpose an audio autoencoder to learn a latent representation of temporal envelopes. In Stage 2, we train a twin (order-invariant) mapper network (2b) in a supervised setup on synthetic training data that encodes perceptually-grounded morphing rules (2a). During training, the autoencoder is kept frozen while the mapper network learns to morph the envelope embeddings. Sound morphing is the process of gradually and smoothly transforming one sound into another to generate novel and perceptually hybrid sounds that simultaneously resemble both [ 1 , 2 ] . The ability to smoothly morph between sounds is a longstanding goal in auditory signal processing, with applications ranging from artistic sound design to cognitive science. Such techniques find applications in blending vocal timbres in music production [ 3 ] , generating transitional sound effects for virtual environments [ 4 ] and auditory research [ 5 ] . Traditional digital signal processing based systems for morphing are limited to musical sounds [ 6 , 2 , 7 ] or vocal sounds [ 8 , 9 ] . These methods employ signal processing techniques to extract features—such as coefficients from a source-filter model representation [ 8 ] or the harmonic components of sounds [ 2 ] —which are then interpolated to produce the morphed outputs. Although these traditional methods perform well for pitched instruments and voiced utterances, such techniques often break down when applied to everyday inharmonic sounds characterized by complex, non-stationary amplitude patterns. More recent ML-based approaches are based on text-to-audio (TTA) models [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] . Methods like MorphFader [ 20 ] and SoundMorpher [ 4 ] propose morphing by interpolating between the latent space representations of the two sounds. However, these systems often rely on naive arithmetic averaging in latent space that fails to produce perceptually meaningful results when morphing between sounds with distinct temporal structures — for example, merging a dog’s short, impulsive bark with a violin’s sustained fluctuations produced by vibrato [ 4 ] . For such sounds, the results are often the same as overlaying the two sounds on top of each other. This failure highlights a gap between ML-based approaches involving latent space interpolation and perceptually grounded morphing approaches. At the heart of this challenge lies the temporal envelope , which characterizes variations in the overall loudness of a sound over time and constitutes critical information for both identification and perceptual grouping [ 21 ] . Psychoacoustic studies have shown that listeners are sensitive to fine-grained properties of temporal envelopes, such as attack time, impulse spacing, and amplitude modulation [ 21 , 22 ] . Yet existing ML systems do not explicitly encode these principles, and accordingly, sound morphing systems based on ML may not handle them in a perceptually intuitive fashion. In this work, we begin by investigating how humans perceive morphs between temporal envelopes. Through a controlled listening study, we identify key perceptual principles that govern listeners’ sense of continuity and naturalness during audio envelope morphing. These principles are based on simple envelope properties such the quantity of impulses, spacing between impulses, and temporal placement. Building on these insights, we construct a synthetic dataset of envelope tuples—two base envelopes, a perceptually valid ground-truth morph, and an interpolation weight \alpha —by systematically varying these properties one at a time. To encode these envelopes into a compact and perceptually meaningful representation, we train an autoencoder [ 23 ] that maps raw envelope signals into a low-dimensional latent space. Next, we train a “twin” (order-invariant) neural network [ 24 ] on our synthetic data to predict the appropriate morph given the two latent representations and \alpha . Finally, we create three benchmarks with synthetic and naturalistic envelopes and evaluate our system, demonstrating that our method consistently outperforms existing techniques in producing perceptually intermediate temporal envelope morphs. Our contributions are as follows. • We derive perceptual principles for envelope morphing from listening studies. • We encode these principles into synthetic data and propose a supervised morphing framework based on a twin neural network. • We train an autoencoder to compress temporal envelope structures into latent representations on envelopes derived from a large-scale dataset of everyday sounds. • We create benchmarks for evaluating audio envelope morphs on synthetic and naturalistic data and demonstrate that our approach is superior to existing methods. 2 Related Work Traditional signal processing approaches. Early research in sound morphing primarily focused on musical and vocal signals, leveraging parametric signal models to interpolate between sources. Methods based on convolution [ 25 , 6 ] , sinusoidal modeling [ 26 ] , and source-filter decomposition [ 8 , 27 ] enabled morphing between pitched sounds by interpolating spectral envelopes, harmonic content, and excitation signals. Extensions to these methods introduced high-level controls over perceptual aspects such as pitch, timbre, and brightness [ 1 ] , allowing for greater flexibility in musical applications. However, most of these systems assume stable harmonic structures and voiced excitation, limiting their applicability to the highly diverse and often aperiodic sounds found in everyday environments. Psychoacoustic Foundations. From a psychoacoustic perspective, temporal envelopes are known to carry essential information for sound identity and perceptual grouping. Moore [ 21 ] emphasized that amplitude modulation cues are critical for distinguishing between different sound sources. McAdams [ 22 ] demonstrated that listeners rely heavily on attack and decay characteristics for recognizing instrument timbres, while Bregman [ 28 ] showed the role of temporal cues in stream formation and segregation. McDermott [ 29 ] showed that statistical properties of temporal envelopes are crucial for identifying complex natural sound textures. More recent work has shown that altering temporal envelopes alone can shift perceptual categorization [ 5 ] , and that both humans and ML models can use temporal and spectral features to identify sound events [ 30 ] . Despite

Read the complete text