Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations
Hugo Flores García io , Oriol Nieto i , Justin Salamon i , Bryan Pardo o , Prem Seetharaman i · 2024
Read Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations by Hugo Flores García io , Oriol Nieto i , Justin Salamon i , Bryan Pardo o , Prem Seetharaman i (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08550.txt.
Introduction
Sound designers often start from a doodle of noise: a hummed imitation, a rough vocal sketch of the sound they want. Five authors present Sketch2Sound, a model that turns such sketches into finished audio. It listens to three simple control signals that vary over time, loudness, brightness, and pitch, together with a text prompt saying what the sound should be.
The model bolts onto an existing text-to-audio generator with a light touch: brief fine-tuning and one small layer per control. During training, random smoothing of the control signals teaches the model to accept sloppy human sketches, not just exact curves. Tests show the output follows the shape of a vocal imitation while keeping the text prompt's identity and clean quality. The authors aim the tool at artists who think in gestures.
Catalog record
| CREATOR | Hugo Flores García io , Oriol Nieto i , Justin Salamon i , Bryan Pardo o , Prem Seetharaman i |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08550
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations Hugo Flores García io , Oriol Nieto i , Justin Salamon i , Bryan Pardo o , Prem Seetharaman i Abstract We present Sketch2Sound, a generative audio model capable of creating high-quality sounds from a set of interpretable time-varying control signals: loudness, brightness, and pitch, as well as text prompts. Sketch2Sound can synthesize arbitrary sounds from sonic imitations (i.e., a vocal imitation or a reference sound-shape). Sketch2Sound can be implemented on top of any text-to-audio latent diffusion transformer (DiT), and requires only 40k steps of fine-tuning and a single linear layer per control, making it more lightweight than existing methods like ControlNet. To synthesize from sketchlike sonic imitations, we propose applying random median filters to the control signals during training, allowing Sketch2Sound to be prompted using controls with flexible levels of temporal specificity. We show that Sketch2Sound can synthesize sounds that follow the gist of input controls from a vocal imitation while retaining the adherence to an input text prompt and audio quality compared to a text-only baseline. Sketch2Sound allows sound artists to create sounds with the semantic flexibility of text prompts and the expressivity and precision of a sonic gesture or vocal imitation. Sound examples are available at https://hugofloresgarcia.art/sketch2sound/ . i Adobe Research, o Northwestern University [email protected] I Introduction Sound design is the craft of storytelling through sonic composition. Within sound design, Foley sound is a technique where special sound effects are designed and performed in sync to a film during post-production [ 1 ] . These sound scenes are typically performed by a Foley artist on a stage equipped with abundant sound instruments and other soundmaking materials 1 1 1 Example of a Foley artist performing a scene: youtu.be/WFVLWo5B81w . Foley sound is a skilled and gestural performance art: performing a sound scene with sound-making objects and instruments (instead of arranging pre-recorded samples post hoc) allows sound artists to create fluent and temporally aligned sounds with a “human” (i.e., gestural) touch. Adding this gestural touch to the resulting sound composition often results in a sonic product of great aesthetic and production value. Recent research in generative modeling for sound has paved the way for text-to-sound systems [ 2 , 3 , 4 ] , where a user can create sound samples from text descriptions of a sound (e.g., “explosion”). While the text-to-sound paradigm can help a sound designer find sounds more quickly (and, perhaps in the future, with a higher degree of specificity), a sound designer still has to painstakingly modify the temporal characteristics of the generated sound so that they can be in sync with the visuals in the editing timeline. This is in opposition to the natural way that Foley artists gesturally create sound effects by physically performing with physical soundmaking objects. To overcome the drawbacks of a purely text-to-audio interaction, several works in the music domain sought to condition generative models on audio [ 5 ] , parallel instrument stems [ 6 ] , melody [ 7 ] , sound event timestamps and frequency [ 8 ] , or multiple structural control signals like song structure and dynamics [ 9 ] . Notably, [ 10 ] condition an audio VAE on control signals such as brightness and loudness, though their experiments are limited to models trained on narrow sound distributions (e.g., violin, darbouka, speech) and not a multi-distribution text-to-audio model. For speech, [ 11 ] proposes a fully interpretable and disentangled representation for speech generation and editing, which allows for fine-grained control over the pitch, loudness, and phonetic pronunciation of speech. The human voice is a gestural sonic instrument [ 12 ] : it allows us to realize sounds without having to perform any symbolic abstraction (i.e., putting a sound into words) beforehand. When humans communicate audio concepts to other people, they typically combine language and vocal imitation [ 13 , 14 , 15 ] , and recent work has shown its utility for query-by-example search of audio databases [ 16 , 17 , 18 ] . This is a more natural method than describing the evolution of pitch, timing, and timbre via pure text descriptions [ 13 ] , and voice-driven sound synthesis interactions have been of interest long before modern generative modelling for their embodied capabilities [ 19 , 20 , 21 , 22 , 23 , 24 ] . We propose Sketch2Sound: a text-to-audio model that can create high-quality sounds from sonic imitation prompts by following interpretable, fine-grained time-varying control signals that can be easily extracted from any audio signal at different levels of temporal detail: loudness, brightness ( spectral centroid ) and pitch . We expand upon previous work [ 25 ] by developing a method capable of following the loudness, brightness and pitch of a vocal imitation, with the option to drop any of the three controls. Additionally, we propose a technique that varies the temporal detail of the control signals used during training by applying median filters of different window sizes to the control signals before using them as input. This allows sound artists to specify the degree of temporal precision to which a generative model should follow the specified control signals, which improves sound quality in sounds that may be too hard to perfectly imitate with one’s voice. This method is not limited to just vocal imitation : any kind of sonic imitation can be used to drive our proposed generative model – we place the focus on vocal imitation due to people’s innate ability to imitate sounds with our voices. Vocal imitations can always be augmented through other sonic gestures like clapping, tapping, playing instruments, etc. Sketch2Sound can be added to any existing latent diffusion transformer (DiT) sound generation model with as little as 40k fine-tuning steps. Unlike ControlNet methods [ 26 , 27 ] that require an extra trainable copy of the entire neural network encoder, Sketch2Sound requires only a single linear layer per control. Our experiments show that Sketch2Sound can generate sounds that closely follow the input control signals (loudness, spectral centroid, and pitch/periodicity) from a vocal imitation while still achieving a high degree of adherence to a text prompt and an audio quality comparable to the text-only pre-trained model. We show that our median filtering technique leads to improved audio quality and text adherence when generating sounds from vocal imitations. We also show that, during inference, a user can arbitrarily specify a degree of temporal detail by choosing a median filter size, allowing them to navigate the trade-off between strict adherence to the vocal imitations and audio quality + text adherence. To the best of our knowledge, this is the first sound generation model capable of following vocal imitations and text prompts by conditioning on a set of holistic control signals suitable for generating sound objects with fine-grained, gestural control of pitch, loudness, and brightness. We believe Sketch2Sound will give sound artists a more expressive, controllable, and gestural interaction for generating sound-objects than existing text-to-audio and other conditional sound generation systems. We highly encourage the reader to listen to our expansive set of audio examples demonstrating Sketch2Sound. 2 2 2 https://hugofloresgarcia.art/sketch2sound TABLE I: Control Signal Evaluation and Sketch Type Ablation. Control Signals : Introducing