Zero-Shot Mono-to-Binaural Speech Synthesis

Alon, Julian, Soroosh, RJ, Nadav, Bastiaan, Eliya · 2024

Read Zero-Shot Mono-to-Binaural Speech Synthesis by Alon, Julian, Soroosh, RJ, Nadav, Bastiaan, Eliya (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08356.txt.

Description

A single-channel recording lacks the sense of space that two ears give, yet training a model to add that space once needed paired two-channel data. Levkovitch and co-authors present a method in 2024 that needs no such training at all. A plain geometry step gives a first draft of the two-channel sound.

That step warps time and scales loudness by where the source sits. A pre-trained voice tool then polishes the draft in rounds. The team also built a fresh test set to judge methods on room types they had not seen. On the standard set, the new method matches trained rivals by ear. On the new set, it beats them. The result hints that large pre-trained tools carry more latent skill than their training aimed at.

Catalog record

CREATORAlon, Julian, Soroosh, RJ, Nadav, Bastiaan, Eliya
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08356

Browse the full library · About the Tower of Babel