A 2 SB : Audio-to-Audio Schrödinger Bridges
Zhifeng Kong, Kevin J Shih, Weili Nie, Arash Vahdat, Sang-gil Lee · 2025
Read A 2 SB : Audio-to-Audio Schrödinger Bridges by Zhifeng Kong, Kevin J Shih, Weili Nie, Arash Vahdat, Sang-gil Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11311.txt.
Introduction
Music recordings often arrive damaged: high notes clipped off by compression, or whole moments missing. Zhifeng Kong and co-authors present a restoration model built on Schrödinger bridges, a method that learns a direct path between damaged and clean audio. It works end to end on the waveform itself, with no separate vocoder stage.
The model handles full-quality music at 44.1 kilohertz and can restore stretches an hour long. It both extends bandwidth, inventing the missing high frequencies, and inpaints, filling gaps seamlessly. Trained only on permissively licensed music, it reached state-of-the-art quality on test sets drawn from outside its training data.
Catalog record
| CREATOR | Zhifeng Kong, Kevin J Shih, Weili Nie, Arash Vahdat, Sang-gil Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-11311
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A 2 SB : Audio-to-Audio Schrödinger Bridges Zhifeng Kong, Kevin J Shih, Weili Nie, Arash Vahdat, Sang-gil Lee Abstract Real-world audio is often degraded by numerous factors. This work presents an audio restoration model tailored for high-res music at 44.1kHz. Our model, Audio-to-Audio Schrödinger Bridges ( A 2 SB ), is capable of both bandwidth extension (predicting high-frequency components) and inpainting (re-generating missing segments). Critically, A 2 SB is end-to-end requiring no vocoder to predict waveform outputs, able to restore hour-long audio inputs, and trained on permissively licensed music data. A 2 SB is capable of achieving state-of-the-art bandwidth extension and inpainting quality on several out-of-distribution music test sets. GitHub Page: https://github.com/NVIDIA/diffusion-audio-restoration Checkpoints: https://huggingface.co/nvidia/audio_to_audio_schrodinger_bridge Demo Website: https://research.nvidia.com/labs/adlr/A2SB/ 1 Introduction Audio signals in the real world may be perturbed due to numerous factors such as recording devices, data compression, and online transferring. For instance, certain recording devices and compression methods may result in low sampling rate, and online transferring may cause a short audio segment to be lost. These problems are usually ill-posed ( Narayanaswamy et al., 2021 ; Moliner et al., 2023 ) and are usually solved with data-driven generative models. For instance, bandwidth extension methods have been proposed to up-sample the audio ( Lee & Han, 2021 ; Liu et al., 2022 ; Serrà et al., 2022 ; Moliner & Välimäki, 2022 ; Shuai et al., 2023 ; Yu et al., 2023 ; Kim et al., 2024 ; Liu et al., 2024 ; Ku et al., 2024 ; Yun et al., 2025 ) , and inpainting methods have been developed to predict segments where audio is missing ( Marafioti et al., 2019 ; Marafioti et al., 2020 ; Borsos et al., 2022 ; Liu et al., 2023b ; Moliner & Välimäki, 2023 ; Asaad et al., 2024 ) . Many of these methods are task-specific, designed for the speech domain, or trained to only restore the degraded magnitude – which requires an additional vocoder to transform the restored magnitude into a waveform. In this work, we investigate high-res music restoration, a more challenging task than speech restoration in terms of typical bandwidth of speech ( \leq 22.05kHz) vs music signals (44.1kHz). We aim to tackle bandwidth extension and inpainting in a single model as in practice it is easier to maintain one model serving multiple tasks. We also aim to build an end-to-end trainable generative model for audio restoration without need of a separate vocoder or a codec. To achieve our goals, we adopt the Schrödinger Bridge framework ( De Bortoli et al., 2021 ; Chen et al., 2021 ; Liu et al., 2023a ; Albergo et al., 2023 ) as it is suitable for translation tasks where a part of the source and target samples are well aligned. We name our model A 2 SB: Audio-to-Audio Schrödinger Bridges . The first challenge is curating a dataset that is both expansive enough to cover most genres of music of interest and being permissively licensed. To achieve this, we collected and filtered permissively licensed music data from public datasets, leading to 2.3K hours in total. As data quality varies significantly across datasets, we adopt the common pre-training and fine-tuning approach ( Ouyang et al., 2022 ) . The second challenge is to support both restoration tasks in a single model. We frame both tasks as the generative spectrogram inpainting task: bandwidth extension as inpainting the high-frequency part of the spectrogram along the frequency axis, and audio inpainting as frame inpainting along the time axis. The rest of the spectrogram should exactly match the input. We adopt the Schrödinger Bridge formulation ( Liu et al., 2023a ; Albergo et al., 2023 ) as it is particularly suitable for our generative spectrogram inpainting setup. The third and most significant challenge is to train an end-to-end model without using a vocoder or a codec. While prior works such as ( Richter et al., 2023 ; Jukić et al., 2024 ; Ku et al., 2024 ) found success training directly on the complex spectrogram for speech enhancement, we find this ineffective when we need to synthesize missing spectrogram data. We use a factorized audio representation with power compression of the magnitude and trigonometric representation of the phase. We additionally apply phase orthogonalization based on the solution of the Procrustes problem to ensure that the generated phase values are consistent. These techniques make A 2 SB the first end-to-end high-res music restoration model. It is also advantageous over prior works that restore only the magnitude spectrum and apply a vocoder ( Liu et al., 2024 ; Liu et al., 2023b ) , as we preserve the original phase values where available. The fourth challenge is to apply our restoration model – which is trained on a fixed segment length – to very long audio inputs. Directly concatenating individually restored segments could produce boundary artifacts especially for the bandwidth extension task. To solve this problem, we adapt MultiDiffusion ( Bar-Tal et al., 2023 ) , which was originally designed for image panorama generation, to our model. We apply MultiDiffusion in overlapping sliding windows along the audio time axis, which allows A 2 SB to produce coherently restored outputs for hour-long sequences. We demonstrate the effectiveness of our A 2 SB on several out-of-distribution test sets. Our model outperforms state-of-the-art baselines on these benchmarks. We also demonstrate the effectiveness of our factorized audio representation, phase orthogonalization, and inference methods qualitatively and quantitatively. We summarize our contributions as follows: 1. We propose A 2 SB , a state-of-the-art, end-to-end (vocoder-free), and multi-task diffusion Schrödinger Bridge model for 44.1kHz high-res music restoration. 2. We propose an effective factorized audio representation and a phase orthogonalization method to achieve end-to-end training and generation. 3. We curate a large collection of permissively licensed public datasets for high-res music restoration, and design a training strategy for better utilization of the data. 4. A 2 SB can coherently restore hour-long audio without boundary artifacts. Inpainting Bandwidth Extension Clean \left\{\rule{0.0pt}{42.67912pt}\right. \left.\rule{0.0pt}{42.67912pt}\right\} A 2 SB Figure 1: A 2 SB targets music restoration with a focus on inpainting and bandwidth extension, each corresponding to a specific corruption pattern in spectrogram. The model is then trained to fit the diffusion Schrödinger Bridge process from the corrupted distribution to the clean distribution. 2 Related Works Table 1: Comparison between our A 2 SB with prior works on music restoration models. In addition to having better generation quality, our method is end-to-end without a vocoder or a codec, supports multiple restoration tasks, applies to high sampling rate, and supports restoration for very long audio. We aim to provide a detailed study of our proposed approaches for achieving these properties, and note that many of our solutions can be incorporated into prior frameworks. \spadesuit Liu et al. (2024) and Liu et al. (2023b) ; \blacklozenge Wang et al. (2023) ; \clubsuit Moliner et al. (2023) and Moliner & Välimäki (2023) . Model End-to-end Multiple tasks Sampling rate Long audio restoration Conditional diffusion model ♠ ✗ ✗ 48kHz ✗ Instruction based model ⧫ ✗ ✓ 22.05kHz ✗ Inverse method ♣ ✓ ✓ 22.05kHz ✗ A 2 SB ✓ ✓ 44.1kHz ✓ 2.1 Diffusion Models Diffusion models and score-based generative models (