Making Separation-First Multi-Stream Audio Watermarking Feasible via Joint Training
Houmin, Zi, Linxi, Yechen, Liwei, Ming · 2026
Read Making Separation-First Multi-Stream Audio Watermarking Feasible via Joint Training by Houmin, Zi, Linxi, Yechen, Liwei, Ming (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16805.txt.
Introduction
Sun and co-authors ask whether each stem of a mixed audio track can carry its own watermark. Their setup embeds a distinct watermark in each stem, then mixes, separates, and decodes each output. A naive pairing of standard watermarking with off-the-shelf separation recovers bits poorly. Separation artifacts, it turns out, differ from generic distortions. The team therefore trains the watermarking with the separator inside the detection loop. On speech-plus-music and vocal-plus-accompaniment mixes, post-separation recovery improves a lot while sound quality holds up.
Catalog record
| CREATOR | Houmin, Zi, Linxi, Yechen, Liwei, Ming |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23230 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16805
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Making Separation-First Multi-Stream Audio Watermarking Feasible via Joint Training Houmin, Zi, Linxi, Yechen, Liwei, Ming Abstract Modern audio is created by mixing stems from different sources, raising the question: can we independently watermark each stem and recover all watermarks after separation? We study a separation-first, multi-stream watermarking framework –embedding distinct information into stems using unique keys but a shared structure, mixing, separating, and decoding from each output. A naive pipeline (robust watermarking + off-the-shelf separation) yields poor bit recovery, showing robustness to generic distortions does not ensure robustness to separation artifacts. To enable this, we jointly train the watermark system and the separator in an end-to-end manner, encouraging the separator to preserve watermark cues while adapting embedding to separation-specific distortions. Experiments on speech+music and vocal+accompaniment mixtures show substantial gains in post-separation recovery while maintaining perceptual quality. Sun Hu Li Wang Jin Li keywords audio watermarking, source separation, multi-stream, joint training † † address: 1 Digital Innovation Research Center, Duke Kunshan University, Kunshan, China 2 School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China 3 OfSpectrum, Inc., Los Angeles, USA † † email: [email protected] 1 Introduction Recent advances in AI-generated speech [ 1 , 2 , 3 , 4 ] and neural audio editing have made it increasingly difficult to protect the copyright and provenance of digital audio content. Audio watermarking has therefore emerged as a key technology for copyright protection and tamper detection [ 5 ] . Classical audio watermarking methods usually operate directly in the time or frequency domain, such as phase coding, spread-spectrum embedding, and low-bit coding [ 6 ] . To improve robustness and payload capacity, deep neural audio watermarking methods have been proposed, including WavMark [ 7 ] , MaskMark [ 8 ] , and SilentCipher [ 9 ] . These approaches demonstrate strong robustness–imperceptibility trade-offs, enabling higher payloads while maintaining perceptual audio quality. However, modern generative pipelines often combine audio from multiple sources and rights holders. A single synthetic waveform may contain a watermarked music accompaniment and a separately watermarked or unwatermarked vocal stream. Existing watermarking systems are typically designed for single-watermark, single-carrier settings and degrade sharply in such multi-source mixtures. A natural strategy is to apply a music source separator (e.g., Demucs [ 10 ] , TF-GridNet [ 11 ] , Open-Unmix [ 12 ] , or Spleeter [ 13 ] ) to decompose the mixture into stems and then detect watermarks per stem. In practice, separation is not lossless: even state-of-the-art separators introduce artifacts that may corrupt watermark-bearing cues, leading to low detection rates despite acceptable perceptual quality [ 14 ] . In this paper, we study whether independently watermarking different stems and decoding after source separation is feasible in realistic mixtures. We show that directly combining a strong watermark with an off-the-shelf separator often fails due to a mismatch between robustness to generic distortions and robustness to separation-induced artifacts. We then demonstrate that end-to-end joint training of the watermarking system and the separator together makes this separation-first Multi-Stream framework feasible. Our contribution can be summarized as follows: • We formalize a separation-first, multi-stream audio watermarking evaluation setting. • We show that strong neural watermarking methods degrade to unacceptable decoding after separation. • We demonstrate that joint end-to-end training substantially improves post-separation robustness while maintaining perceptual quality. 2 Method 2.1 Problem Setup: Separation-First Multi-Stream Watermarking We consider mixtures formed by multiple stems (e.g., vocal and accompaniment) that may originate from different sources. Each stem carries an independent watermark embedded with a distinct secret key. Given stems \{x_{i}\} , watermarks are embedded independently to obtain \{\tilde{x}_{i}\} . The watermarked stems are mixed into a single waveform: y=\sum_{i}\tilde{x}_{i}. (1) A source separator S(\cdot) is applied to the mixture to obtain separated estimates \{\hat{x}_{i}\}=S(y) . The corresponding watermark is then decoded independently from each separated stem \hat{x}_{i} . 2.2 Watermarking Backbone We adopt a key-conditioned Conformer [ 15 ] -based neural audio watermarking backbone derived from a recently accepted yet unpublished deep watermarking framework, AURA [ 16 ] , as our watermarking backbone 1 1 1 An anonymized manuscript of [ 16 ] (aura.pdf) is provided in the detailed supplementary material for reviewer reference. It has been peer-reviewed and accepted for publication at ICASSP 2026. Audio samples are available at: https://anonymous114810.github.io/SF-MSAW-JT/ . Figure 1: The joint training pipeline for the proposed separation-first audio watermarking framework. In our experiments, {\mathbf{i}}=1,2 2.2.1 Encoder. Given an n -second audio segment and a binary secret key, the encoder produces a bounded watermark perturbation in the time–frequency domain. The audio is transformed into a magnitude spectrogram, while the key is embedded into a conditioning vector. A stack of Conformer [ 15 ] blocks, equipped with Feature-wise Linear Modulation (FiLM) layers [ 17 ] immediately after the LayerNorm of each of its four core modules (two FFNs, MHSA, and CONV), predicts a real-valued modification mask applied to the magnitude spectrogram. The modified spectrogram is then inverted to the time domain using the original phase to produce the watermarked waveform. The modified spectrogram is inverted to wavedomain combined with the original phase to produce the watermarked waveform. 2.2.2 Decoder. The decoder takes a potentially distorted audio segment and predicts the embedded bits. It follows a RobustDNN-style [ 18 ] architecture operating directly on the waveform, using multiple 1D convolutional layers to extract watermark-related features before bit prediction. 2.3 Source Separation Network 2.3.1 Demucs We employ Demucs [ 10 ] as our source separation network. Demucs operates directly in the time domain, which is crucial for our framework as it avoids potential artifacts introduced by time-frequency transformations that could misalign or distort the embedded watermark signals. Its U-Net-like encoder-decoder architecture with strided convolutions enables multi-scale feature extraction while maintaining fine temporal details through skip connections [ 19 ] . The bidirectional LSTM [ 20 ] layers at the bottleneck effectively model long-range dependencies in audio, which is essential for coherently separating overlapping sources and preserving the continuity of watermark patterns. We use the standard Demucs architecture without modification to leverage its proven separation performance and ensure our findings are not biased by architectural changes. 2.4 Impact of Separation on Watermarking Source separation is inherently lossy [ 21 ] . Audio separators often introduce artifacts such as phase misalignment, spectral smearing [ 22 ] , and cross-source interference (e.g., residual vocals in the separated accompaniment track)-limitations explicitly acknowledged by the authors of Demucs themselves [ 14 ] . These artifacts act as non-stationary, structured noise that is fundamentally different from the generic distortions (e.g., noise addition, compression) against which most watermarking systems are robust. Consequently, a watermark decoder trained only for generic robustness fails to decode correctly from separated stems, as shown in our experiments (Section 4). This mismatch necessitates a co-design approach where the watermarking system learns to be robust to the specific distortion profile introduced by the separator, and the separator is encouraged to preserve features critical for watermark recovery. 2.5 Joint Training Pipeline To bridge the robustness gap identified in Section 2.4, we propose an end-to-end joint training framework that optimizes the watermarking encoder/decoder and the Demucs separator concurrently. This co-optimization is the core of our contribution. It allows the gradient from the watermark decoding loss to flow back