Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration
Haowen Li, Tianxiang Li, Yi Yang, Boyu Cao, Qi Liu † · 2026
Read Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration by Haowen Li, Tianxiang Li, Yi Yang, Boyu Cao, Qi Liu † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10203.txt.
Introduction
Haowen Li and four colleagues take on a precise music-production job. They change the sound of one instrument inside a dense mix, say turning a vocal line into a violin, while leaving every other track untouched. Their 2026 paper introduces Polyphonia, a system that edits music without retraining, built for polyphonic recordings where several parts sound at once.
Polyphonia pairs ordinary semantic attention with an acoustic prior. This sound-based guide marks rough borders around the target part, so the rest of the mix stays intact. The authors also built PolyEvalPrompts, a test set of 1,170 sound-change tasks. Target alignment rose by 15.5 percent over rival methods, while music quality and untouched tracks held up well.
Catalog record
| CREATOR | Haowen Li, Tianxiang Li, Yi Yang, Boyu Cao, Qi Liu † |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10203
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration Haowen Li, Tianxiang Li, Yi Yang, Boyu Cao, Qi Liu † Abstract The advancement of diffusion-based text-to-music generation has opened new avenues for zero-shot music editing. However, existing methods fail to achieve stem-specific timbre transfer, which requires altering specific stems while strictly preserving the background accompaniment. This limitation severely hinders practical application, since real-world production necessitates precise manipulation of components within dense mixtures. Our key finding is that, while vanilla cross-attention captures semantic features of stems, it lacks the spectral resolution to strictly localize targets in dense mixtures, leading to boundary leakage. To resolve this dilemma, we propose Polyphonia , a zero-shot editing framework with Acoustic-Informed Attention Calibration. Rather than relying solely on diffuse semantic attention, Polyphonia leverages a probabilistic acoustic prior to establish coarse boundaries, enabling non-target stems preserved precise semantic synthesis. For evaluation, we propose PolyEvalPrompts , a standardized prompt set with 1,170 timbre transfer tasks in polyphonic music. Specifically, Polyphonia achieves an increase of 15.5% in target alignment compared to baselines, while maintaining competitive music fidelity and non-target integrity. † † affiliationnotice: Equal contribution 1 Introduction The landscape of generative music has been revolutionized by Diffusion Models ( 14 ; 18 ; 31 ; 12 ; 10 ) , enabling the synthesis of high-fidelity music from textual descriptions. However, the transition from novelty tools to professional music production assistants necessitates more than just generation; surgical editing controllability is paramount. In professional workflows, text-driven music editing generally addresses two levels of manipulation: inter-stem editing (e.g., adding or removing a specific instrument) and intra-stem editing (e.g., replacing a specific instrument). Within the latter, stem-specific timbre transfer represents a critical yet challenging task: surgically altering the timbre of a target stem (e.g., transforming a vocals track into a violin) while strictly preserving the acoustic integrity of its original pitch contours, rhythmic structure and all non-target stems (e.g., the co-occurring melody and accompaniment). This task poses a dual challenge: the newly transferred timbre must accurately reflect the semantic prompt while seamlessly blending with the rigidly preserved non-target stems. Figure 1 : Illustration of stem-specific timbre transfer with Polyphonia compared against baselines on two tasks. While MusicGen and DDPM-Friendly suffer from Non-target Distortion where the vocals are distorted, Polyphonia achieves robust Non-Target Preservation , strictly preserving the background rhythm. The right panel displays a vocal-to-saxophone transfer. Furthermore, Melodia and SteerMusic fail to disentangle the target from the accompaniment, resulting in Target Misalignment . In contrast, Polyphonia successfully synthesizes the desired timbre with precise Target Alignment . While zero-shot editing paradigms have shown promise, applying them to multi-track music mixtures is impeded by severe spectral interference. As illustrated in Fig. 1 , existing approaches generally succumb to a dilemma rooted in their attention mechanisms. The first category ( 7 ; 21 ; 23 ) relies on vanilla cross-attention. While effective at capturing semantic features, it struggles to confine the generation within precise spectral boundaries in dense mixtures. This leads to boundary leakage, causing the non-target stems to be regenerated alongside the target. To mitigate this, the second category ( 43 ; 25 ; 47 ) introduces rigid preservation constraints via internal feature preservation. However, in dense mixtures, these strict constraints indiscriminately preserve entangled features, conflicting with the editing objective and preventing the target from manifesting. Essentially, current models lack an acoustic boundary to distinguish the “to-be-edited” content from the “to-be-preserved” non-target stems. This diagnosis points to a crucial conclusion: to achieve precise stem-specific timbre transfer, the localization paradigm needs to incorporate acoustic prior to constrain the diffuse nature of semantic attention. To realize this, we propose Polyphonia , a zero-shot framework designed to tackle this semantic-acoustic misalignment, resolving the mismatch where high-level semantic attention fails to align with the fine-grained precision of acoustic energy distribution. Surpassing naive separation-remixing strategies and naive magnitude masks, Polyphonia rigorously formulates a probabilistic acoustic prior called Ideal Ratio Mask (IRM) ( 38 ; 24 ) by analyzing the spectral energy competition within the mixture via Blind Source Separation (BSS). Subsequently, we introduce Acoustic-Informed Attention Calibration to leverage the acoustic prior to selectively preserve background features while aligning generative focus to the target, enforcing precise spectral editing control. To standardize the evaluation of stem-specific timbre transfer, we construct PolyEvalPrompts , a comprehensive prompt set applied to the MUSDB18-HQ ( 28 ) test subset and MusicDelta ( 1 ) datasets. Extensive experiments demonstrate that Polyphonia outperforms state-of-the-art (SOTA) baselines, achieving the highest textual alignment (CLAP) and an optimal balance between editing strength and non-target integrity. Our contributions are summarized as follows: • We formalize the Semantic-Acoustic Misalignment as a fundamental barrier, demonstrating that semantic attention inherently lacks the spectral resolution to localize targets in dense mixtures. • We propose Polyphonia , a zero-shot framework that introduces Acoustic-Informed Attention Calibration . By injecting a rigorously formulated probabilistic acoustic prior, it solves the semantic-acoustic misalignment, strictly preserving the non-target stems while enforcing target semantics. • We contribute PolyEvalPrompts , a standardized prompt set with 1,170 editing tasks applied to MUSDB18-HQ and MusicDelta, establishing a reproducible testbed for assessing stem-specific timbre transfer. 2 Related Work 2.1 Text-to-Music Generation The domain of music generation has been reshaped by autoregressive transformers ( 7 ) and Latent Diffusion Models (LDMs) ( 17 ; 18 ; 31 ; 12 ) . While models like AudioLDM 2 ( 18 ) and Stable Audio ( 10 ) achieve high-fidelity holistic generation, they fundamentally lack surgical controllability over individual acoustic components. To address this, recent research has resorted to supervised fine-tuning. For instance, Music ControlNet ( 39 ) adapts the ControlNet architecture to music, requiring extensive retraining to inject specific spatially or temporally dense controls. Similarly, methods like Instruct-MusicGen ( 46 ) and Jen-1 Composer ( 45 ) employ extensive instruction tuning to enable editing capabilities. However, these supervised paradigms incur prohibitive computational costs, require massive paired datasets during training. 2.2 Zero-Shot Text-to-Music Editing To bypass fine-tuning costs, research has pivoted toward zero-shot editing pipelines. Early approaches adapted global processing techniques like SDEdit ( 23 ) and DDIM/DDPM Inversion ( 34 ; 21 ) to music. However, operating on the entire latent space without localization often leads to structural collapse or non-target distortion, where the background accompaniment is inadvertently altered ( 7 ) . To improve preservation, recent works have sought to impose constraints via internal feature manipulation. Approaches such as Melodia ( 43 ) , MEDIC ( 19 ) , and PPAE ( 41 ) extend attention manipulation paradigms successful in vision ( 13 ; 36 ; 3 ) to the music domain. Notably, PPAE is primarily optimized for general audio events with sparse spectral layouts; in contrast, our task focuses on highly entangled polyphonic music, where targets are deeply embedded and spectrally overlapped within dense mixtures. Others like SteerMusic ( 25 ) and MusicMagus ( 47 ) employ gradient-based energy guidance. Nevertheless, these frameworks fundamentally rely on internal representations (e.g., self/cross-attention maps or energy gradients) to maintain coherence. We argue that in dense multi-track mixtures, these internal features are inherently contaminated by spectral interference, rendering them insufficient for precise disentanglement. 2.3 Audio-Guided Masking Paradigms The utilization of audio cues to construct spatial masks has been predominantly explored