MMEDIT: A Unified Framework for Multi-Type Audio Editing via Audio Language Model † Work done partially during internship at Shanghai AI Lab. ∗ Corresponding author.
Ye Tao †1,2 , Xuenan Xu 1,2 , Wen Wu 2 , Shuai Wang 3 , Mengyue Wu 1 , Chao Zhang ∗2 · 2025
Read MMEDIT: A Unified Framework for Multi-Type Audio Editing via Audio Language Model † Work done partially during internship at Shanghai AI Lab. ∗ Corresponding author. by Ye Tao †1,2 , Xuenan Xu 1,2 , Wen Wu 2 , Shuai Wang 3 , Mengyue Wu 1 , Chao Zhang ∗2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20339.txt.
Introduction
Ye Tao and co-authors built MMEdit, a system that edits audio following plain text instructions. A good edit changes only the target sound and leaves the rest untouched. Earlier training-free methods hurt the signal, while trained systems suffered from scarce paired data and narrow task coverage.
MMEdit widens the task list to addition, replacement, removal, reordering, and attribute changes. A scalable synthesis pipeline builds large paired data sets with event-level labels to train on. The design pairs a Qwen2-Audio encoder with a diffusion-style generator so instructions align tightly with acoustic context. Experiments show stronger edit placement, better command following, and cleaner keeping of unedited regions than prior approaches.
Catalog record
| CREATOR | Ye Tao †1,2 , Xuenan Xu 1,2 , Wen Wu 2 , Shuai Wang 3 , Mengyue Wu 1 , Chao Zhang ∗2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20339
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMEDIT: A Unified Framework for Multi-Type Audio Editing via Audio Language Model † Work done partially during internship at Shanghai AI Lab. ∗ Corresponding author. Ye Tao †1,2 , Xuenan Xu 1,2 , Wen Wu 2 , Shuai Wang 3 , Mengyue Wu 1 , Chao Zhang ∗2 Abstract Text-guided audio editing requires modifying specific acoustic events while strictly preserving non-target content. Despite recent progress, existing approaches remain fundamentally limited: training-free methods often suffer from signal degradation due to diffusion inversion. Training-based methods exhibit better generation quality but remain severely constrained by the scarcity of high-quality paired data and task formulations that cover only a narrow subset of editing operations. Moreover, standard architectures in this paradigm often decouple text and audio processing, limiting the model’s ability to align instructions with specific acoustic contexts. To overcome these constraints, we propose MMEdit , an ALM-driven framework for unified audio editing. We systematically extend task definitions to cover a comprehensive range of operations, including addition, replacement, removal, reordering, and attribute modification. Furthermore, we construct a scalable synthesis pipeline to generate large-scale paired data with fine-grained event-level annotations. To capture complex editing semantics, we integrate Qwen2-Audio encoder with an MMDiT-based generator, achieving precise cross-modal alignment and localized editing. Experiments demonstrate that our approach delivers superior editing localization accuracy, robust instruction following, and high fidelity in non-edited regions. The project page can be accessed at Project Page . I Introduction With the rapid development of generative modeling, modern text-to-audio (TTA) systems [ 22 , 10 , 11 ] are now capable of producing high-fidelity, diverse, and text-aligned sounds from natural language prompts. Unlike TTA, which synthesizes audio from scratch, text-guided audio editing seeks to modify specific regions of an existing audio signal according to user-provided textual instructions, while preserving the remaining content unchanged. This paradigm offers users a more intuitive and efficient way to achieve fine-grained control over audio content through natural language, rather than relying on manual waveform editing or expert post-processing. Since the introduction of this task in AUDIT [ 17 ] , research in this field has attracted increasing research interest [ 19 , 12 , 8 , 13 ] . Existing studies on text-guided audio editing can be broadly categorized into training-free and training-based paradigms. Training-free approaches typically adapt pre-trained diffusion-based TTA models by inverting the denoising process to reconstruct latent noise and re-generate audio under new textual conditions. PPAE [ 19 ] and AudioEditor [ 8 ] are representative methods. Despite free from additional training, these approaches rely heavily on diffusion inversion, resulting in high inference cost and a strong dependence on full captions or handcrafted masks, restricting their applicability to complex editing tasks. Crucially, the mechanism of inversion and subsequent regeneration may compromise the fidelity to the original structure and fine details. In contrast, training-based frameworks explicitly learn instruction-conditioned generation models from paired data. AUDIT [ 17 ] pioneered this paradigm by training a latent diffusion model. Paissan et al. [ 13 ] explored prompt interpolation and fine-tuning within the embedding space, while RFM-Editing [ 5 ] incorporated rectified flow matching to improve efficiency. Compared to training-free inversion, this paradigm offers better fidelity and faster inference speeds. Despite these advances, the progress of audio editing faces challenges in both data resources and model design . In terms of data , real editing pairs are rarely available, and existing studies thus rely on small-scale or manually synthesized datasets to simulate editing operations. However, prior synthetic pipelines often adopt simple mixing strategies with limited event-level controllability, resulting in incomplete coverage of edit types, and therefore failing to reflect a comprehensive range of real-world manipulations. The absence of an open and unified benchmark further exacerbates the challenge, leading to inconsistent evaluation and limited reproducibility across different studies. In terms of model design , most existing works rely on text-only encoders to process instructions. The interaction between textual instructions and the source audio is limited to the cross-attention mechanism in the generation model. Yet in complex audio scenes, accurately editing a specific event entails not only interpreting the instruction but also locating the event within the source audio, a capability that text-only semantic encoders inherently lack. Consequently, such a decoupled, late-fusion design is often inadequate for achieving context-aware understanding of complex editing instructions. To address these limitations, we propose MMEdit , a unified framework driven by an Audio Language Model for general, instruction-following audio editing. First, we systematically expand the editing scope to six representative operations: addition , removal , replacement , loudness adjustment , reordering , and speed modification , and develop a scalable synthetic pipeline, together with a unified benchmark for standardized evaluation. Although some operations can be achieved by signal processing tools in single-event audio, event overlap in realistic soundscapes makes event-specific editing difficult. Therefore we integrate them into a unified editing model to learn shared audio editing capabilities across operations. Moreover, MMEdit utilizes an Audio Language Model (ALM) to process instructions and audio content jointly. This unified processing enables precise cross-modal alignment, allowing the model to precisely interpret instructions within the specific acoustic context. The ALM encoder is integrated with an MMDiT-based generator [ 4 ] to generate high-fidelity audio. Our main contributions are summarized as follows: • Comprehensive Task Definition: We systematically expand the editing landscape to encompass six representative operations, better reflecting real-world requirements. • Scalable Data Pipeline and Unified Benchmark: We develop a synthetic generation pipeline that produces large-scale paired data with fine-grained, event-level annotations. To foster reproducibility, we establish and open-source an evaluation benchmark. 1 1 1 Will be released upon acceptance. • ALM-Driven Architecture and Superior Performance: MMEdit involves the ALM encoder with MMDiT to enhance the model’s ability to understand complex editing instructions for precise editing. Extensive experiments demonstrate that MMEdit significantly outperforms existing baselines in following editing instructions and maintaining the integrity of unedited content. II Data pipeline To facilitate scalable training and evaluation, we construct a large-scale synthetic dataset comprising over one million audio editing pairs. Each pair is represented as a triplet \langle\text{instruction},\text{audio}_{\text{source}},\text{audio}_{\text{edited}}\rangle . The construction pipeline, illustrated in Fig. 1 , comprises three stages: segment curation, audio composition, and instruction synthesis. Fig. 1: Construction pipeline and examples. (1) Extract clean event-level segments from AudioCaps using TAG model and CLAP filtering; (2) Simulate foreground–background compositions and record metadata; (3) Generate template-based instructions for each editing task. II-A Audio Segment Curation Existing text-guided audio editing studies like AUDIT typically synthesize training pairs by combining entire audio clips with their full global captions. However, to enable more precise and interpretable editing such as specifying event timing and occurrence counts, a finer granularity is required. We first perform event-level segmentation and grounding on an audio-caption corpora, AudioCaps [ 9 ] . Specifically, inspired by AudioTime [ 18 ] , we employ an LLM to decompose each caption into single-event descriptions. The descriptions are then timestamped by the Text-to-Audio Grounding (TAG) model [ 20 ] to determine event boundaries, and only the non-overlapping segments are retained and cropped as base audio units for subsequent synthesis. Finally, to guarantee semantic alignment, we compute CLAP scores [ 3 ] between the textual label and the extracted segment and discard pairs with a similarity score below 0.3