DegDiT: Controllable Audio Generation with Dynamic Event Graph Guided Diffusion Transformer

Yisu Liu, Chenxing Li, Wanqian Zhang, Wenfu Wang, Meng Yu, Ruibo Fu, Zheng Lin, Weiping Wang, Dong Yu · 2025

Read DegDiT: Controllable Audio Generation with Dynamic Event Graph Guided Diffusion Transformer by Yisu Liu, Chenxing Li, Wanqian Zhang, Wenfu Wang, Meng Yu, Ruibo Fu, Zheng Lin, Weiping Wang, Dong Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13786.txt.

Introduction

Yisu Liu and eight colleagues give audio generation a storyboard: a graph of sound events with times attached. Text-to-audio tools often smear when events start and stop, or miss requested sounds. Their system, DegDiT, turns the prompt's events into a dynamic graph before any sound is made.

Each graph node packs what the event is, when it runs, and how it links to its neighbors. A graph transformer blends those into guidance for a diffusion transformer that renders the audio. A cleaned, quality-scored training set and a consensus reward step sharpen the output. Benchmarks on three data sets put the method at the front.

Catalog record

CREATORYisu Liu, Chenxing Li, Wanqian Zhang, Wenfu Wang, Meng Yu, Ruibo Fu, Zheng Lin, Weiping Wang, Dong Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13786

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DegDiT: Controllable Audio Generation with Dynamic Event Graph Guided Diffusion Transformer Yisu Liu, Chenxing Li, Wanqian Zhang, Wenfu Wang, Meng Yu, Ruibo Fu, Zheng Lin, Weiping Wang, Dong Yu Abstract Controllable text-to-audio generation aims to synthesize audio from textual descriptions while satisfying user-specified constraints, including event types, temporal sequences, and onset and offset timestamps. This enables precise control over both the content and temporal structure of the generated audio. Despite recent progress, existing methods still face inherent trade-offs among accurate temporal localization, open-vocabulary scalability, and practical efficiency. To address these challenges, we propose DegDiT , a novel dynamic event graph-guided diffusion transformer framework for open-vocabulary controllable audio generation. DegDiT encodes the events in the description as structured dynamic graphs. The nodes in each graph are designed to represent three aspects: semantic features, temporal attributes, and inter-event connections. A graph transformer is employed to integrate these nodes and produce contextualized event embeddings that serve as guidance for the diffusion model. To ensure high-quality and diverse training data, we introduce a quality-balanced data selection pipeline that combines hierarchical event annotation with multi-criteria quality scoring, resulting in a curated dataset with semantic diversity. Furthermore, we present consensus preference optimization, facilitating audio generation through consensus among multiple reward signals. Extensive experiments on AudioCondition, DESED, and AudioTime datasets demonstrate that DegDiT achieves state-of-the-art performances across a variety of objective and subjective evaluation metrics. Some generated examples are available at DegDiT Demo Page . I Introduction Diffusion models [ 1 , 2 ] have established themselves as a dominant framework for generative artificial intelligence, demonstrating remarkable success across various modalities. These models, grounded in the theory of stochastic differential equations [ 3 ] , gradually transform the random noise into high-quality samples. This iterative refinement process has demonstrated particular effectiveness for complex data distributions, leading to breakthroughs in image generation with models like Stable Diffusion [ 4 ] and DALL·E [ 5 ] , as well as in video synthesis such as Sora [ 6 ] . The inherent flexibility of diffusion models has also enabled their successful adaptation to audio generation tasks [ 7 ] , where they have shown superior performance compared to traditional generative approaches like GANs [ 8 , 9 ] or autoregressive models [ 10 ] . Fig. 1: Comparison between naive text-only guided (left) and our proposed dynamic event graph (DEG) guided (right) diffusion model generation. The naive text-only guided approach relies solely on textual input to drive the diffusion process. By integrating structured event graphs, our approach effectively captures temporal dependencies, enabling the generation of accurate and consistent timestamps. Building upon these advances, text-to-audio (TTA) generation [ 11 , 12 , 13 ] has emerged as a particularly promising application of diffusion models. TTA systems aim to synthesize realistic audio content from natural language descriptions, offering transformative potential for multimedia production, accessibility tools, and creative applications. Current state-of-the-art systems like AudioLDM [ 14 , 15 ] , Tango [ 12 , 16 ] and TangoFlux [ 17 ] have demonstrated impressive capabilities in audio generation. Despite these advances, achieving fine-grained control over the generated audio remains a significant challenge for TTA systems. Such limitations hinder the use of TTA systems in applications that require detailed audio control. To solve this problem, one might consider drawing inspiration from controllable image generation methods [ 18 , 19 , 20 ] , which have achieved remarkable success in manipulating visual content according to the user-specified conditions. However, these approaches cannot be directly adapted to audio due to modality differences, especially the temporal dynamics and continuous signal characteristics of audio data. Toward enabling precise control of sound events, recent works have explored distinct but complementary strategies. As the pioneer in conditional audio generation, MC-Diffusion [ 21 ] introduces multi-scale conditioning to simultaneously govern temporal and spectral attributes of generated audio. PicoAudio [ 22 ] addresses the precision of timestamps more aggressively by leveraging LLM to conduct a class-timestamp matrix as guidance. However, this approach is limited to a fixed set of classes due to the predefined dimensionality of the matrix. While AudioComposer [ 23 ] introduces a fully end-to-end natural language-based generation approach, it often suffers from missing audio events and degraded output quality due to its over-reliance on text descriptions. Most recently, FreeAudio [ 24 ] introduces a training-free method using aggregation of preliminary and timestamp-independent attention maps, which introduces operational complexity due to its reliance on carefully designed parameters. To address these issues, we propose DegDiT, which models sound events within the multi-event audio prompt as temporally dynamic graphs that evolve across the audio timeline, as depicted in Fig. 1 . Each graph comprises four node categories that collectively represent the event’s spatio-temporal signature: (1) event category, (2) frame-level occurrence, (3) temporal boundary attributes (localized onset/offset), and (4) inter-event relations. A graph transformer processes these dynamic graphs, aggregating the spatio-temporal context through attention mechanisms to produce contextualized embeddings. These enriched embeddings are then fused with the textual features and provided as guidance for the diffusion model. To establish an open-vocabulary setting while enhancing dataset diversity, we also introduce a Quality-Balanced Data Selection pipeline that combines hierarchical event annotation with multi-criteria quality scoring. Our approach utilizes HTS-AT [ 25 ] to detect the event boundaries and generate confidence-calibrated pseudo-labels of AudioSet [ 26 ] samples. We then conduct a systematic evaluation of each sample through four key metrics: event count, type diversity, temporal alignment accuracy, and duration plausibility, yielding a rigorously curated dataset. The diffusion model and the graph transformer are jointly trained on this curated dataset, followed by fine-tuning on the AudioCondition dataset [ 21 ] . Considering the multi-dimensional evaluation requirements in controllable audio generation, we propose Consensus Preference Optimization (CoPO), a reinforcement learning framework that integrates diverse reward signals for comprehensive preference modeling. Unlike conventional binary preference learning, CoPO leverages quantitative preference intensities derived from comprehensive quality assessments, including text alignment, event alignment, temporal accuracy, and audio quality. By aggregating these reward components with learnable weights, CoPO captures the consensus among multiple reward signals, facilitating more nuanced model optimization. In brief, our contributions are summarized as: 1. We introduce DegDiT, a diffusion transformer framework for open-vocabulary, fine-grained controllable audio generation. The temporally dynamic event graphs are designed to encode precise temporal attributes and inter-event relationships, which are modeled using a graph transformer encoder as guidance for the diffusion model. 2. A quality-balanced data selection pipeline is proposed to combine hierarchical event annotation and multi-criteria quality scoring, resulting in a curated dataset with acoustic and semantic diversity for model training. 3. We further develop Consensus Preference Optimization, a reinforcement learning-based method that leverages multi-dimensional reward signals for fine-grained preference modeling. 4. Extensive experiments show that DegDiT achieves state-of-the-art performance on the AudioCondition [ 21 ] , DESED [ 27 ] , and AudioTime [ 28 ] datasets across various subjective and objective evaluation metrics. II Related Work II-A Text-to-Audio Generation The advances in text-to-audio (TTA) generation have been propelled by the development of diffusion models. Auffusion [ 13 ] combines the strengths of diffusion models and large language models (LLMs) to improve the quality of TTA generation. Make-an-Audio [ 11 ] and AudioLDM [ 14 , 15 ] have explored

Read the complete text