SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton
2026
Read SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25498.txt.
Introduction
He and nine colleagues present SymphonyGen, a 3D hierarchical frame for orchestra music. Making such music needs form and dense scores, yet models struggle to balance scale and control. Its parts split bar, track, and event axes, keeping memory far below flat token streams. A beat-set harmony frame gives short-score control for outlines. Reward-based tuning with a CLaMP 3 reward refines the model, and clash-averse sampling cuts tonal clashes. Both post-training tricks cut dissonance while keeping melodic scores, and tests rate it above baselines.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27816 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25498
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton Abstract Generating symphonic music requires simultaneously managing high-level structural form and dense, multi-track orchestration. Existing symbolic models often struggle with a “complexity-control imbalance”, in which scaling bottlenecks limit long-term granular steerability. We present SymphonyGen, a 3D hierarchical framework for contemporary cinematic orchestration. SymphonyGen employs a cascading decoder architecture that decomposes the Bar, Track, and Event axes, improving computational efficiency and scalability over conventional 1D or 2D models. We introduce “short-score” conditioning via a beat-quantized multi-voice harmony skeleton, enabling outline control while preserving textural diversity. The model is further refined using Group Relative Policy Optimization (GRPO) with a cross-modal audio-perceptual reward, aligning symbolic output with modern acoustic expectations. Additionally, we implement a dissonance-averse sampling algorithm to suppress unintended tonal clashes during inference. Objective evaluations show that both reinforcement learning and dissonance-averse sampling effectively enhance harmonic cleanliness while maintaining melodic expression. Subjective evaluations demonstrate that SymphonyGen outperforms baselines in musicality and preference for orchestral music generation. Demo page: https://symphonygen.github.io/ 1 Introduction Figure 1 : Overview of the SymphonyGen system. For centuries, symphonic composition has served as the primary vehicle for expansive musical storytelling through the collective power of the orchestra. Replicating this complexity, however, remains a formidable challenge for symbolic AI, as it demands the simultaneous management of high-level form and multi-track textures. Specifically, models are required to capture dependencies across dozens of voices while maintaining a sophisticated, evolving harmonic landscape over long temporal horizons. Toward this goal, considerable progress has been achieved through the adoption of Transformer architectures [ 14 , 5 ] and the development of symbolic musical representations [ 6 , 4 ] . Frameworks such as SymphonyNet [ 8 ] have employed permutation-invariant modeling for symphonic scores, while more recent models like NotaGen [ 16 ] leverage large language model (LLM) training paradigms to generate classical scores with impressive fidelity. However, while these models effectively capture characteristic patterns of target styles, they often treat generation as a monolithic sequence modeling problem that overlooks intermediate structural decisions, such as the use of a piano-reduction sketch. This neglect of the multi-stage workflows inherent to professional production leaves a critical gap in controllable symphonic creation, which we characterize as a “complexity-control imbalance”. On one hand, many approaches for orchestral composition or arrangement [ 24 , 19 , 7 , 23 , 10 , 22 ] flatten multi-track scores into unidimensional token streams, leading to scaling bottlenecks that hinder progress in large-scale orchestral generation. On the other, prevailing control mechanisms [ 15 , 13 , 21 , 17 , 1 ] remain insufficient for decomposing the task into workflows that support deep human-AI collaboration. Furthermore, models trained on broad MIDI datasets frequently produce spurious dissonance, failing to achieve the acoustic clarity and aesthetic value required for cinematic-grade orchestral composition. To bridge these gaps, we present SymphonyGen , a hierarchical framework controlled by a multi-voice harmony skeleton for contemporary cinematic orchestration. To the best of our knowledge, this is the first work that specifically targets symbolic cinematic orchestration rather than general orchestration. Our central contributions are: • 3D Hierarchical Architecture: A cascading encoder-decoder system that processes the Bar, Track, and Event axes separately, reducing computational overhead while improving multi-track scalability. • Multi-Voice Harmony Skeleton: A novel “short-score” condition that provides a granular music outline, steering the composition through beat-quantized harmonic and melodic contours. • Audio-Perceptual RL: A reinforcement pipeline utilizing a cross-modal reward. By aligning symbolic output with acoustic perception, this approach mitigates the qualitative limitations of MIDI datasets. • Dissonance-Averse Sampling: A logit-adjustment algorithm that references the harmony skeleton to suppress unintended clashes during inference, especially those between the non-harmonic tones. 2 Related Work 2.1 Symbolic Orchestral Generation Generative models specifically targeting orchestral music are less common than those for general multi-track tasks. Early efforts like MuseGAN [ 2 ] utilized GANs, but the field has largely shifted toward Transformers to capture long-range dependencies. Pop Music Transformer [ 6 ] and Compound Word Transformer [ 4 ] introduced fundamental tokenization schemes that remain influential. SymphonyNet [ 8 ] pioneered the task of symphonic generation and introduced a large-scale symphony dataset. More recently, NotaGen [ 16 ] leveraged CLaMP-DPO to generate high-fidelity classical scores. While successful, these models operate as single-stage generators that limit human collaboration. 2.2 Music Arrangement and Orchestration Automatic orchestration is the process of expanding or transforming musical material into an ensemble version. Early data-driven approaches like AccoMontage [ 24 , 20 ] and MuseMorphose [ 19 ] focused on accompaniment and style transfer, with the latter introducing bar-level control over polyphonic density. METEOR [ 7 ] extended these concepts to the orchestral domain while utilizing a melody-aware Transformer-VAE. Subsequent advancements include style prior modeling [ 23 ] and sequence-to-sequence frameworks that utilize multiple conditional streams [ 10 ] . Despite these developments, arrangement models are often built on VAE latent spaces to mitigate the scarcity of paired data [ 22 ] with 1D Transformer decoders. While effective for short-term transfer, they often lack the structural awareness for long-form symphonic movements. The Seq2Seq approach in [ 10 ] is closer to our work in its use of explicit conditions to guide arrangement. However, its conditions are mainly analyzed from the input music, while our condition comes from diverse sources ( Figure 1 ). 2.3 Controllable and Structured Music Generation Fine-grained control is essential for professional AI music. FIGARO [ 15 ] utilized bar-level interpretable features for controllable generation, while Theme Transformer [ 13 ] focused on thematic conditioning. To address phrase structure, researchers have proposed structure-related attention mechanisms [ 21 ] and cascaded diffusion models for whole-song generation [ 17 ] . Additionally, Text2midi [ 1 ] explored generating symbolic music from text captions. While previous methods offer diverse conditions for steering music generation, they still lack the detailed multi-voice grounding that could greatly enhance control for complex symphonic orchestration. In contrast, our approach more effectively bridges compositional intent with orchestral realization by outlining the harmony skeleton. 3 Methodology 3.1 3D Hierarchical Architecture Figure 2 : Model architecture. The pretrained model is supervised by a total loss \mathcal{L}=0.05\mathcal{L}_{\text{meta}}+0.5\mathcal{L}_{\text{harm}}+\mathcal{L}_{\text{music}} . Architecture Computational Complexity Inference VRAM 3D [B,T,E] O(BTE^{2}+BT^{2}+B^{2}) O(B+T+E) 2D [BT,E] O(BTE^{2}+B^{2}T^{2}) O(BT+E) 2D [B,TE] O(BT^{2}E^{2}+B^{2}) O(B+TE) 1D [BTE] O(B^{2}T^{2}E^{2}) O(BTE) Table 1 : Scalability comparison across architectures. B , T and E denote the number of bars, tracks and events. The backbone of our model is a series of cascading Transformer encoders and decoders to process one structural level at a time. This hierarchical approach of decomposing the sequence into Bar, Track, and Event levels is designed to manage the high density of symphonic scores. (We define events as the sequence of tokenized musical notes or note attributes within a specific bar-track intersection.) While 2D models (e.g., NotaGen) divide tokens into patches, our 3D architecture groups notes by bars and tracks. As shown in Table 1 , the 3D architecture scales quadratically with respect to each axis independently, and substantially reduces inference VRAM peak usage. This decomposition is particularly effective when coupled with a music outline provider such as our harmony skeleton; by delegating global planning to the skeleton, the model circumvents the exhaustive computation typically required to maintain coherence across