Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Training
Yanru Wu Jianning Wang Chongxin Gan Yang Li, [10pt] Shenzhen International Graduate School, Tsinghua University · 2026
Read Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Training by Yanru Wu Jianning Wang Chongxin Gan Yang Li, [10pt] Shenzhen International Graduate School, Tsinghua University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19101.txt.
Introduction
Yanru Wu and three colleagues tackle training audio language models on data sets, where clashing data pulls slow learning. They propose grouped sequential training, which sorts data sets into like groups and adds them step by step, joining joint steadiness with staged speed. Scores based on gradients gauge how data sets relate, with no pricey transfer tests. On fourteen AudioQA sets with speech, music, and nature sounds, the method learns 30 to 40 percent faster than joint training while matching or beating mix-all training.
Catalog record
| CREATOR | Yanru Wu Jianning Wang Chongxin Gan Yang Li, [10pt] Shenzhen International Graduate School, Tsinghua University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48362 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19101
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Training Yanru Wu Jianning Wang Chongxin Gan Yang Li, [10pt] Shenzhen International Graduate School, Tsinghua University Abstract Training general-purpose Audio Large Language Models (ALLMs) across diverse datasets is essential for holistic audio understanding, yet it faces significant challenges due to dataset heterogeneity, which often leads to conflicting gradients and slow convergence. Despite its impact, how to explicitly manage this heterogeneity during training remains underexplored, with current practices relying primarily on uniform mixture. In this work, we analyze multi-dataset AudioQA training from a convergence perspective and propose Grouped Sequential Training (GST). GST strategically organizes datasets into affinity-aware groups and introduces them via a progressive scheduling protocol, effectively balancing the stability of parallel training with the efficiency of sequential optimization. To ensure scalability, we develop gradient-based affinity metrics that capture inter-dataset relationships without the prohibitive cost of empirical transferability estimation. Extensive evaluations on 14 AudioQA datasets spanning speech, music, and environmental sounds demonstrate that GST achieves 30–40% faster convergence than standard parallel training while maintaining or even surpassing the performance of mix-all training. Our results provide both theoretical insights and a practical, model-agnostic framework for efficient large-scale ALLM optimization. 1 Introduction Large language models (LLMs) have recently been extended beyond text to incorporate audio perception, enabling progress in tasks such as classification, captioning, and question answering 24 ; 16 . Among these, AudioQA 13 has attracted increasing attention as a unified data format for training Audio LLMs (ALLMs), consolidating diverse audio tasks into a single framework and supporting broad application scenarios from assistive technology to human–computer interaction. While recent efforts have integrated pretrained audio encoders into LLMs to build general-purpose AudioQA models 4 ; 40 , scaling across datasets with disparate domains, annotation styles, and difficulty levels introduces significant yet often overlooked training inefficiencies. Multi-task learning 46 provides a natural theoretical lens for generalization across diverse datasets, where in practice each dataset is often treated as an instantiation of a task. Extensive explorations have been conducted in traditional machine learning 39 ; 30 , encompassing investigations of curriculum learning 38 , task sampling 36 , and various training paradigms 8 . However, these approaches typically rely on task-specific assumptions, architectural modifications, or auxiliary training mechanisms, limiting their applicability to general ALLMs. 1 1 1 See Sec. 2.2 for further discussion. Consequently, most existing ALLM training pipelines resort to naive strategies such as uniform dataset mixing, and fail to account for substantial data heterogeneity. Although such mixing ensures task coverage, it often induces suboptimal training dynamics characterized by redundant updates and slow convergence. These inefficiencies are further exacerbated in ALLMs by the inherent complexities of long audio sequences, cross-modal alignment, and the sheer scale of the models. Recent advances in taskonomy and transferability estimation 45 ; 42 have established that dataset-level relationships can be successfully quantified. Yet despite this progress, a systematic understanding of how dataset-level heterogeneity governs the training of ALLMs remains elusive. Current analyses are largely restricted to post-hoc evaluations, leaving a gap in understanding how these heterogeneous traits directly impact real-time training dynamics, and more importantly, how they can inform the design of training strategies. Building on this intuition, this work addresses a central question: How can these dataset properties be explicitly leveraged to schedule efficient multi-dataset training for ALLMs, thereby accelerating convergence without architectural overheads? To address this, we investigate multi-dataset training for ALLMs through the lens of convergence analysis. We begin by theoretically analyzing the convergence behaviors of different training regimes, focusing on how gradient variance and dataset heterogeneity affect training dynamics. Our analysis reveals the relative advantages of parallel versus sequential training under differing heterogeneity degrees, which then motivates our proposal of Grouped Sequential Training (GST). As a paradigm that interpolates between parallel and sequential optimization, GST organizes datasets into groups with reduced intra-group heterogeneity. By executing sequential updates at the group level, this strategy yields provably tighter convergence bounds in realistic multi-dataset settings. For practical implementation, we cluster datasets using metrics that capture optimization-level disparities, such as gradient-based distance and empirical transferability. To further scale the framework to large-scale ALLMs, we introduce an efficient progressive training procedure that approximates grouped sequential optimization while minimizing computational overhead and mitigating catastrophic forgetting. Importantly, our approach is model-agnostic and requires no architectural modifications. Extensive evaluations across 14 AudioQA datasets demonstrate that our method significantly accelerates convergence compared to naive multi-task baselines, while simultaneously maintaining or enhancing performance across diverse domains. Our contributions can be summarized as follows: • Theoretical Framework: We establish a convergence analysis for ALLMs that rigorously characterizes the impact of gradient variance and dataset heterogeneity. • Paradigm Shift: We propose Grouped Sequential Training (GST), which interpolates between parallel and sequential optimization and moves dataset affinity estimation from post-hoc analysis into training loop design. • Scalable Implementation: We introduce a progressive training procedure that approximates the optimal grouping strategy, enabling efficient large-scale training without architectural changes. • Benchmark & Insights: We consolidate 14 datasets into a unified AudioQA format, providing systematic insights and actionable strategies for heterogeneous multi-dataset training. Figure 1: Illustration of sequential (left-up), parallel (left-down), and grouped sequential training (right). 2 Related Works 2.1 Audio Large Language Model ALLMs have transformed audio-related tasks by demonstrating remarkable zero-shot performance and classification, understanding, and reasoning capabilities. Existing approaches typically adopt either an encoder-only or an encoder-decoder architecture 35 . The encoder-only paradigm employs a dual-tower framework where the text and audio encoders process paired text-audio data. Following the training objective of Contrastive Language-Image Pre-training (CLIP) 33 , such models are trained to learn a joint embedding space by drawing the embeddings of paired data closer while pushing apart those of unpaired data. In parallel, driven by the success of vision-language models, various encoder-decoder ALLMs have been proposed to leverage the large language model for audio comprehension. Prior works such as SALMONN 40 , Pengi 6 , Qwen-Audio 5 ; 4 , and the Audio Flamingo series 24 ; 11 ; 12 have achieved state-of-the-art performances across a wide range of audio tasks. Those methods integrate an audio encoder with a decoder-only language model in a cascade manner. Unlike encoder-only methods that solely rely on contrastive training objectives, encoder–decoder ALLMs are trained with generative objectives, such as masked reconstruction or next-token prediction. This paradigm enables the model to capture richer semantic audio information, facilitating more flexible reasoning and instruction-following abilities. 2.2 Training Strategies for Multi-Dataset Learning Training models on multiple datasets is a fundamental challenge across multi-task 46 , continual 20 , and federated learning 27 . In the context of LLMs, research has increasingly shifted from purely architectural innovations 15 ; 37 to data-centric optimization strategies. While substantial progress has been made in data selection and sampling weighting 43 ; 3 ; 26 to curate optimal training mixtures, the role of learning scheduling remains underexplored. Existing paradigms generally fall into two categories: parallel and sequential training (Fig. 1 ). Parallel training jointly optimizes all datasets via uniform mixing or weighted sampling 34 . While stable, it often suffers from gradient interference 44 , where conflicting optimization directions from diverse domains lead to suboptimal convergence. In contrast, sequential training introduces datasets chronologically 1 . While this can prioritize high-quality