Data-Balanced Curriculum Learning for Audio Question Answering
Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier · 2025
Read Data-Balanced Curriculum Learning for Audio Question Answering by Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06815.txt.
Introduction
Gijs Wijngaard, Elia Formisano, Michele Esposito, and Michel Dumontier present a 2025 method for audio question answering that joins curriculum learning with data balancing. Such tasks ask a model to hear a sound clip and reason about it to answer a question, and models trip on skewed data and shaky training.
Language models grade how hard each question is, and training then runs from easy to hard. A filter cuts classes that show up too much. Guided decoding keeps the model's replies in the set of valid choices. On a named benchmark, the gains from data care were large.
Catalog record
| CREATOR | Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06815
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Data-Balanced Curriculum Learning for Audio Question Answering Gijs Wijngaard, Elia Formisano, Michele Esposito, Michel Dumontier Abstract Audio question answering (AQA) requires models to understand acoustic content and perform complex reasoning. Current models struggle with dataset imbalances and unstable training dynamics. This work combines curriculum learning with statistical data balancing to address these challenges. The method labels question difficulty using language models, then trains progressively from easy to hard examples. Statistical filtering removes overrepresented audio categories, and guided decoding constrains outputs to valid multiple-choice formats. Experiments on the DCASE 2025 training set and five additional public datasets show that data curation improves accuracy by 11.7% over baseline models, achieving 64.2% on the DCASE 2025 benchmark. I Introduction Audio Question Answering (AQA) represents a fundamental challenge at the intersection of acoustic understanding and natural language processing. Unlike simple sound classification, AQA requires models to comprehend complex acoustic scenes, identify temporal relationships between sounds, and generate coherent answers to diverse questions [ 1 , 2 , 3 ] . While earlier AQA models focused on binary tasks or limited label sets [ 1 , 4 ] , later models contain answer generation capabilities in natural language beyond strict answer sets. Some tasks require text generation metrics to evaluate the output of the model, while other tasks require selecting the correct answer from provided options, which makes answers verifiable against actual answers. Audio-language modeling faces limited diversity and quality of training data [ 5 ] . Many models train on overlapping datasets, and this redundancy also appears in benchmarks in the form of data contamination, resulting in homogeneous dataset landscapes. This limits the generalization and robustness of audio-language models. Analysis reveals that audio datasets suffer from severe class imbalances. Certain sound categories show dramatic over-representation. This leads to models that perform well on common sounds but fail on rare acoustic events. While traditional approaches to audio question answering have relied on supervised learning, recent advances have explored reinforcement learning techniques [ 6 ] . Reinforcement learning offers potential advantages for complex reasoning tasks but introduces training instabilities. This work investigates how curriculum learning and guided decoding can stabilize reinforcement learning for audio understanding. This work tests the following hypotheses for improving audio question answering: 1. Curriculum-guided reinforcement learning : The model trains on easy examples first to establish reliable reward signals. Difficult samples enter training progressively. This method stabilizes learning dynamics. 2. Statistical data balancing : Statistical thresholds identify and remove overrepresented categories, thereby balancing the training datasets. 3. Guided decoding : Regular expressions constrain generation to valid multiple-choice answers (A, B, C, D). 4. Hybrid training : The model trains with Supervised Fine-Tuning (SFT) to provide stable initialization. This is followed by Group Relative Policy Optimization (GRPO) through reward-based learning. Experiments on six datasets validate the approach. Data quality determines performance more than algorithmic complexity. The method achieves 64.2% accuracy on DCASE 2025 Task 5: Audio Question Answering. II Background II-A Audio-Language Models Large-scale audio-language models transform audio understanding. Early approaches combine audio encoder models with language model decoders [ 7 , 8 ] . Here, the language model’s performance is dependent on the quality of the features it receives from the audio encoder. Qwen2-Audio [ 9 ] and other recent models address this through unified architectures. The model processes a combination of speech, audio, and environmental sounds with a Whisper-large-v3 encoder and a Qwen-7B language model, and is used in this work as a foundational model. Examples of training strategies in the audio-language domain include: temporal progression from 30-second to 5-minute contexts that enable 3B models to outperform larger architectures [ 10 ] . Perception-before-understanding improves comprehension [ 11 ] . Multi-phase thinking, which allows for planning, captioning, reasoning, and summarization [ 12 ] . II-B Training Strategies for Audio Understanding Curriculum learning is the process of first showing a language model examples that can be classified as easy or simple, and gradually feeding the model more difficult examples [ 13 , 14 ] . It shows promise in various machine learning domains [ 15 ] . For audio-language models specifically, curriculum learning has enabled performance improvements through various progressive strategies: temporal progression from 30-second to 5-minute audio contexts allows 3B parameter models to outperform larger architectures [ 10 ] , and learning the model first to perceive and then understand facilitates better general audio comprehension [ 11 ] . The SARI framework [ 16 ] shows that curriculum-guided GRPO with structured reasoning can improve performance on complex audio reasoning tasks. Our method differentiates from the SARI framework by uniquely combining curriculum learning with statistical data balancing techniques to address dataset imbalances, and using guided decoding to ensure the model generates valid multiple-choice answers. Reinforcement learning for language models gains attention through RLHF (Reinforcement Learning from Human Feedback) [ 17 ] . Group Relative Policy Optimization (GRPO) [ 18 ] extends these ideas by eliminating the separate value model, which reduces memory requirements while maintaining training stability. The method optimizes for complex objectives defined as reward functions that are difficult to capture through supervised learning alone. Recent work extends GRPO to audio language models with promising results. Training with reinforcement learning shows significant improvements with fine-tuning Qwen2-Audio-7B-Instruct compared to supervised fine-tuning [ 6 ] . Further applications with Qwen2.5-Omni achieved state-of-the-art performance of 71.3% accuracy, suggesting gains primarily stem from enhanced text reasoning capabilities rather than improved audio processing [ 19 ] . Curriculum learning integrated with the GRPO framework enables training models progressively from easy to difficult examples, where structured four-part reasoning (planning, caption, reasoning, summary) combined with curriculum scheduling achieves superior performance at 67.08% accuracy [ 16 ] . A fundamentally different approach of decomposing audio into interpretable semantic components enhances model understanding of the audio beyond simple audio-to-text mapping [ 20 ] . Dataset-level curation strategies for audio-language models have recently gained attention as researchers recognize the importance of addressing class imbalances inherent in large-scale audio datasets. AudioSet, with its hierarchical ontology of 527 classes, exhibits significant imbalances where classes such as Music and Speech are dramatically overrepresented [ 21 ] . In addition, LLMs play an important role in quality assessment by evaluating LLM-generated captions against human-written standards. Captions rated below a threshold are filtered out to ensure that only high-quality data is retained [ 22 ] . This work identifies and removes the most prevalent categories based on their standard deviation from the mean distribution. This ensures that our reinforcement learning phase focuses on learning from diverse examples to improve generalization to rare acoustic events. II-C Audio Question Answering Datasets Challenge rules from the DCASE Challenge Task 5 restrict dataset selection to 20 approved sources. Many datasets target other tasks such as audio captioning and audio generation rather than question answering. This work selects only datasets with multiple-choice formats or relevant question-answer pairs. Five extra datasets are used to train the models, in addition to the training set of the challenge: • AVQA [ 23 ] : Contains 56,369 question-answer pairs from VGGSound (originally 57,335 before YouTube deletions). Questions span eight semantic types. The dataset includes audio and visual information. AVQA requires models to discover causal correlations between modalities. • ClothoAQA [ 24 ] : Contains 35,838