Improving Audio Question Answering with Variational Inference
Haolin Chen · 2026
Read Improving Audio Question Answering with Variational Inference by Haolin Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-12700.txt.
Introduction
Haolin Chen works on audio question answering, where a model hears a sound clip and answers questions about it. He uses a method from statistics called variational inference. It treats the weights of a model as ranges, not fixed points. That way, the model can show how sure it is as it trains.
Chen applies a recent tool for this task, called IVON. He fine tunes a large model that works with both sound and text. A model that knows its own doubt gives answers users can trust more. Its stated trust in an answer is more likely to match how often it is right. The study tests if this gain holds for hard tasks that mix sound with words.
Catalog record
| CREATOR | Haolin Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-12700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Audio Question Answering with Variational Inference Haolin Chen Abstract Variational inference (VI) provides a principled framework for estimating posterior distributions over model parameters, enabling explicit modeling of weight uncertainty during optimization. By capturing this uncertainty, VI improves the reliability of predictions, yielding better calibrated outputs. In this work, we investigate the benefits of VI for challenging multimodal understanding and reasoning by applying the Improved Variational Online Newton (IVON), a recent VI optimizer, to fine-tuning a multimodal large language model on audio question answering tasks. Our results show that VI not only enhances predictive accuracy but also significantly improves calibration, reducing the model’s overconfidence. These advances further support risk-sensitive applications such as selective prediction, where reliable confidence estimates are crucial. † † address: Idiap Research Institute, Martigny, Switzerland École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland 1 Introduction Audio question answering (AQA) [ 1 , 2 , 3 , 4 ] is a challenging multimodal task that requires models to interpret complex acoustic scenes in response to natural language queries. Unlike audio classification or captioning, AQA requires the extraction of fine-grained acoustic cues, temporal dependencies, and contextual information from potentially long and intricate audio sequences, followed by the generation of precise, contextually grounded answers. Recent advances in large audio language models (LALMs) [ 5 , 6 , 7 , 8 ] have made substantial progress toward this goal by unifying audio representation learning and language reasoning within a single architecture. However, producing reliable predictions in multimodal understanding tasks such as AQA remains challenging. Models are often overconfident, poorly calibrated, and prone to errors they cannot recognize, particularly when fine-tuned with limited data. In such cases, predicted probabilities fail to reflect true confidence, undermining their reliability for risk-sensitive applications where robust uncertainty estimation is essential. While these challenges have been demonstrated in unimodal domains such as natural language processing [ 9 , 10 ] , we argue that they extend directly to multimodal scenarios. Variational inference (VI) [ 11 , 12 , 13 , 14 , 15 ] provides a principled framework for addressing these limitations by modeling the uncertainty of neural network parameters. Unlike traditional deep learning optimizers that produce point estimates, VI models full posterior distributions over parameters, explicitly capturing weight uncertainty during training. This leads to more accurate predictive uncertainty estimation, better calibration, and improves risk-sensitive applications such as selective prediction (SP), where the model can abstain from answering when uncertain. Recent advances in VI [ 15 ] demonstrate superior calibration and uncertainty quantification compared not only to conventional optimizers like Adam [ 16 , 17 ] , but also to uncertainty estimation methods such as Monte Carlo dropout [ 18 ] , stochastic weight averaging [ 19 ] , and deep ensemble [ 20 ] , while maintaining high efficiency and effectiveness for large-scale networks. In this work, we utilize the Improved Variational Online Newton (IVON), a recent VI optimizer that matches or surpasses Adam’s accuracy with comparable computational cost, to fine-tune a state-of-the-art large audio language model on challenging audio question answering tasks. Our experiments show that IVON not only achieves superior predictive accuracy compared to Adam, but also produces substantially better-calibrated and more reliable predictions. This improves the trustworthiness of multimodal understanding and reasoning systems in risk-sensitive applications like selective prediction [ 21 , 22 , 23 ] where the reliable confidence scores are required. 2 Audio Question Answering 2.1 Overview Audio question answering [ 1 , 2 , 3 , 4 ] is an emerging multimodal challenge at the intersection of acoustic understanding and natural language processing. The task requires a system to listen to an audio clip, interpret its content, and generate a precise, contextually relevant answer to a question posed in natural language. Building upon the foundation of automated audio captioning, AQA extends beyond simple description to enable more interactive understanding and reasoning of audio. Unlike conventional audio classification which focuses on identifying pre-defined sound events, AQA requires a higher level of semantic comprehension and complex reasoning. The complexity of the task is multi-layered: a model must not only process raw audio and identify acoustic information but also understand temporal relationships between sounds, infer implicit context, and provide a coherent response to multiple-choice or open-ended questions. This combination of auditory perception and linguistic reasoning across potentially long audio sequences makes AQA a highly challenging task and a valuable benchmark for evaluating advanced reasoning capabilities of audio language models. 2.2 Large Audio Language Models Recent advances in LALMs [ 5 , 6 , 7 , 8 ] are transforming audio processing by combining the reasoning capabilities of large language models (LLMs) with auditory understanding. Unlike traditional cascaded pipelines with separate modules for speech recognition and language processing, LALMs handle diverse audio inputs, including speech, music, and general sounds, within a single, unified architecture. A typical design uses a powerful audio encoder, often initialized from pre-trained models like Whisper [ 24 ] , to convert raw audio into feature embeddings. These are then aligned with an LLM backbone, enabling audio-grounded reasoning and multi-turn dialogue. For example, Qwen2-Audio [ 5 ] and Qwen2.5-Omni [ 7 ] employ a Whisper-based encoder with a pooling layer to shorten audio representations. Similarly, Audio Flamingo 3 [ 8 ] uses a unified AF-Whisper encoder trained across varied audio modalities. These techniques allow LALMs to move beyond simple transcription to tackle complex reasoning and problem-solving in audio intelligence. 2.3 Selective Prediction Question answering (QA) is traditionally formulated as a prediction task in which the model must always provide an answer from a fixed answer space \mathcal{A} , regardless of uncertainty. Formally, this is represented as a function f:\mathcal{X}\rightarrow\mathcal{A} , where each input x=(c,q)\in\mathcal{X} consists of the context c (in AQA, the audio) and a corresponding question q . This formulation compels the model to produce an answer for every input, even in cases where it is uncertain or likely to be incorrect, potentially leading to unreliable predictions. To mitigate this issue, selective prediction (SP) allows the model to abstain from answering when appropriate, effectively enabling it to respond with “I don’t know”. Following the notation of [ 23 ] , the selective QA task can be defined as a selective model h:\mathcal{X}\rightarrow\mathcal{A}\cup\{\emptyset\} , where the model can either output an answer from \mathcal{A} or choose to abstain (denoted by \emptyset ). A typical approach is to construct h from two components: an answer predictor f , and a confidence function g:\mathcal{X}\rightarrow[0,1] , which estimates the reliability of the prediction. The model answers only when its confidence exceeds a threshold \gamma ; otherwise, it abstains: h(x)=(f,g)(x)=\begin{cases}f(x)&\text{if }g(x)\geq\gamma,\\ \emptyset&\text{if }g(x)<\gamma.\end{cases} (1) The threshold \gamma governs the trade-off between coverage and reliability. A higher value leads to more conservative behavior, as the model answers only when highly confident. Lowering \gamma increases the number of answered questions but may reduce prediction accuracy. Ideally, g(x) should produce high values for correct predictions and low values for incorrect ones. The simplest confidence function is MaxProb, which utilizes the softmax probability of the model’s predicted class with the highest confidence. Given the output distribution p_{i} produced by a deterministic model, the selective function is defined as: g_{MaxProb}(x)=p_{k},\quad\text{where }k=\underset{i}{\operatorname{argmax}}\,p_{i}. (2) We use MaxProb in our experiments as it can