Audita : A New Dataset to Audit Humans vs. AI Skill at Audio QA
Tasnim Kabir, Dmytro Kurdydyk, Aadi Palnitkar, Liam Dorn, Ahmed Haj Ahmed, Jordan Lee Boyd-Graber · 2026
Read Audita : A New Dataset to Audit Humans vs. AI Skill at Audio QA by Tasnim Kabir, Dmytro Kurdydyk, Aadi Palnitkar, Liam Dorn, Ahmed Haj Ahmed, Jordan Lee Boyd-Graber (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21766.txt.
Introduction
Kabir and five colleagues introduce AUDITA, a benchmark that tests genuine audio reasoning rather than shortcut cues in audio question answering. It offers human-authored trivia questions grounded in real-world audio, with hard distractors and long-range temporal dependencies that cannot be answered from isolated text or sound alone. Humans average 32.13 percent accuracy, while state-of-the-art models average under 9 percent. Item Response Theory analysis then exposes systematic model and data weaknesses.
Catalog record
| CREATOR | Tasnim Kabir, Dmytro Kurdydyk, Aadi Palnitkar, Liam Dorn, Ahmed Haj Ahmed, Jordan Lee Boyd-Graber |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 107184 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21766
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audita : A New Dataset to Audit Humans vs. AI Skill at Audio QA Tasnim Kabir, Dmytro Kurdydyk, Aadi Palnitkar, Liam Dorn, Ahmed Haj Ahmed, Jordan Lee Boyd-Graber Abstract Existing audio question answering benchmarks largely emphasize sound event classification or caption-grounded queries, often enabling models to succeed through shortcut strategies, short-duration cues, lexical priors, dataset-specific biases, or even bypassing audio via metadata and captions rather than genuine reasoning Thus, we present Audita ( A udio U nderstanding from D iverse I nternet T rivia A uthors), a large-scale, real-world benchmark to rigorously evaluate audio reasoning beyond surface-level acoustic recognition. AUDITA 1 1 1 The codebase and data are available at https://github.com/Pinafore/audio_data , and https://huggingface.co/datasets/TasnimKabir12/audita-audio comprises carefully curated, human-authored trivia questions grounded in real-world audio, designed to stress robust auditory reasoning through challenging distractors and long-range temporal dependencies, using probing queries that cannot be answered from isolated text or sound cues alone. Human average accuracy of 32.13 32.13 % shows both the challenge of the task while demonstrating meaningful comprehension of the audio. In stark contrast, state-of-the-art audio question answering models perform poorly, with average accuracy below 8.86 8.86 %. Beyond raw accuracy, we apply Item Response Theory (IRT) to estimate latent proficiency, question difficulty, and expose systematic deficiencies of the models and data. 1 Introduction Question answering (QA) is a central paradigm for evaluating language understanding, with modern benchmarks such as Natural Questions 39 and large-scale multimodal tasks 84 driving rapid progress. Combined with advances in large language models 9 ; 55 , these systems now achieve near- or superhuman performance on many text-based QA tasks. However, this success does not extend uniformly to audio question answering ( Audio QA ), where models must reason over complex and often indirect auditory signals. Despite progress in speech recognition 5 , audio tagging 37 , and multimodal modeling 24 , current systems still struggle to reason about audio beyond surface-level recognition. Existing benchmarks fail to capture this gap. Datasets such as VGGSound 8 focus on closed-set classification, while captioning datasets like Clotho 14 emphasize description rather than reasoning. Many Audio QA datasets further rely on synthetic scenes or templated questions 16 ; 45 , enabling models to exploit language priors or shallow cues rather than true auditory grounding (Section 2 ). A key limitation is the absence of systematic human benchmarks, making it difficult to assess whether models are approaching human-level auditory understanding. Recent work has highlighted the need for more reliable evaluation frameworks such as Item Response Theory ( 42 , IRT ) , which jointly models question difficulty and participant ability. This work introduces Audita (Section 3 ), a large-scale benchmark of human-authored audio trivia questions sourced from real-world domains including quizzes, media, and cultural knowledge. Unlike prior datasets, Audita avoids synthetic audio and templated formats, focusing instead on crafted questions that require multi-cue auditory reasoning and world knowledge. Under controlled human evaluation (Section 4 ), expert participants have strong accuracy (best-per-category: 69.17% free-form, 86.67% MCQ), while state-of-the-art audio and multimodal models lag far behind, with accuracies under 8.86 %. This indicates a substantial gap in auditory reasoning capability. We further analyze this gap using IRT 25 , revealing a clear separation between human and model ability distributions. Our analysis also surfaces common issues in existing benchmarks (Section 5 ): ambiguity, underspecified answers, and shortcut-prone designs, which can inflate model performance without requiring true auditory grounding. Our contributions are: (1) Audita , a benchmark for challenging audio QA requiring multi-cue reasoning; (2) human baselines demonstrating a large and consistent human–model gap; and (3) IRT-based analysis providing fine-grained insights into difficulty, reasoning structure, and model failure modes. 2 Why Audio QA Requires Better Questions Recent advances in text-based question answering (QA) and large language model training have highlighted that dataset quality remains a primary bottleneck despite advances in model scale. Poorly designed questions introduce ambiguity 54 ; 48 , false presuppositions 83 , unreliable supervision 49 , and exploitable shortcuts 57 , resulting in inflated model performance and unstable evaluations. 48 ; 66 show that a substantial fraction of real-world questions remain inherently ambiguous or underspecified, often leading to apparent hallucinations that stem from dataset limitations rather than model failure. Consequently, contemporary QA research has increasingly emphasized careful question design, ambiguity-aware evaluation, and human calibration to ensure benchmarks truly test reasoning rather than dataset artifacts. Many of these failure modes identified in text QA also appear in Audio QA datasets. We analyze these issues through Item Response Theory ( 25 , IRT ) , a classical framework for modeling question difficulty and discrimination, which captures how effectively questions distinguish between high and low-ability models or annotators (Appendix Section A ). 60 and 40 have applied IRT to benchmark and leaderboard analysis, providing a principled mechanism for diagnosing dataset quality. This diagnostic perspective aligns with more recent efforts toward adaptive and fluid benchmarking that emphasize maintaining meaningful evaluation as models improve 26 . Appendix Table 8 and 9 summarize common issues across several popular Audio QA datasets, illustrating these problems with concrete examples and IRT-based evidence. Ambiguity, a central challenge extensively studied in 54 , occurs when questions admit multiple plausible answers or interpretations. For example, in Clotho- Audio QA 14 , “What animal is making the sound?” receives different yet valid responses such as bird or dog , as both are in the corresponding clip reflecting overlapping audio sources or vague wording. This ambiguity results in low discrimination scores (e.g., VGGSound QA, Appendix Table 8 , row 1) and low feasibility. False presuppositions, well-documented in text QA 83 , also appear in Audio QA datasets such as FSD50K 17 . These questions assume patterns not grounded in the audio, leading to inconsistent responses; correspondingly, IRT reveals low discrimination, indicating poor ability to differentiate annotator or model competence (Appendix Table 9 ). Weak grounding and shortcut learning are also prevalent, especially in caption-derived datasets like AudioCaps QA 36 and AudioSet QA 19 . Questions such as “Is someone laughing?” exhibit low discrimination, as they can often be answered without audio, undermining evaluation validity (Appendix Table 8 ). Underspecified answer keys (e.g., dog , puppy , hound ) in datasets like MUSIC-21 QA 10 and Clotho- Audio QA introduce ambiguity, which IRT captures as reduced discrimination and noisy item estimates. Finally, overlapping audio sources (e.g., VGGSound QA , AudioCaps QA) increase ambiguity, producing items with high difficulty but low discrimination— hard yet uninformative questions (Appendix Table 7 ). 3 Audio Questions for Triva Experts We introduce a benchmark of human-authored audio questions designed by knowledgeable experts to test auditory and multimodal reasoning skills. This approach aligns with what 63 call the “probing” paradigm and what 59 term the Manchester paradigm, where questions are crafted explicitly to evaluate human-level understanding. While probing-style questions are common in text-only QA datasets such as 33 and TriviaQA 32 , they remain underexplored in the audio domain. We scrape questions from publicly available online audio tests, reflecting a broader practice across various communities– such as educators, trivia enthusiasts, and researchers– that release question sets to facilitate human study and benchmarking. This openness aligns with the Manchester paradigm’s emphasis on carefully crafted questions designed to probe deep understanding. However,