VoiceGiraffe : A Benchmark for Extreme Long-Context Audio-Language Understanding
Jashin Ye, Dongxiao Wang, Yixuan Ye, Sashuai Zhou, Weihuang Lin, Kunpeng Wang, Zeyu Yuan, Boyu Li, Haoxiang Shi, Jun Song, Bo Zheng · 2026
Read VoiceGiraffe : A Benchmark for Extreme Long-Context Audio-Language Understanding by Jashin Ye, Dongxiao Wang, Yixuan Ye, Sashuai Zhou, Weihuang Lin, Kunpeng Wang, Zeyu Yuan, Boyu Li, Haoxiang Shi, Jun Song, Bo Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27976.txt.
Introduction
Ye and colleagues benchmark audio-language models on hour-long speech, where current tests use only short clips. VoiceGiraffe poses fifteen hundred curated tasks spanning perception and multi-step reasoning across real scenarios and languages. Models struggled throughout, especially at tracking sparse events over long spans, while humans showed the opposite strength.
Catalog record
| CREATOR | Jashin Ye, Dongxiao Wang, Yixuan Ye, Sashuai Zhou, Weihuang Lin, Kunpeng Wang, Zeyu Yuan, Boyu Li, Haoxiang Shi, Jun Song, Bo Zheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45703 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27976
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VoiceGiraffe : A Benchmark for Extreme Long-Context Audio-Language Understanding Jashin Ye, Dongxiao Wang, Yixuan Ye, Sashuai Zhou, Weihuang Lin, Kunpeng Wang, Zeyu Yuan, Boyu Li, Haoxiang Shi, Jun Song, Bo Zheng Abstract While large audio language models (LALMs) have achieved remarkable progress in audio processing at the second- or minute-level scale, understanding hour-level audio remains a fundamental bottleneck. Existing benchmarks predominantly rely on short clips or artificially concatenated segments, failing to faithfully assess LALMs’ capacity for long-range information comprehension in real-world scenarios such as podcasts and lengthy speeches. To address this gap, we introduce VoiceGiraffe , a novel benchmark designed to rigorously evaluate LALMs across diverse real-world scenarios, modalities, and languages under long-context settings. It comprises 1,500 curated triplets structured into a dual-level taxonomy of single-hop perception and multi-hop reasoning. Single-hop questions assess temporal grounding, semantics, paralinguistics, and acoustic events, while multi-hop questions require models to aggregate evidence across multiple non-contiguous segments. We evaluate a broad suite of open-source and proprietary LALMs against human performance. Results underscore three fundamental findings. First, VoiceGiraffe remains highly challenging and far from saturation, with only one end-to-end (E2E) LALM surpassing the human reference, and no open-source model reaches the passing score even under cascaded caption aggregation. Second, we show that no single inference paradigm universally dominates. The E2E inference benefits models with native long-context audio understanding, cascaded caption aggregation stabilizes small models overwhelmed by hour-scale audio, and reasoning-enhanced cascading with external LLM helps weaker models but can bottleneck stronger proprietary systems. Third, we reveal long-range memory persistence as a key bottleneck. LALMs are better at answering questions that require connecting salient causal cues than those requiring sustained tracking of sparse events across long audio, whereas humans show the opposite pattern. This suggests that current LALMs can reason over prominent evidence once it is localized, but struggle to continuously memorize and retrieve event states over hour-scale contexts. These findings position VoiceGiraffe as a challenging and diagnostic testbed for long-form audio understanding, highlighting the need for LALMs with persistent memory and robust long-range aggregation. 1 Introduction Figure 1: Examples of VoiceGiraffe across five real-world domains: e-sports commentary, sports broadcast, TV drama, news, and podcast. Orange panels show single-hop questions (temporal localization, semantics, audio events, paralinguistics). Purple panels show multi-hop questions (causal reasoning and event tracking) requiring evidence aggregation across long spans. Large Audio Language Models (LALMs) Chu et al. 2023 ; Chu et al. 2024 ; Kong et al. 2024 ; Ghosh et al. 2025 ; Yu et al. 2025 have rapidly emerged as a unifying paradigm for speech, sound and music understanding, combining an audio encoder with a large language model to address diverse auditory tasks such as automatic speech recognition, audio captioning and audio question answering Sakshi et al. 2025 ; Yang et al. 2024 ; Ma et al. 2025 . As LALMs are increasingly deployed as audio assistants for meetings, podcasts, live broadcasts, and long interviews, reasoning over hour-scale audio inputs has become a crucial capability. However, current LALMs face fundamental challenges in long-context understanding and reasoning, stemming from the high audio token rate and insufficient pre-training on long-horizon audio understanding. The computational overhead imposed by dense tokens, coupled with limited exposure to longer contexts during training, systematically impedes effective long-horizon scaling. Consequently, models exhibit significant performance degradation when applied to long-form sequences. A critical reason the community has struggled to tackle this challenge is that evaluation protocols lag far behind both model capabilities and real-world requirements. As summarized in Table 1(a) , existing short-form benchmarks such as MMAU Sakshi et al. 2025 , MMAR Ma et al. 2025 , AudioBench Wang et al. 2025 , and AIR-Bench Yang et al. 2024 provide robust and comprehensive evaluation of audio understanding and reasoning for clips of 10–30 seconds, yet severe limitations emerge at longer temporal scales. While recent efforts have extended evaluation to minute-level contexts, such as AudioMarathon He et al. 2025 , ChronosAudio Luo et al. 2026 , and LongSpeech Yang et al. 2026 , these benchmarks still fall short of real-world scenarios requiring hour-level fine-grained understanding, such as sports events, TV dramas, and podcasts. More importantly, several benchmarks construct long-context inputs by concatenating short audio clips rather than using native long-form recordings, making it difficult to capture the natural temporal continuity and sparse long-range dependencies present in real-world audio. Beyond the duration and authenticity gap, existing benchmarks tend to focus on single aspects and lack comprehensive integration of bilingual coverage, cross-domain diversity, and multi-hop reasoning. To address these gaps, we introduce VoiceGiraffe , the first extremely long-context audio question-answer (AQA) benchmark for evaluating long-context understanding capacities in LALMs. As illustrated in Figure 1 and Table 1 , VoiceGiraffe is built on three pillars. First, hour-scale context . VoiceGiraffe collects 123 long-form recordings that total approximately 113.1 hours, with an average duration of 55.2 minutes and 34% of recordings exceeding one hour. To our knowledge, it is the first open-domain AQA benchmark at this scale. Second, bilingual, open-domain, and full-modality coverage. The benchmark includes five domains across bilingual: sports, e-sports commentary, TV dramas, news, and podcasts. All recordings contain natural mixtures of speech, sound effects, and background music, challenging models on both verbal comprehension and cross-modal reasoning. Third, a two-tier task taxonomy from single-hop perception to multi-hop reasoning. Tier-1 questions assess single-hop perception across four categories: temporal localization, semantic content, audio event detection, and paralinguistic analysis. Tier-2 questions target multi-hop reasoning through causal event tracing and state-trajectory modeling, requiring evidence synthesis from 2 to 4 non-contiguous segments spanning tens of minutes. Based on VoiceGiraffe , we evaluate 9 open-source and 4 proprietary audio-capable models together with a human reference. Our analysis reveals four key findings. First, VoiceGiraffe presents substantial challenges to current LALMs. Among the models capable of hour-scale E2E inference, only Qwen3.5-Omni-Plus surpasses the human reference, while open-source systems remain below the passing threshold under cascaded caption aggregation without any enhancement. Second, the optimal inference paradigm is highly model-dependent. E2E inference can better preserve fine-grained perceptual cues for models with strong native long-context audio understanding ability, whereas cascaded caption aggregation provides a more stable way for models overwhelmed by long-form audio. Reasoning-enhanced cascading further improves open-source models by compensating for weak long-range aggregation, but can also bottleneck strong proprietary systems when the external reasoning model is less capable than the evaluated model itself. Third, long-term memory persistence remains a key bottleneck. Unlike humans, who are better at tracking sparse states over extended context, LALMs are stronger at reasoning over salient causal cues but struggle to memorize and retrieve states across distant clips. In summary, our contributions are listed as follows: • We present VoiceGiraffe , the first bilingual, hour-scale audio benchmark for LALMs, covering five real-world scenarios with naturally interleaved speech, sound and music. • We design a two-tier task taxonomy with 1,500 carefully designed AQA pairs that progress from perception to reasoning, supporting systematic evaluation of long-context understanding. • We conduct comprehensive analysis and comparison experiments across broad suite of LALMs together with a human reference, identifying long-context degradation, long-form memory failures and language bias as the key bottlenecks of current models. 2 Related Work 2.1 Large Audio-Language Models The landscape of Large Audio Language Models (LALMs) has expanded rapidly along