ALICE: A Multifaceted Evaluation Framework of Large Audio-Language Models’ In-Context Learning Ability
Yen-Ting, Jay Chiehen, Wei-Tang, Toshiki, Shang-Tse, Yun-Nung, Chun-Yi, Shao-Yuan · 2026
Read ALICE: A Multifaceted Evaluation Framework of Large Audio-Language Models’ In-Context Learning Ability by Yen-Ting, Jay Chiehen, Wei-Tang, Toshiki, Shang-Tse, Yun-Nung, Chun-Yi, Shao-Yuan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20433.txt.
Introduction
Piao and co-authors probe in-context learning in large audio-language models. Such models follow instructions poorly at times, but example use was untested. Their pipeline removes textual guidance in three stages, step by step. Six models faced four audio tasks under two output constraint types. Demonstrations reliably improved format compliance across the models. They did not improve core task performance, though. Label shuffling and attention tests explain why this happens. Models use examples to learn output format, not input-output links.
Catalog record
| CREATOR | Yen-Ting, Jay Chiehen, Wei-Tang, Toshiki, Shang-Tse, Yun-Nung, Chun-Yi, Shao-Yuan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23752 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20433
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ALICE: A Multifaceted Evaluation Framework of Large Audio-Language Models’ In-Context Learning Ability Yen-Ting, Jay Chiehen, Wei-Tang, Toshiki, Shang-Tse, Yun-Nung, Chun-Yi, Shao-Yuan Abstract While Large Audio-Language Models (LALMs) have been shown to exhibit degraded instruction-following capabilities, their ability to infer task patterns from in-context examples under audio conditioning remains unstudied. To address this gap, we present ALICE, a three-stage framework that progressively reduces textual guidance to systematically evaluate LALMs’ in-context learning ability under audio conditioning. Evaluating six LALMs across four audio understanding tasks under two output constraint categories, we uncover a consistent asymmetry across all stages and LALMs: in-context demonstrations reliably improve format compliance but fail to improve, and often degrade, the core task performance. This suggests that LALMs can glean surface-level formatting patterns from demonstrations but may struggle to leverage cross-modal semantic grounding to reliably infer task objectives from audio-conditioned examples, highlighting potential limitations in current cross-modal integration. Piao Liao Chien Ogimoto Chen Chen Lee Lo keywords large audio-language model, in-context learning, evaluation framework † † address: National Taiwan University, Taiwan † † email: {r14922010, r13922210, r13922197, r14922163, stchen}@csie.ntu.edu.tw, [email protected], {cylee, sylo}@csie.ntu.edu.tw 1 Introduction Large Audio-Language Models (LALMs) [ 11 , 6 , 16 , 25 , 28 , 1 ] extend Large Language Models (LLMs) to the auditory domain, supporting applications such as semantic-level speech question answering, spoken dialogue generation, paralinguistic analysis (e.g., emotion or speaker traits), and general non-speech audio event understanding [ 30 , 29 , 32 , 24 ] . Despite this progress, recent studies [ 17 , 10 , 14 ] reveal that LALMs often exhibit degraded instruction-following ability compared to their text-only counterparts, particularly under explicit output constraints [ 17 ] . This gap is hypothesized to stem from catastrophic forgetting during speech-text integration, suggesting that multimodal adaptation could affect not only modality alignment but also higher-level task inference mechanisms inherited from text pretraining. In the text-only domain, in-context learning (ICL) enables LLMs to infer latent task structures and output formats through few-shot demonstrations at inference time [ 2 , 18 , 8 , 4 ] . Crucially, ICL and instruction-following share a common underlying mechanism: both require the model to infer task objectives and output format constraints from contextual signals, whether those signals are explicit natural language instructions or implicit demonstrations [ 20 , 27 ] . Therefore, evaluating LALMs’ ICL behaviors provides a principled framework for probing whether these higher-level task recognition mechanisms generalize to the audio modality after speech–text integration. However, despite extensive studies of ICL in text-only LLMs [ 2 , 18 , 8 , 4 ] , its behavior in audio-conditioned multimodal settings remains underexplored. Prior work has explored ICL for specific tasks such as automatic speech recognition [ 22 ] or auditory knowledge editing [ 31 ] . To the best of our knowledge, there is no systematic study that characterizes ICL behavior in the audio-conditioned setting of LALMs. Rather than comparing LALMs with their text-only counterparts, our goal is to disentangle the joint influence of acoustic inputs and in-context demonstrations on task inference. To bridge this gap, we introduce ALICE ( A udio- L anguage I n- C ontext learning E valuation), a systematic three-stage evaluation framework for analyzing LALMs’ ICL ability. As illustrated in Figure 1 , under a fixed audio-conditioning setting, we design three conditions that progressively reduce textual guidance in the in-context demonstrations, which are all task-domain consistent, while keeping the audio inputs and demonstration outputs unchanged. Such controlled setup isolates the contribution of textual cues and assesses whether LALMs can infer task objectives and generate correctly formatted responses from audio-conditioned examples. Across six LALMs, we observe a strongly asymmetric ICL effect consistently: in-context demonstrations only improve format compliance but not core speech task performance , even when examples with reasoning traces are provided. Notably, the evaluated LALMs with stronger instruction-following ability undergo more degradation in format compliance when explicit constraint instructions are removed, indicating that good instruction-following capability does not imply robust ICL deduction ability from demonstrations alone. Overall, our key contributions are threefold: (i) we conduct the first systematic analysis of LALMs’ ICL deduction ability under audio conditioning; (ii) we provide empirical evidence that instruction-following ability does not guarantee ICL deduction ability; (iii) we show that providing examples with reasoning traces fails to improve task performance, suggesting that LALMs’ ICL limitations possibly stem from integrating auditory information rather than a deficiency in text-based reasoning. The inference code and related resources are available at: https://github.com/yenting-biao/ALICE . 2 ALICE: ICL Evaluation Framework To investigate whether LALMs exhibit ICL behavior under audio conditioning, we adopt a subset of Speech-IFEval [ 17 ] as our evaluation testbed, a benchmark of audio understanding tasks augmented with verifiable output constraints [ 33 , 17 ] that enables systematic assessment of format compliance and speech-processing ability under audio-conditioned prompts. 2.1 Audio Understanding Tasks and Format Constraints We adopt the audio understanding tasks from Speech-IFEval, including A utomatic S peech R ecognition (ASR), S peech E motion R ecognition (SER), G ender R ecognition (GR), and M assive M ulti-Task A udio U nderstanding (MMAU) [ 23 ] . We consider two constraint categories: (1) Closed-Ended Questions (CEQ) , which enforce strict deterministic formats (e.g., all-uppercase output, JSON formatting, or a required prefix/suffix); and (2) Chain-of-Thought (CoT) , which requires the model to produce explicit intermediate reasoning steps alongside the final answer. Refer to Figure 1 for an example. Figure 1: ALICE’s three evaluation stages: Textual guidance progressively decreases from Stage 1 (Explicit Constraint) to Stage 3 (Audio Only), with removed elements grayed out. 2.2 Construction of ICL Examples Instruction rephrasing. For each task-constraint combination, we use GPT-5 [ 19 ] to rephrase task instructions and format constraints respectively while preserving semantics, diversifying linguistic expressions across demonstrations and reducing lexical and structural bias. Audio and answer sourcing. For ASR, SER, and GR, we sample audio from CREMA-D [ 3 ] , a corpus disjoint from Speech-IFEval’s data containing diverse emotional speech with labelled speaker information. For MMAU, we sample audio clips from the MMAU dataset, ensuring that none overlap with Speech-IFEval. To maintain annotation quality, we only include MMAU items with either provided ground-truth answers or confidently human-annotated labels. Table 1 summarizes the audio duration statistics for our evaluation dataset (988 samples) and demonstration dataset (912 samples). Format label generation. CEQ demonstration labels are generated via rule-based transformations on the answers (e.g., uppercasing text), while CoT reasoning traces are generated by Gemini 2.5 Pro [ 7 ] and verified by a human expert. The resulting demonstration pool consists of audio-instruction-label triples that are linguistically diverse and disjoint from the evaluation data. They are label-balanced to avoid exposing the models to skewed label distributions during ICL. 2.3 Three-Stage Evaluation Framework The core evaluation methodology of ALICE is illustrated in Figure 1 . We propose a three-stage framework that progressively reduces the amount of explicit textual instruction provided, thereby probing the degree to which LALMs can learn from audio-conditioned examples when textual guidance is systematically withheld. In Stage 1 (Explicit Constraint) , each demonstration example and test sample contains both a task description and explicit format constraint instruction alongside the audio input. Demonstrations are drawn from the same task-constraint category as the