Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, Bryan Catanzaro, NVIDIA, USA, University of Maryland, College Park, USA · 2025
Read Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models by Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, Bryan Catanzaro, NVIDIA, USA, University of Maryland, College Park, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08128.txt.
Introduction
Arushi Goel and a large team present a 2025 arXiv paper on Audio Flamingo 3, a fully open model that reasons over speech, sound, and music. A single new encoder, AF-Whisper, learns shared representations across all three audio types. The system can think step by step on demand, hold multi-turn chats with several clips, and handle audio up to ten minutes long.
The team built several large training sets with fresh curation methods and trained the model in five stages. Trained only on open data, the model set new best scores on more than twenty audio benchmarks. It passed both open and closed rivals trained on far larger datasets.
Catalog record
| CREATOR | Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, Bryan Catanzaro, NVIDIA, USA, University of Maryland, College Park, USA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08128
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, Bryan Catanzaro, NVIDIA, USA, University of Maryland, College Park, USA Abstract We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities of speech, sound, and music; (ii) flexible, on-demand thinking, allowing the model to do chain-of-thought-type reasoning before answering; (iii) multi-turn, multi-audio chat; (iv) long audio understanding and reasoning (including speech) up to 10 minutes; and (v) voice-to-voice interaction. To enable these capabilities, we propose several large-scale training datasets curated using novel strategies, including AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat, and train AF3 with a novel five-stage curriculum-based training strategy. Trained on only open-source audio data, AF3 achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open-weight and closed-source models trained on much larger datasets. 1 Introduction Figure 1 : AF3 vs. prior SOTA LALMs (values normalized and WER=100-WER). AF3 outperforms most open-source/weights (e.g., Qwen2.5-Omni) and closed (e.g., Gemini 2.5 Pro) LALMs while being fully open. Audio—including speech, sounds, and music—is central to human perception and interaction. It enables us to understand our surroundings, engage in conversations, express emotions, interpret videos, and enjoy music. For AI systems to approach artificial general intelligence (AGI) ( morris2024position ) , they must similarly develop the ability to comprehend and reason over diverse audio signals. While Large Language Models (LLMs) excel at language-based reasoning, their audio comprehension remains limited — both in accessibility and capability hurst2024gpt ; touvron2023llama . Extending LLMs to process and reason over audio is essential for building truly context-aware, intelligent agents. Audio-Language Models (ALMs) extend the capabilities of LMs to the auditory domain. Early works such as CLAP elizalde2022clap align audio and text in a shared embedding space, enabling them with tasks like retrieval ( oncescu2021audio ) . More recently, the emergence of Large ALMs (LALMs)—decoder-only language models augmented with audio understanding chu2023qwenaudio ; chu2024qwenaudio2 ; team2023gemini —has unlocked powerful capabilities, including open-ended audio question-answering (AQA) that demands both reasoning and world knowledge ( sakshi2024mmau ) . These capabilities have further enabled tasks like audio analysis drossos2020clotho ; kim2019audiocaps , conversational assistants daniel2018toward , etc. Models Audio Understanding Voice Multi-turn Chat Long Audio ( > 30 secs) Open-Source Sound Music Speech In Out* Single A Multiple A Speech Sound Music Model Data Code LTU ✓ ✓ × × × × × × × × ✓ ✓ ✓ LTU-AS ✓ ✓ ✓ × × × × × × × ✓ ✓ ✓ GAMA ✓ ✓ × × × × × × × × ✓ ✓ ✓ SALMONN ✓ ✓ ✓ × × × × × × × ✓ ✓ ✓ MuLLaMa × ✓ × × × × × × × × ✓ ✓ ✓ Phi-4-mm ✓ ✓ ✓ × × × × ✓ ✓ ✓ ✓ × × Qwen-Audio ✓ ✓ ✓ ✓ ✓ ✓ × × × × ✓ × × Qwen2-Audio ✓ ✓ ✓ ✓ ✓ ✓ × × × × ✓ × × Qwen2.5-Omni ✓ ✓ ✓ ✓ ✓ ✓ × ✓ ✓ ✓ ✓ × × GPT-4o Audio ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ × × × Gemini 2.0 / 2.5 ✓ ✓ ✓ ✓ ✓ ✓ × ✓ ✓ ✓ × × × Audio Flamingo ✓ ✓ × × × ✓ × × × × ✓ ✓ ✓ Audio Flamingo 2 ✓ ✓ × × × × × × ✓ ✓ ✓ ✓ ✓ Audio Flamingo 3 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 1 : Comparison of various LALMs in terms of capabilities and openness. AF3 stands out as the most capable and open model to date, achieving SOTA results across benchmarks (A in Chat stands for Audio). * Voice-out is powered by our novel streaming TTS implementation, which is also applicable to other LALMs. However, existing models still fall short in key areas critical to AGI, such as expert-level reasoning morris2024position ; sakshi2024mmau , multi-turn and multi-audio dialogue goel2024audio , and long audio understanding kong2025audioflamingo2 . We identify two core limitations: (i) most LALMs are trained primarily on short audio for recognition tasks rather than ones that require deliberate reasoning; and (ii) in turn, they lack exposure to the skill sets required for complex tasks. Additionally, most LALMs that support all three modalities of speech, sound, and music are closed-source: while some have publicly released models weights chu2023qwenaudio ; chu2024qwenaudio2 ; abouelenin2025phi , they offer limited to no information about their data, code, or recipes (more details in Table 1 ). Main Contributions. To address these issues, we introduce Audio Flamingo 3 (AF3) , a fully open-source 1 1 1 By fully open , we mean that the model’s weights, training data, and code will be publicly released, with full transparency about the training methodology. Due to the licensing and scope of the training data used in the work, all releases will be under a research-only license. LALM with state-of-the-art performance in audio understanding and reasoning across 20+ benchmarks. In addition, AF3 brings several novel capabilities, including multi-turn, multi-audio chat, on-demand thinking, voice-to-voice interaction, and long-context audio reasoning (up to 10 minutes). We propose three core innovations to enable these capabilities: (i) Data : We focus on curating high-quality data at scale and propose (a) AudioSkills-XL : a large-scale dataset of 8M diverse AQA pairs, (b) LongAudio-XL : large-scale dataset of 1.25M diverse audio QA pairs for long audio reasoning; (c) AF-Chat : a multi-turn multi-audio chat dataset curated using a novel algorithm with 75k instances and (d) AF-Think : a dataset with 250k+ AQA pairs with short length prefixes to encourage CoT-type reasoning before arriving at the answer (ii) AF-Whisper : We train AF-Whisper, a unified audio encoder pretrained using a novel strategy on large-scale audio-caption pairs, capable of learning general-purpose representations across speech, sounds, and music; and (iii) Learning Curriculum : We train AF3 with a five-stage curriculum-based training strategy that progressively increases context length and task complexity. In summary, our main contributions are: • We introduce Audio Flamingo 3 (AF3) , the most open and capable foundational LALM to date. AF3 introduces key capabilities including: (i) long-context audio QA (extending beyond sounds as in ( kong2025audioflamingo2 ) and including speech), and (ii) flexible, on-demand thinking, enabling the model to generate concise, CoT-style reasoning steps when prompted. AF3 achieves state-of-the-art performance on 20+ audio understanding and reasoning benchmarks. • We also present AF3-Chat , a fine-tuned variant of AF3 designed for multi-turn, multi-audio chat and voice-to-voice interaction. • We propose novelties in data curation, audio encoder representation learning, and training strategies. Being fully open, we release our code, training recipes, and 4 new datasets to promote research in this space. 2 Related Work Audio Language Models. The rapid progress of LLMs has catalyzed the development of multimodal LLMs (MLLMs) capable of understanding and reasoning across diverse data