Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music
Sreyan Ghosh 1,2,∗ Arushi Goel 1,∗ Kaousheik Jayakumar 2 Lasha Koroshinadze 2 Nishit Anand 2 Zhifeng Kong 1 Siddharth Gururani 1 Sang-gil Lee 1 Jaehyeon Kim 1 Aya Aljafari 1 Chao-Han Huck Yang 1 Sungwon Kim 1 Ramani Duraiswami 2 Dinesh Manocha 2 Mohammad Shoeybi 1 Bryan Catanzaro 1 Ming-Yu Liu 1 Wei Ping 1 1 NVIDIA, USA 2 University of Maryland, USA ∗ Project-Leads. Ordering was decided with a coin toss. Code Model Project Page · 2026
Read Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music by Sreyan Ghosh 1,2,∗ Arushi Goel 1,∗ Kaousheik Jayakumar 2 Lasha Koroshinadze 2 Nishit Anand 2 Zhifeng Kong 1 Siddharth Gururani 1 Sang-gil Lee 1 Jaehyeon Kim 1 Aya Aljafari 1 Chao-Han Huck Yang 1 Sungwon Kim 1 Ramani Duraiswami 2 Dinesh Manocha 2 Mohammad Shoeybi 1 Bryan Catanzaro 1 Ming-Yu Liu 1 Wei Ping 1 1 NVIDIA, USA 2 University of Maryland, USA ∗ Project-Leads. Ordering was decided with a coin toss. Code Model Project Page (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10905.txt.
Introduction
Ghosh and seventeen colleagues present Audio Flamingo Next (AF-Next), the strongest audio-language model of the Audio Flamingo line for speech, sounds, and music. Against Audio Flamingo 3, it adds a stronger base model, data plans, long-audio support to 30 minutes, and Temporal Audio Chain-of-Thought, grounding reasoning steps to timestamps. The team scaled datasets past 1 million hours, training across pre-training, mid-training, and post-training. Across 20 benchmarks, AF-Next beats open models by wide margins and rivals far larger open and closed models, with three variants open.
Catalog record
| CREATOR | Sreyan Ghosh 1,2,∗ Arushi Goel 1,∗ Kaousheik Jayakumar 2 Lasha Koroshinadze 2 Nishit Anand 2 Zhifeng Kong 1 Siddharth Gururani 1 Sang-gil Lee 1 Jaehyeon Kim 1 Aya Aljafari 1 Chao-Han Huck Yang 1 Sungwon Kim 1 Ramani Duraiswami 2 Dinesh Manocha 2 Mohammad Shoeybi 1 Bryan Catanzaro 1 Ming-Yu Liu 1 Wei Ping 1 1 NVIDIA, USA 2 University of Maryland, USA ∗ Project-Leads. Ordering was decided with a coin toss. Code Model Project Page |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38028 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music Sreyan Ghosh 1,2,∗ Arushi Goel 1,∗ Kaousheik Jayakumar 2 Lasha Koroshinadze 2 Nishit Anand 2 Zhifeng Kong 1 Siddharth Gururani 1 Sang-gil Lee 1 Jaehyeon Kim 1 Aya Aljafari 1 Chao-Han Huck Yang 1 Sungwon Kim 1 Ramani Duraiswami 2 Dinesh Manocha 2 Mohammad Shoeybi 1 Bryan Catanzaro 1 Ming-Yu Liu 1 Wei Ping 1 1 NVIDIA, USA 2 University of Maryland, USA ∗ Project-Leads. Ordering was decided with a coin toss. Code Model Project Page Abstract We present Audio Flamingo Next (AF-Next), the next-generation and most capable large audio-language model in the Audio Flamingo series, designed to advance understanding and reasoning over speech, environmental sounds, and music. Compared to Audio Flamingo 3, AF-Next introduces: (i) a stronger foundational audio–language model that significantly improves accuracy across diverse audio understanding tasks; (ii) scalable strategies for constructing large-scale audio understanding and reasoning data beyond existing academic benchmarks; (iii) support for long and complex audio inputs up to 30 minutes; and (iv) Temporal Audio Chain-of-Thought, a new reasoning paradigm that explicitly grounds intermediate reasoning steps to timestamps in long audio, enabling fine-grained temporal alignment and improved interpretability. To enable these capabilities, we first conduct a systematic analysis of Audio Flamingo 3 to identify key gaps in audio understanding and reasoning. We then curate and scale new large-scale datasets totaling over 1 million hours to address these limitations and expand the existing AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat datasets. AF-Next is trained using a curriculum-based strategy spanning pre-training, mid-training, and post-training stages. Extensive experiments across 20 audio understanding and reasoning benchmarks, including challenging long-audio tasks, show that AF-Next outperforms similarly sized open models by large margins and remains highly competitive with, and sometimes surpasses, much larger open-weight and closed models. Beyond benchmark performance, AF-Next exhibits strong real-world utility and transfers well to unseen tasks, highlighting its robustness and generalization ability. In addition to all data, code, and methods, we open-source 3 variants of AF-Next, including AF-Next-Instruct, AF-Next-Think, and AF-Next-Captioner, meant for QA, advanced reasoning, and detailed captioning, respectively. 1. Introduction Audio, spanning speech, environmental sounds, and music, is central to how humans perceive and interact with the world. Robust audio understanding enables core capabilities such as conversation, situational awareness, and music listening, and underpins applications including automatic speech recognition (ASR), audio captioning, and music information retrieval (MIR). Historically, these problems were studied in isolation using small, task-specific models ( 41 ; 22 ) . More recently, Large Audio Language Models (LALMs) trained at scale have begun to unify these tasks, demonstrating strong transfer and broad coverage across domains ( 16 ; 57 ) . Yet, compared to vision-language models (VLMs), progress in scaling open LALMs has been noticeably slower, further limiting audio’s role in general-purpose multimodal systems and downstream efforts such as audio generation and world modeling ( 54 ; 27 ; 12 ) . Figure 1 : Performance comparison of AF-Next against prior SOTA LALMs across key audio understanding and reasoning benchmarks. A key barrier is that much of open LALM development has been either closed or tightly coupled to a small set of academic benchmarks. While benchmarks are valuable, they encode biases and incomplete coverage ( 31 ) , and audio benchmarks in particular are still emerging. As a result, benchmark-centric training can yield models that perform well on curated test sets but generalize poorly to long, noisy, and diverse real-world audio. Recent frontier systems illustrate both the opportunity and the gap: models in the Audio Flamingo ( 29 ) and Qwen ( 5 ) families introduced capabilities such as long-form audio understanding and multi-turn audio dialogue that are not yet comprehensively evaluated by standardized benchmarks, motivating a shift toward data and training recipes that better reflect applications in the real world. Main Contributions. We present Audio Flamingo Next (AF-Next), a fully open 1 1 1 By fully open , we mean that the model’s weights, training data, and code will be publicly released, with full transparency about the training methodology (unlike open-weights and closed models). Due to the licensing and scope of the training data used in the work, all releases will be under a research-only license. generalist Large Audio-Language Model that achieves state-of-the-art performance across 20+ audio understanding and reasoning benchmarks, while substantially improving robustness to long and complex real-world audio. AF-Next is a first step towards scaling fully open audio understanding beyond academic datasets and benchmarks by leveraging internet-scale audio data and post-training for reasoning. Concretely, we (i) scale training data beyond academic datasets by curating high-quality data from internet-scale sources, with a focus on long, diverse, and acoustically challenging audio that better reflect real deployment conditions; (ii) strengthen and broaden model capabilities across the Audio Flamingo task suite, including improvements in ASR and audio captioning, and the introduction of new capabilities such as multi-talker ASR, timestamped prediction, long-form audio captioning, and instruction following; and (iii) introduce Temporal Audio Chain-of-Thought, a reasoning paradigm that explicitly grounds intermediate reasoning steps to timestamps in long audio. To support these advances, AF-Next is trained with a four-stage curriculum that includes multiple rounds of supervised fine-tuning and GRPO-based reinforcement learning over carefully curated data mixtures. In summary, our main contributions are: 1. We introduce AF-Next, an open frontier generalist LALM that advances audio understanding and reasoning along multiple axes. AF-Next is, to our knowledge, the first fully open LALM to scale audio understanding to internet-scale data, and extensive experiments across 20+ benchmarks show that it outperforms similarly sized open models by large margins while remaining highly competitive with, and sometimes surpassing, much larger open-weight and closed models, particularly on long and complex real-world audio. 2. We develop a scalable training recipe for next-generation LALMs, spanning internet-scale data curation, targeted capability expansion, and temporally grounded reasoning for long audio. We open-source our training and inference code, and associated techniques to support future research in open LALMs. 3. We open-source three model checkpoints: AF-Next-Instruct , AF-Next-Think , and AF-Next-Captioner , designed for general question answering, advanced reasoning, and detailed captioning, respectively. 2. Methodology 2.1 Audio Flamingo Next Architecture In this section, we describe our proposed architecture for Audio Flamingo Next, also illustrated in Fig. 3 . Similar to Audio Flamingo 3 and Music Flamingo, AF-Next has four main components: i) an audio encoder with sliding window feature extraction, ii) an audio projector to project the audio embeddings into the language space of the LLM, iii) a text-only pre-trained LLM backbone, and iv) a streaming TTS. We provide details of each component below. AF-Whisper Audio Encoder. Following AF3 and Music Flamingo, we adopt the same Whisper-based AF-Whisper audio encoder, further pre-trained on a larger and more diverse corpus, including multilingual speech and multi-talker ASR data. We refer readers to 15 for the training details of AF-Whisper. Figure 2 : Examples of new data types introduced to scale AF-Next training. More examples are shown in Figures – , and details are provided in Section 2.2.1 . Feature Extraction. Given an audio input A , we first resample it to 16 kHz mono and convert the waveform into a 128-channel log mel-spectrogram using a 25 ms window and 10 ms hop size. The spectrogram is then passed