UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models

Qundong Shi, Jie Zhou, Biyuan Lin, Junbo Cui, Guoyang Zeng, Yixuan Zhou, Ziyang Wang, Xin Liu, Zhen Luo, Yudong Wang, Zhiyuan Liu · 2026

Read UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models by Qundong Shi, Jie Zhou, Biyuan Lin, Junbo Cui, Guoyang Zeng, Yixuan Zhou, Ziyang Wang, Xin Liu, Zhen Luo, Yudong Wang, Zhiyuan Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01373.txt.

Introduction

Eleven authors led by Qundong Shi present UltraEval-Audio, a single framework for testing audio foundation models. Since models that hear and speak arrived, progress has outrun measurement. Test sets and code lie scattered, audio codecs lack a shared scoring method, and speech benchmarks lean heavily on English, leaving Chinese skills unjudged.

The framework covers both understanding and generation, supports ten languages and fourteen task families, and plugs in twenty-four models and thirty-six benchmarks with a one-command run and live public leaderboards. Codecs are scored on three axes: meaning accuracy, voice likeness, and sound quality. Two new Chinese benchmarks test knowledge and fluency in speech form. The team hopes a transparent, shared platform will make audio model claims comparable across labs and companies, and speed work on the parts of the field that lag, especially generation.

Catalog record

CREATORQundong Shi, Jie Zhou, Biyuan Lin, Junbo Cui, Guoyang Zeng, Yixuan Zhou, Ziyang Wang, Xin Liu, Zhen Luo, Yudong Wang, Zhiyuan Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-01373

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models Qundong Shi, Jie Zhou, Biyuan Lin, Junbo Cui, Guoyang Zeng, Yixuan Zhou, Ziyang Wang, Xin Liu, Zhen Luo, Yudong Wang, Zhiyuan Liu Abstract The development of audio foundation models has accelerated rapidly since the emergence of GPT-4o. However, the lack of comprehensive evaluation has become a critical bottleneck for further progress in the field, particularly in audio generation. Current audio evaluation faces three major challenges: (1) audio evaluation lacks a unified framework, with datasets and code scattered across various sources, hindering fair and efficient cross-model comparison; (2) audio codecs, as a key component of audio foundation models, lack a widely accepted and holistic evaluation methodology; (3) existing speech benchmarks are heavily reliant on English, making it challenging to objectively assess models’ performance on Chinese. To address the first issue, we introduce UltraEval-Audio , a unified evaluation framework for audio foundation models, specifically designed for both audio understanding and generation tasks. UltraEval-Audio features a modular architecture, supporting 10 languages and 14 core task categories, while seamlessly integrating 24 mainstream models and 36 authoritative benchmarks. To enhance research efficiency, the framework provides a one-command evaluation feature, accompanied by real-time public leaderboards. For the second challenge, UltraEval-Audio adopts a novel comprehensive evaluation scheme for audio codecs, evaluating performance across three key dimensions: semantic accuracy, timbre fidelity, and acoustic quality. To address the third issue, we propose two new Chinese benchmarks, SpeechCMMLU and SpeechHSK , designed to assess Chinese knowledge proficiency and language fluency. We wish that UltraEval-Audio will provide both academia and industry with a transparent, efficient, and fair platform for comparison of audio models. Our code, benchmarks, and leaderboards are available at https://github.com/OpenBMB/UltraEval-Audio. 1 Introduction Following the groundbreaking success of large language models (LLMs), the rapid development of multimodal large models has ensued. Notably, the release of OpenAI’s GPT-4o ( 41 ) marked the advent of the native audio interaction era, accelerating the explosive development of audio foundation models (AFMs). Subsequently, a series of models with complex understanding and generation capabilities emerged, including Qwen2.5-Omni ( 64 ) , Moshi ( 15 ) , GLM-4-Voice ( 70 ) , Step-Audio ( 29 ) , Kimi-Audio ( 16 ) , and MiniCPM-o 2.6 ( 67 ) , significantly expanding the boundaries of human-computer interaction. With the rapid iteration of model capabilities, the challenge of objectively and systematically evaluating these models has become a focal point of academic interest. However, current audio evaluation lacks a unified framework, with datasets and code scattered across various sources, greatly hindering fair and efficient comparisons between models. In addition to the issue of fragmented evaluation frameworks, the existing evaluation systems also exhibit significant limitations in terms of evaluation depth and language coverage. Traditional evaluation tools are often designed for specific tasks, such as automatic speech recognition (ASR) or automatic speech translation (AST), making it difficult to adapt them to audio foundation models (AFMs) with general-purpose interactive capabilities. This is particularly true in areas like prompt management and inference parameter tuning, both of which are critical for the comprehensive assessment of AFMs. As model capabilities rapidly evolve, user-centered speech benchmarks are emerging in addition to traditional ASR and AST tasks. However, these benchmarks still face the following core challenges: (1) Audio codecs, which serve as the backbone of AFMs, lack systematic performance metrics. A codec consists of an audio tokenizer (which converts audio into discrete tokens) and a vocoder (which reconstructs audio from generated tokens). The design of audio codecs directly affects the fidelity and efficiency of the audio representation, which significantly impacts the overall performance of AFMs ( 68 ) . Existing methods are broad and provide limited insight into specific performance dimensions. (2) Current benchmarks are heavily reliant on English, making it challenging to objectively assess models’ performance on Chinese. Mainstream benchmarks, such as SpeechTriviaQA ( 15 ) , SpeechWebQuestions ( 39 ) , and SpeechAlpacaEval ( 23 ) , are primarily English-centric, leading to inadequate measurement of models’ knowledge and language proficiency in the Chinese context. To address these challenges, we propose UltraEval-Audio , a unified evaluation framework for audio foundation models. By decoupling data loading, prompt management, inference parameter control, and diverse post-processing and aggregation methods, this framework provides researchers with a unified and flexible evaluation environment. Users can quickly initiate the evaluation process using a simple “one-click” automated script. This decoupled framework design not only enhances the reproducibility of experiments but also facilitates rapid adaptation and agile extension for researchers. Furthermore, UltraEval-Audio innovatively introduces an audio codec evaluation scheme and several Chinese-language evaluation benchmarks, filling the gaps from both model components and evaluation benchmarks. Through this full-stack integration, UltraEval-Audio aims to standardize and enhance the transparency the entire evaluation process. By providing real-time public audio leaderboards, it strives to advance the field of audio foundation models towards greater interpretability and fairness. Our contributions are summarized as follows: • The first unified audio evaluation framework : UltraEval-Audio supports a wide range of input-output modalities, including “Text → Audio”, “Text + Audio → Text”, “Audio → Text”, and “Text + Audio → Audio”. The framework supports 10 languages, 14 core task categories and deeply integrates 24 mainstream models and 36 authoritative benchmarks, covering three key areas: speech, environmental sound, and music. With its user-friendly design, the framework offers a “one-click” evaluation feature and publicly available leaderboards for transparent comparisons. • A multi-dimensional evaluation scheme for audio codecs : We have established a systematic evaluation scheme that covers semantic accuracy, timbre fidelity, and acoustic quality, addressing the lack of widely accepted and systematic multi-dimensional performance metrics for audio codecs. • Two new Chinese evaluation benchmarks : We propose SpeechCMMLU and SpeechHSK , which are designed to systematically measure the knowledge proficiency and language fluency of AFMs in the Chinese context. 2 Related Work In this section, we introduce the latest developments in audio foundation models, audio evaluation frameworks, evaluation benchmarks, and evaluation of audio codecs. The Advancements of Audio Foundation Models. Emerging audio foundation models typically adopt an Audio Codec+LLM architecture, which generally comprises three core components: (1) an audio tokenizer , which converts raw audio signals into discrete tokens while preserving both semantic and acoustic information; (2) an LLM backbone , responsible for contextual modeling and autoregressive token prediction; and (3) a vocoder , which synthesizes natural speech waveforms from the generated audio tokens. Based on whether they incorporate a vocoder, audio foundation models can be classified into two categories: (1) audio understanding foundation models, which accept both audio and text as input and produce only text as output (e.g., Qwen-Audio ( 10 ; 9 ) , Gemini-1.5 ( 57 ) ). (2) audio generation foundation models, which accept both audio and text as input and generate both speech and text as output (e.g., GPT-4o-Realtime ( 41 ) , Moshi ( 15 ) , MiniCPM-o 2.6 ( 67 ) , Qwen2.5-Omni ( 64 ) , and Kimi-Audio ( 16 ) ). Meanwhile, audio codecs are also rapidly evolving. SoundStream ( 69 ) is the first universal audio codec capable of handling diverse audio types. EnCodec ( 14 ) , DAC (descript-audio-codec) ( 33 ) , HiFi-Codec ( 65 ) , X-codec ( 68 ) , BigCodec

Read the complete text