VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models

Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, Irwin King · 2025

Read VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models by Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, Irwin King (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04962.txt.

Description

Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King introduce VoxEval in a 2025 arXiv paper. VoxEval is a benchmark that tests spoken language models on knowledge questions through speech alone. Older question-answering tests cannot judge models that take in and reply with speech.

The benchmark keeps both input and output in speech form. It varies audio conditions to test robustness, and it even poses math reasoning tasks in spoken form. Tests show current spoken models struggle with the benchmark. Scores shift with audio quality, and reasoning is a clear weak point. The authors offer the set to guide work on stronger spoken models.

Catalog record

CREATORWenqian Cui, Xiaoqi Jiao, Ziqiao Meng, Irwin King
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-04962

Browse the full library · About the Tower of Babel