AU-Harness: An Open-Source Toolkit for Holistic Evaluation of AU dio-LLMs

Sidharth Surapaneni, Hoang Nguyen, Jash Mehta, Aman Tiwari,, Oluwanifemi Bamgbose, Akshay Kalkunte, Sai Rajeswar, Sathwik Tejaswi Madhusudhan · 2025

Read AU-Harness: An Open-Source Toolkit for Holistic Evaluation of AU dio-LLMs by Sidharth Surapaneni, Hoang Nguyen, Jash Mehta, Aman Tiwari,, Oluwanifemi Bamgbose, Akshay Kalkunte, Sai Rajeswar, Sathwik Tejaswi Madhusudhan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08031.txt.

Description

Audio language models finally get a shared measuring stick in a 2025 toolkit paper by Hoang Nguyen and 11 co-authors. Large audio models are advancing quickly, but evaluation tools are slow, non-standard, and hard to compare across studies. Long multi-turn conversations, where context must carry across exchanges, are especially poorly supported by existing frameworks.

AU-Harness is open-source and built for holistic evaluation, aiming at faster pipelines and unified, scalable assessment that keeps pace with model growth. Fair comparison is not bureaucratic tidiness: without it, claims about listening, reasoning, and dialogue ability cannot be checked. A common harness lets researchers see which audio models genuinely improve and which merely test well on friendly setups.

Catalog record

CREATORSidharth Surapaneni, Hoang Nguyen, Jash Mehta, Aman Tiwari,, Oluwanifemi Bamgbose, Akshay Kalkunte, Sai Rajeswar, Sathwik Tejaswi Madhusudhan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08031

Browse the full library · About the Tower of Babel