AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation

Potsawee Manakul, Woody Haosheng Gan, Michael J. Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Held, Diyi Yang · 2025

Read AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation by Potsawee Manakul, Woody Haosheng Gan, Michael J. Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Held, Diyi Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12705.txt.

Description

Potsawee Manakul and co-authors test big audio models as judges of speech, in a 2025 paper. Grading speech today needs a bespoke tool per trait, and auto scores often clash with human taste. Their AudioJudge study asks whether one audio model can grade many traits at once.

Tasks span pronunciation, pace, speaker identity, and sound quality, plus full system rankings against human panels. Joining audio clips end to end and adding worked examples lifted scores on most tasks. A team-of-judges variant splits the work by content, quality, and tone, reaching 0.91 rank agreement with human panels on system order. Noise tests show the judges hold up under rough audio, though gaps remain.

Catalog record

CREATORPotsawee Manakul, Woody Haosheng Gan, Michael J. Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Held, Diyi Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12705

Browse the full library · About the Tower of Babel