AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation
Potsawee Manakul, Woody Haosheng Gan, Michael J. Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Held, Diyi Yang · 2025
Read AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation by Potsawee Manakul, Woody Haosheng Gan, Michael J. Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Held, Diyi Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12705.txt.
Description
Potsawee Manakul and co-authors test big audio models as judges of speech, in a 2025 paper. Grading speech today needs a bespoke tool per trait, and auto scores often clash with human taste. Their AudioJudge study asks whether one audio model can grade many traits at once.
Tasks span pronunciation, pace, speaker identity, and sound quality, plus full system rankings against human panels. Joining audio clips end to end and adding worked examples lifted scores on most tasks. A team-of-judges variant splits the work by content, quality, and tone, reaching 0.91 rank agreement with human panels on system order. Noise tests show the judges hold up under rough audio, though gaps remain.
Catalog record
| CREATOR | Potsawee Manakul, Woody Haosheng Gan, Michael J. Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Held, Diyi Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12705