On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
Jeff Chan-Jan Sju, Liang-Hsuan Tseng, Yi-Cheng Lin, Yen-Chun Kuo, Ju-Chieh Chou, Kai-Wei Chang, Hung-yi Lee, Carlos Busso · 2026
Read On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation by Jeff Chan-Jan Sju, Liang-Hsuan Tseng, Yi-Cheng Lin, Yen-Chun Kuo, Ju-Chieh Chou, Kai-Wei Chang, Hung-yi Lee, Carlos Busso (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06329.txt.
Description
Speech models should not be graded like text models, argue Chan-Jan Hsu and 7 co-authors in a 2026 paper. Generative spoken language models continue a speech prompt while preserving speaker and emotion. Researchers often score them with global token perplexity, a text formula applied directly to speech tokens, which ignores how speech differs from text.
The authors propose likelihood-based and generation-based alternatives to replace that naive measure. Their evaluations track human mean opinion scores more faithfully than perplexity does. Under the new metrics, the ranking of spoken models changes. The gap between the best model and human performance shrinks significantly. Appropriate evaluation, they conclude, is essential to judge real progress in spoken language modeling.
Catalog record
| CREATOR | Jeff Chan-Jan Sju, Liang-Hsuan Tseng, Yi-Cheng Lin, Yen-Chun Kuo, Ju-Chieh Chou, Kai-Wei Chang, Hung-yi Lee, Carlos Busso |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06329