Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training
Yifan Yang, Bing Han, Hui Wang, Wei Wang, Ziyang Ma, Long Zhou, Zengrui Jin, Guanrou Yang, Tianrui Wang, Xu Tan, Xie Chen · 2026
Read Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training by Yifan Yang, Bing Han, Hui Wang, Wei Wang, Ziyang Ma, Long Zhou, Zengrui Jin, Guanrou Yang, Tianrui Wang, Xu Tan, Xie Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03065.txt.
Description
Yifan Yang and ten co-authors present FCaps, a large dataset for teaching machines the fine details of speaking style. Existing speech-text models learn from coarse captions or narrow task labels, and detailed style annotations are hard to scale. FCaps holds about forty-seven thousand hours of speech with nineteen million fine-grained free-text captions, produced by a pipeline that grounds captions directly in the audio.
Because the captions are not rewritten by a language model after the fact, the pipeline avoids the error build-up of cascaded methods, and judge-based evaluations rate its captions above cascaded ones. On top of FCaps, the authors train CLSP, a contrastive model that mixes global and fine-grained supervision. It performs reliably across retrieval, zero-shot classification of speech traits, and style similarity scoring.
Catalog record
| CREATOR | Yifan Yang, Bing Han, Hui Wang, Wei Wang, Ziyang Ma, Long Zhou, Zengrui Jin, Guanrou Yang, Tianrui Wang, Xu Tan, Xie Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03065