Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training

Yifan Yang, Bing Han, Hui Wang, Wei Wang, Ziyang Ma, Long Zhou, Zengrui Jin, Guanrou Yang, Tianrui Wang, Xu Tan, Xie Chen · 2026

Read Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training by Yifan Yang, Bing Han, Hui Wang, Wei Wang, Ziyang Ma, Long Zhou, Zengrui Jin, Guanrou Yang, Tianrui Wang, Xu Tan, Xie Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03065.txt.

Description

Yifan Yang and ten co-authors present FCaps, a large dataset for teaching machines the fine details of speaking style. Existing speech-text models learn from coarse captions or narrow task labels, and detailed style annotations are hard to scale. FCaps holds about forty-seven thousand hours of speech with nineteen million fine-grained free-text captions, produced by a pipeline that grounds captions directly in the audio.

Because the captions are not rewritten by a language model after the fact, the pipeline avoids the error build-up of cascaded methods, and judge-based evaluations rate its captions above cascaded ones. On top of FCaps, the authors train CLSP, a contrastive model that mixes global and fine-grained supervision. It performs reliably across retrieval, zero-shot classification of speech traits, and style similarity scoring.

Catalog record

CREATORYifan Yang, Bing Han, Hui Wang, Wei Wang, Ziyang Ma, Long Zhou, Zengrui Jin, Guanrou Yang, Tianrui Wang, Xu Tan, Xie Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03065

Browse the full library · About the Tower of Babel