TellWhisper: Tell Whisper Who Speaks When
Yifan Hu, Peiji Yang, Zhisheng Wang, Yicheng Zhong, Rui Liu · 2026
Read TellWhisper: Tell Whisper Who Speaks When by Yifan Hu, Peiji Yang, Zhisheng Wang, Yicheng Zhong, Rui Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03712.txt.
Description
Yifan Hu and four co-authors present TellWhisper, a system for multi-speaker speech recognition: deciding who spoke when, and what they said. Meetings and group conversations need exactly that record. Older systems separate the timing problem from the speaker problem, either injecting speaker cues before encoding, which can destroy information, or mixing identity scores afterward, which tangles voice content with identity.
TellWhisper instead models identity and timing jointly inside the speech encoder. Its new positional encoding uses time coordinates from frame positions and speaker coordinates from activity and pause cues. That joint design should be sturdier under fast turn-taking and overlapping speech, where separated pipelines degrade. Better speaker-attributed transcripts support meeting summaries, accessibility tools, and any record where who said what is the whole point.
Catalog record
| CREATOR | Yifan Hu, Peiji Yang, Zhisheng Wang, Yicheng Zhong, Rui Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03712