TellWhisper: Tell Whisper Who Speaks When

Yifan Hu, Peiji Yang, Zhisheng Wang, Yicheng Zhong, Rui Liu · 2026

Read TellWhisper: Tell Whisper Who Speaks When by Yifan Hu, Peiji Yang, Zhisheng Wang, Yicheng Zhong, Rui Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03712.txt.

Description

Yifan Hu and four co-authors present TellWhisper, a system for multi-speaker speech recognition: deciding who spoke when, and what they said. Meetings and group conversations need exactly that record. Older systems separate the timing problem from the speaker problem, either injecting speaker cues before encoding, which can destroy information, or mixing identity scores afterward, which tangles voice content with identity.

TellWhisper instead models identity and timing jointly inside the speech encoder. Its new positional encoding uses time coordinates from frame positions and speaker coordinates from activity and pause cues. That joint design should be sturdier under fast turn-taking and overlapping speech, where separated pipelines degrade. Better speaker-attributed transcripts support meeting summaries, accessibility tools, and any record where who said what is the whole point.

Catalog record

CREATORYifan Hu, Peiji Yang, Zhisheng Wang, Yicheng Zhong, Rui Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03712

Browse the full library · About the Tower of Babel