HumanOmni-Speaker: Identifying Who said What and When

Detao Bai, Shimin Yao, Weixuan Chen, Xihan Wei, Zhiheng Ma · 2026

Read HumanOmni-Speaker: Identifying Who said What and When by Detao Bai, Shimin Yao, Weixuan Chen, Xihan Wei, Zhiheng Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21664.txt.

Description

Bai and co-authors tackle a hard video task: who said what, and when, in multi-person scenes. Omni-modal models often cheat here, leaning on visual bias in test sets instead of true sound-vision links. Sparse frame sampling also erases lip motion. Their test setup removes visual shortcuts and forces real identity binding from plain language queries. Their model samples video at 25 frames per second and squeezes motion into six tokens per frame. It can lip-read and locate speakers with no cropping, they report.

Catalog record

CREATORDetao Bai, Shimin Yao, Weixuan Chen, Xihan Wei, Zhiheng Ma
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38083
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21664

Browse the full library · About the Tower of Babel