VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents
Amrita Mazumdar, Seonwook Park, Rajarshi Roy, Nikhil Srihari, Shengze Wang, Yuhao Zhou Julia Wang, Koki Nagano Shalini De Mello · 2026
Read VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents by Amrita Mazumdar, Seonwook Park, Rajarshi Roy, Nikhil Srihari, Shengze Wang, Yuhao Zhou Julia Wang, Koki Nagano Shalini De Mello (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30256.txt.
Description
Amrita Mazumdar and eight co-authors present VideoFDB, the first benchmark for full-duplex audio-visual agents that speak, listen and read nonverbal cues at once. Current benchmarks cover speech only. VideoFDB contributes 237 dyadic clips spanning 11 dynamics from video calls, a taxonomy separating perception from generation, and a rubric-based judge evaluation with clear quality axes. Across vision-speech agents, failures appear: captioning collapse and visual-stream ignorance, with vision used for visual question answering but not streaming audiovisual grounding. Cascaded speech-to-avatar systems fundamentally cannot produce nonverbal cues.
Catalog record
| CREATOR | Amrita Mazumdar, Seonwook Park, Rajarshi Roy, Nikhil Srihari, Shengze Wang, Yuhao Zhou Julia Wang, Koki Nagano Shalini De Mello |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96611 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30256