VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents

Amrita Mazumdar, Seonwook Park, Rajarshi Roy, Nikhil Srihari, Shengze Wang, Yuhao Zhou Julia Wang, Koki Nagano Shalini De Mello · 2026

Read VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents by Amrita Mazumdar, Seonwook Park, Rajarshi Roy, Nikhil Srihari, Shengze Wang, Yuhao Zhou Julia Wang, Koki Nagano Shalini De Mello (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30256.txt.

Description

Amrita Mazumdar and eight co-authors present VideoFDB, the first benchmark for full-duplex audio-visual agents that speak, listen and read nonverbal cues at once. Current benchmarks cover speech only. VideoFDB contributes 237 dyadic clips spanning 11 dynamics from video calls, a taxonomy separating perception from generation, and a rubric-based judge evaluation with clear quality axes. Across vision-speech agents, failures appear: captioning collapse and visual-stream ignorance, with vision used for visual question answering but not streaming audiovisual grounding. Cascaded speech-to-avatar systems fundamentally cannot produce nonverbal cues.

Catalog record

CREATORAmrita Mazumdar, Seonwook Park, Rajarshi Roy, Nikhil Srihari, Shengze Wang, Yuhao Zhou Julia Wang, Koki Nagano Shalini De Mello
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96611
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30256

Browse the full library · About the Tower of Babel