Can Vision-Language Models Answer Face to Face Questions in the Real-World
Reza Pourreza Rishit Dagli Apratim Bhattacharyya Sunny Panchal, Guillaume Berger Roland Memisevic · 2025
Read Can Vision-Language Models Answer Face to Face Questions in the Real-World by Reza Pourreza Rishit Dagli Apratim Bhattacharyya Sunny Panchal, Guillaume Berger Roland Memisevic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19356.txt.
Introduction
Reza Pourreza and five co-authors published a study in 2025 testing whether vision-language models can hold a live, face-to-face conversation about what a camera sees. Such skill is a standing goal for real-world assistants and robots. The team built a data set of interactive video sessions where users ask questions that must be answered on the spot from camera and microphone input.
Present models fell far short of human performance on the benchmark. The authors trace the gap to specific weak skills rather than a general failure. Fine-tuning on this kind of session data closed much of the distance on many skills. The work gives builders a yardstick for the jump from describing photos to joining a live exchange.
Catalog record
| CREATOR | Reza Pourreza Rishit Dagli Apratim Bhattacharyya Sunny Panchal, Guillaume Berger Roland Memisevic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Vision-Language Models Answer Face to Face Questions in the Real-World? Thanks: Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Reza Pourreza Rishit Dagli Apratim Bhattacharyya Sunny Panchal, Guillaume Berger Roland Memisevic Abstract AI models have made significant strides in recent years in their ability to describe and answer questions about real-world images. They have also made progress in the ability to converse with users in real-time using audio input. This raises the question: have we reached the point where AI models, connected to a camera and microphone, can converse with users in real-time about scenes and events that are unfolding live in front of the camera? This has been a long-standing goal in AI and is a prerequisite for real-world AI assistants and humanoid robots to interact with humans in everyday situations. In this work, we introduce a new dataset and benchmark, the Qualcomm Interactive Video Dataset (QIVD), which allows us to assess the extent to which existing models can support these abilities, and to what degree these capabilities can be instilled through fine-tuning. The dataset is based on a simple question-answering setup, where users ask questions that the system has to answer, in real-time, based on the camera and audio input. We show that existing models fall far behind human performance on this task, and we identify the main sources for the performance gap. However, we also show that for many of the required perceptual skills, fine-tuning on this form of data can significantly reduce this gap. Figure 1: We present the Qualcomm Interactive Video Dataset (QIVD), a dataset collected in an online question-answering setup, where users pose open-ended questions using their camera and microphone. QIVD offers videos with raw audio, annotated textual transcriptions of the spoken questions, and text answers with annotated timestamps. These timestamps indicate when a question can be sensibly answered given the video context. QIVD serves as a realistic and challenging dataset for situated visual reasoning in Large Multi-modal Models. 1 Introduction Recent advancements in Large Multimodal Models (LMM) have significantly enhanced the ability of AI systems to interact naturally and fluently with users in real-time. Existing AI agents can process audio, speech, and visual inputs to engage in conversations about images or videos. However, the conversational capabilities of state-of-the-art LMMs such as GPT-4o Hurst et al. (2024) are limited to question-answering on visual understanding and reasoning tasks, such as describing images or answering questions that require inferring object positions and relations in the visual input. These systems often fail to provide truly situated, live, conversational experiences ( Figure 1 ) that we may expect from humanoid robots or real-time video-call chatbots in the future. We hypothesize that this limitation stems from the fact that current vision-language datasets and benchmarks are biased toward offline reasoning about images and videos. That is, the models receive the entire visual input and the entire question at once before being required to provide an answer. This is because the training data for such tasks can be easily sourced on the internet or easily generated through automated pipelines. There is a distinct lack of benchmarks and datasets that test genuine, real-time, “face-to-face” conversational skills. A separate but related problem is that models are not trained to respond at the appropriate time in a conversation – knowing “when to speak” is crucial for conducting real-world conversations, yet this timing skill remains underdeveloped and understudied in current benchmarks. To address these challenges and assess the limitations of existing models, we introduce the Qualcomm Interactive Video Dataset (QIVD), a new dataset and benchmark designed for end-to-end trained systems aimed at real-time user interaction. QIVD is structured as an online question-answering setup, where users pose open-ended questions using their camera and microphone, and the system must respond appropriately. Our work differs fundamentally from other related datasets and benchmarks by introducing an entirely online question-answering paradigm where both questions and answers evolve in real-time as the video unfolds, requiring models to maintain contextual awareness while handling inherent ambiguities in human references to visual elements. We show how this simple type of interaction allows us to capture a rich set of visual concepts that fall under the umbrella of situated visual understanding, including deictic (referring) expressions, pointing gestures, object ambiguities, behavior and action understanding, and counting, as well as audio-visual concepts. An overview of our dataset is shown in Figure 1 . Due to the in-the-wild nature of the recordings, the videos exhibit considerable variation in lighting conditions, background settings, the range and nature of questions posed, actions performed by subjects, and other audio-visual characteristics. To showcase the unique challenges our dataset presents, we conduct a series of experiments where we evaluate multiple open and closed-source state-of-the-art models, and fine-tuned models on our dataset. Our experiments reveal that the seemingly simple interaction of answering questions live, in real-time, is highly challenging for existing AI systems Hurst et al. (2024) , even if they are otherwise good at performing visual reasoning. Our experiments indicate that the failure modes of existing systems can be attributed to their: (1) difficulty integrating visual and auditory information in real-time to disambiguate questions, (2) inability to determine the appropriate time at which to answer, and (3) inability to answer questions whose answers require situational common sense. Our dataset supports research on online LMMs capable of situated audio-visual reasoning and can be leveraged to build conversational agents that interact with users in real-time. Our contributions are summarized as follows: 1. We introduce QIVD, a novel multi-modal dataset designed to evaluate online situated audio-visual reasoning and real-time conversational skills. 3 3 3 qualcomm.com/developer/software/qualcomm-interactive-video-dataset-qivd 2. We benchmark existing LMMs and identify critical weaknesses in their ability to handle real-life conversations. 3. We demonstrate that these limitations can be effectively mitigated by fine-tuning models on appropriate audio-visual conversational data. 4. We develop a simple yet effective baseline to process streaming audio-visual inputs, departing from traditional offline paradigms. 2 Related Work Offline Video Evaluation Benchmarks : Prior work on video understanding benchmarks has primarily focused on offline evaluation paradigms. There have been multiple temporal video understanding benchmarks for open-domain understanding Li et al. (2024b) ; Liu et al. (2024b) ; Xu et al. (2023) ; Patraucean et al. (2023) ; Ning et al. (2023) ; Hu et al. (2025) , hand movements Goyal et al. (2017) ; Materzynska et al. (2019) , articulated motion Dagli et al. (2024) , full human body motion Panchal et al. (2024) , robotics Haresh et al. (2024) ; Yu et al. (2024) ; Bao et al. (2023a) ; Gu et al. (2023) ; Brohan et al. (2023) ; Jiang et al. (2022) , and embodied reasoning Yang et al. (2025b) . These benchmarks evaluate models’ ability to comprehend temporal relationships but operate in a fully offline manner. Long-form video understanding has been addressed by datasets such as LVBench Wang et al. (2025) , and MoVQA Zhang et al. (2023b) , which extend the context window but fail to simulate real-time constraints. In contrast, our QIVD dataset and benchmark focuses on real-world QA. Situated Video Evaluation Benchmarks : Situated question answering has also been studied by Das et al. (2018) ; Ma et al. (2023) and follow-up works (