IntentVLM: Open-Vocabulary Intention Recognition through Forward–Inverse Modeling with Video-Language Models Conference: Proceedings of the 28th International Conference on Multimodal Interaction; October 5–9, 2026; Napoli, Italy Proceedings of the 28th International Conference on Multimodal Interaction (ICMI ’26), October 5–9, 2026, Napoli, Italy CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Machine learning approaches
Hamed Rahimi, Clémence Grislain, Adrien Jacquet Crétides, Olivier Sigaud, Mohamed Chetouani Institute of Intelligent Systems and Robotics (ISIR), Sorbonne University Paris, France firstname.lastname @isir.upmc.fr · 2026
Read IntentVLM: Open-Vocabulary Intention Recognition through Forward–Inverse Modeling with Video-Language Models Conference: Proceedings of the 28th International Conference on Multimodal Interaction; October 5–9, 2026; Napoli, Italy Proceedings of the 28th International Conference on Multimodal Interaction (ICMI ’26), October 5–9, 2026, Napoli, Italy CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Machine learning approaches by Hamed Rahimi, Clémence Grislain, Adrien Jacquet Crétides, Olivier Sigaud, Mohamed Chetouani Institute of Intelligent Systems and Robotics (ISIR), Sorbonne University Paris, France firstname.lastname @isir.upmc.fr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24002.txt.
Introduction
Rahimi and four colleagues present IntentVLM, a two-stage video-language system that guesses what a person intends in open, everyday language. It first proposes goal candidates, then picks among them with reasoning drawn from forward-inverse modeling, which cuts invented reasoning steps. On two test sets it reaches up to 80 percent accuracy, beating its baseline by 30 percent and matching human performance.
Catalog record
| CREATOR | Hamed Rahimi, Clémence Grislain, Adrien Jacquet Crétides, Olivier Sigaud, Mohamed Chetouani Institute of Intelligent Systems and Robotics (ISIR), Sorbonne University Paris, France firstname.lastname @isir.upmc.fr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39862 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.24002
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
IntentVLM: Open-Vocabulary Intention Recognition through Forward–Inverse Modeling with Video-Language Models Conference: Proceedings of the 28th International Conference on Multimodal Interaction; October 5–9, 2026; Napoli, Italy Proceedings of the 28th International Conference on Multimodal Interaction (ICMI ’26), October 5–9, 2026, Napoli, Italy CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Machine learning approaches Hamed Rahimi, Clémence Grislain, Adrien Jacquet Crétides, Olivier Sigaud, Mohamed Chetouani Institute of Intelligent Systems and Robotics (ISIR), Sorbonne University Paris, France firstname.lastname @isir.upmc.fr Abstract . Improving the effectiveness of human–robot interaction requires social robots to accurately infer human goals through robust intention understanding. This challenge is particularly critical in multimodal settings, where agents must integrate heterogeneous signals including text, visual cues to form a coherent interpretation of user intent. This paper presents IntentVLM , a novel two-stage video-language framework designed for open-vocabulary human intention recognition. The approach is inspired by forward-inverse modeling in cognitive science by decomposing intention understanding into goal candidate generation followed by structured inference through selection, effectively reducing hallucinations in latent reasoning. Evaluated on the IntentQA and Inst-IT Bench datasets, IntentVLM achieves state-of-the-art results with up to 80% accuracy, notably surpassing the baseline performance by 30% and matches human performance. Our findings demonstrate that this structured reasoning approach enhances open-vocabulary intention understanding without catastrophic forgetting, offering a robust foundation for human-centered robotics. Figure 1 . IntentVLM for open-vocabulary human intention recognition from video streams. Given a continuous video stream of a human in a kitchen environment, IntentVLM integrates multimodal visual cues — scene context, object interactions, and body movements — to infer free-form user intentions without relying on a predefined action vocabulary. The model reasons over temporal video evidence to produce natural language intention descriptions ("Make coffee", "Heat water for tea"), demonstrating the capacity of video-language understanding to ground open-vocabulary human goals in real-world robotic settings. † † cc-license: by Code and Data 1. Introduction As robots increasingly transition from controlled industrial environments into everyday proactive robot human-centered spaces ( 56 ) , their ability to understand and respond appropriately to human behavior becomes critically important ( 37 ) . In such settings, effective interaction requires more than simply perceiving observable actions, meaning robots must also reason about the underlying human intentions that drive those actions. Human intention refers to a mental state representing a goal-directed commitment to performing a particular action ( 8 ; 12 ) . Within the framework of mental state attribution, intentions function as internal states that guide and motivate behavior, enabling an observer to interpret why an agent behaves in a particular way. For example, as shown in Figure 1 , when a person enters a reception area and begins scanning the room (e.g. looking at people, objects, or walls) their behavior cannot be misinterpreted as casual curiosity, but in reality, this pattern more commonly signals an underlying intention to seek guidance or assistance ( 66 ) . Recognizing such subtle behavioral cues is essential to accurately infer user intent and respond appropriately. Figure 2 . Closed-vocabulary solutions vs our approach. Conventional approaches formulate intention recognition as closed-set classification over a predefined label space given visual inputs. Inspired by forward–inverse modeling in cognitive science, our method removes this requirement by decomposing the task into (i) goal candidate generation and (ii) structured selection via a video–language model. This two-stage formulation reduces hallucinations in latent reasoning and supports open-vocabulary intention prediction. While intention is often discussed in the context of human cognition, it is not exclusive to humans. Animals exhibit goal-directed behavior, and autonomous robotic systems act according to internal models and task-oriented objectives ( 54 ) . In this broader view, an agent’s behavior can be understood through the interplay of beliefs (probabilistic representations of the world), goals (desired states to be achieved), intentions (commitments to particular courses of action), and policies (action-selection mechanisms) ( 12 ) . Consequently, intention recognition can be defined as the process of inferring an agent’s latent intentions (i.e., its committed plans or policies) by reasoning over observable actions in relation to underlying beliefs, goals, and environmental constraints. Rather than focusing solely on what an agent does, intention recognition seeks to explain why those actions are performed and to predict future behavior accordingly ( 23 ) . Despite its importance, intention recognition remains a challenging problem ( 28 ) . The primary difficulty arises from the fact that intentions are inherently unobservable. Unlike physical actions, which can be directly perceived through sensory input, intentions exist only as latent mental states that must be inferred indirectly ( 23 ) . Observers therefore infer an agent’s intention, which is its underlying goal-directed commitment, by integrating behavioral patterns, contextual information, prior knowledge, and communicative cues ( 14 ) . Furthermore, intention expression is inherently multimodal ( 55 ) . Humans convey intent through interactions with the environment with a combination of visual and linguistic signals. Another challenge stems from the imperfect relationship between actions and intentions ( 41 ) . Not every intention culminates in an observable action, nor does every action faithfully reveal the agent’s underlying intention. A goal denotes the desired outcome an individual seeks to achieve, whereas intention captures the specific commitment or plan to act toward that outcome. In practice, these constructs may diverge: an agent may intend to perform a beneficial action yet fail due to external constraints, or may execute an apparently helpful behavior while pursuing a self-serving objective ( 18 ) . Such misalignment between goals, intentions, and observable actions introduces substantial ambiguity in behavior interpretation, thereby making robust intention inference a central challenge for robotic systems. A large body of research has addressed intention recognition from multiple perspectives ( 23 ) , including Bayesian inverse planning ( 9 ) , probabilistic reasoning ( 57 ) , plan recognition ( 29 ) , and learning-based approaches ( 53 ) . However, as shown in Figure 2 many existing methods simplify the problem by restricting possible intentions to a predefined set, often represented by a limited vocabulary of intention labels ( 65 ) . While such formulations simplify learning and evaluation, they significantly restrict the expressiveness of intention modeling. Human intentions frequently involve nuanced psychological states, social goals, or compound motivations that cannot be easily represented by a single discrete label ( 9 ) . Additionally, a large portion of prior work relies primarily on static image-based perception, which limits the system’s ability to capture temporal dynamics and contextual evolution present in real-world scenes. As a result, integrating intention recognition into complex, temporally evolving visual scenes remains an open challenge. To address these limitations, as shown in Figure 3 , we introduce IntentVLM , a video foundation model ( 36 ) designed for open-ended human intention understanding. We formulate intention recognition as a cognitive modeling problem grounded in forward–inverse reasoning ( 21 ) , and instantiate it as a video–language reasoning task using two vision–language models (VLMs) ( 64 ) . Given a video of a scene and a textual query about the intention of an observed agent, the system outputs a textual description of the inferred intention. Our framework comprises two complementary modules. The first generates a diverse set of candidate intentions conditioned on