Video Event Reasoning and Prediction by Fusing World Knowledge from LLMs with Vision Foundation Models
Léa Dubois, Klaus Schmidt, Chengyu Wang, Ji-Hoon Park, Lin Wang, Santiago Muñoz · 2025
Read Video Event Reasoning and Prediction by Fusing World Knowledge from LLMs with Vision Foundation Models by Léa Dubois, Klaus Schmidt, Chengyu Wang, Ji-Hoon Park, Lin Wang, Santiago Muñoz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05822.txt.
Introduction
Léa Dubois and co-authors built in 2025 a video system that does not just name what happens on screen but reasons about causes and guesses what comes next. Ordinary video models stop at recognition because they lack everyday world knowledge. Their design pairs a vision model for deep seeing with a large language model as the thinking core.
A fusion module, modeled on a known bridging design, squeezes rich visual detail into a compact form the language model can use. Training ran in two stages: broad alignment on video and text, then targeted lessons in reasoning. The system topped several hard benchmarks and handled unseen reasoning tasks without retraining. The authors pitch it as a step from machine seeing toward machine understanding.
Catalog record
| CREATOR | Léa Dubois, Klaus Schmidt, Chengyu Wang, Ji-Hoon Park, Lin Wang, Santiago Muñoz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05822
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video Event Reasoning and Prediction by Fusing World Knowledge from LLMs with Vision Foundation Models Léa Dubois, Klaus Schmidt, Chengyu Wang, Ji-Hoon Park, Lin Wang, Santiago Muñoz Abstract Current video understanding models excel at recognizing "what" is happening but fall short in high-level cognitive tasks like causal reasoning and future prediction, a limitation rooted in their lack of commonsense world knowledge. To bridge this cognitive gap, we propose a novel framework that synergistically fuses a powerful Vision Foundation Model (VFM) for deep visual perception with a Large Language Model (LLM) serving as a knowledge-driven reasoning core. Our key technical innovation is a sophisticated fusion module, inspired by the Q-Former architecture, which distills complex spatiotemporal and object-centric visual features into a concise, language-aligned representation. This enables the LLM to effectively ground its inferential processes in direct visual evidence. The model is trained via a two-stage strategy, beginning with large-scale alignment pre-training on video-text data, followed by targeted instruction fine-tuning on a curated dataset designed to elicit advanced reasoning and prediction skills. Extensive experiments demonstrate that our model achieves state-of-the-art performance on multiple challenging benchmarks. Notably, it exhibits remarkable zero-shot generalization to unseen reasoning tasks, and our in-depth ablation studies validate the critical contribution of each architectural component. This work pushes the boundary of machine perception from simple recognition towards genuine cognitive understanding, paving the way for more intelligent and capable AI systems in robotics, human-computer interaction, and beyond. Keywords: Computer Vision, Video Understanding, Large Language Models (LLMs), Multimodal AI, Event Reasoning, Future Prediction, Vision-Language Models (VLMs), Foundation Models. 1 Introduction The proliferation of video data has positioned it as a primary medium for information exchange and environmental perception, driving a significant evolution in computer vision research ( Fu et al. 2024 ; Xiong et al. 2024 ) . Historically, the field has achieved remarkable success in discriminative tasks—moving from foundational image recognition ( Dosovitskiy et al. 2021 ) to complex action and gesture recognition in videos ( Abu Farha et al. 2021 ) . This progress is mirrored in specialized domains such as sign language translation, where hierarchical models have demonstrated powerful recognition capabilities ( Guo et al. 2018 ; Guo et al. 2019b ) , and in fine-grained sensing technologies like WiFi-based gesture and activity recognition, which have become increasingly robust and resistant to interference ( Gu et al. 2023a ; Huang et al. 2023 ; Huang et al. 2020 ; Zhang et al. 2025a ) . However, this paradigm of "recognition" primarily addresses the question of "what" is happening in a visual scene. The community is now shifting towards a more profound challenge: moving from mere perception to genuine cognition ( Zellers et al. 2019 ) . This involves enabling machines to reason about "why" an event is occurring and to predict "what" might happen next—a task that requires a level of understanding far beyond statistical pattern matching. The primary obstacle to achieving this cognitive leap is the "knowledge gap." Current models, despite their architectural sophistication, often operate within a closed world, lacking the vast repository of commonsense, physical intuition, and social knowledge that humans effortlessly apply. For instance, while a model might accurately classify a video clip as "a person picking up an egg and flour," it typically fails to infer the underlying intent, such as "the person is about to bake a cake." This limitation becomes particularly salient when considering the demands of next-generation applications. Advanced human-machine systems ( Zhang et al. 2025a ) , affective computing that aims to suppress label noise for reliability ( Zhang et al. 2025b ) , and even emerging fields like psychological understanding via language models ( Hu et al. 2024 ) all necessitate a deeper, causal understanding of events. Similarly, while we can now capture nuanced physiological data through commodity devices ( Zhao et al. 2025 ; Wang et al. 2024 ) , interpreting this data in the context of complex human activities requires advanced reasoning. The core challenge, therefore, is to imbue visual models with this external world knowledge. To bridge this gap, we propose a novel framework that synergistically fuses the capabilities of two of the most powerful paradigms in modern AI: Vision Foundation Models (VFMs) and Large Language Models (LLMs). Our approach leverages VFMs, such as those built upon the principles of Vision Transformers ( Dosovitskiy et al. 2021 ) and trained with multimodal supervision ( Radford et al. 2021 ; Wang et al. 2022 ) , to serve as the "eyes" of our system. These models are adept at extracting rich, spatiotemporal features and providing a detailed, pixel-level perception of the visual world. Concurrently, we employ a pre-trained LLM, such as LLaMA ( Touvron et al. 2023 ) , as the "brain"—a reasoning core endowed with a vast reservoir of world knowledge, causal relationships, and abstract concepts, as demonstrated in seminal works like GPT-4 ( Bubeck et al. 2023 ) . The crux of our methodology lies in a meticulously designed fusion mechanism that translates the continuous, unstructured visual evidence from the VFM into a discrete, language-like format that the LLM can process and reason upon. This approach is inspired by the success of pioneering vision-language architectures like Flamingo ( Alayrac et al. 2022 ) and BLIP-2 ( Li et al. 2023 ) , but is explicitly tailored for complex event-level inference and prediction, moving beyond simple captioning or direct question-answering. This work makes several significant contributions to the advancement of cognitive video understanding. First and foremost, we introduce a novel and effective framework that, for the first time, cohesively integrates a state-of-the-art Vision Foundation Model with a large-scale LLM to perform high-level event reasoning and prediction, moving decisively beyond simple recognition tasks. Central to this framework is our second contribution: the design of a lightweight yet powerful cross-modal fusion module. This component acts as an efficient information bottleneck, aligning rich visual features with the LLM’s semantic space, which not only enables complex inference but also ensures that the model’s reasoning is firmly grounded in direct visual evidence. To validate our approach, our third contribution is a set of extensive experiments conducted on multiple challenging video reasoning benchmarks. The results demonstrate that our model significantly outperforms existing state-of-the-art methods and, notably, exhibits remarkable zero-shot capabilities in predicting future events, underscoring the profound benefits of transferring world knowledge from LLMs. We believe the principles developed herein also hold considerable promise for enhancing related multimodal tasks, such as visual dialog ( Guo et al. 2021 ) and audio-visual event analysis ( Zhou et al. 2022 ) . The remainder of this paper is organized as follows. Section 2 provides a comprehensive review of related literature. Section 3 presents the detailed architecture and components of our proposed methodology. Section 4 describes our experimental setup, datasets, and presents both quantitative and qualitative results. Finally, Section concludes the paper and discusses promising directions for future work. 2 Related Work The endeavor to imbue machines with the ability to reason about and predict events in videos stands at the confluence of several key research streams in artificial intelligence. This section surveys the landscape of relevant work, beginning with the foundations of visual representation learning, moving through the evolution of multimodal models, and