Rethinking Visual Intelligence: Insights from Video Pretraining

Pablo Acuaviva, Aram Davtyan, Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Alexandre Alahi, Paolo Favaro · 2025

Read Rethinking Visual Intelligence: Insights from Video Pretraining by Pablo Acuaviva, Aram Davtyan, Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Alexandre Alahi, Paolo Favaro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24448.txt.

Introduction

Pablo Acuaviva and six co-authors investigate video pretraining as a route to broader visual intelligence. Large scale pretraining let language models adapt quickly to new problems, but visual models still struggle with compositional understanding, sample efficiency, and general problem solving. Video diffusion models, trained on data that unfolds over time, may carry strong biases for structure and dynamics.

The team designs a controlled evaluation comparing a video pretrained model against alternatives on downstream tasks. They test whether spatiotemporal pretraining supports broad task adaptability with little supervision, as language pretraining does for text. The study offers insights into what visual pretraining must provide for general purpose visual systems.

Catalog record

CREATORPablo Acuaviva, Aram Davtyan, Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Alexandre Alahi, Paolo Favaro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24448

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Visual Intelligence: Insights from Video Pretraining Pablo Acuaviva, Aram Davtyan, Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Alexandre Alahi, Paolo Favaro Abstract Large language models (LLMs) have demonstrated that large-scale pretraining enables systems to adapt rapidly to new problems with little supervision in the language domain. This success, however, has not translated as effectively to the visual domain, where models, including LLMs, continue to struggle with compositional understanding, sample efficiency, and general-purpose problem-solving. We investigate Video Diffusion Models (VDMs) as a promising direction for bridging this gap. Pretraining on spatiotemporal data endows these models with strong inductive biases for structure and dynamics, which we hypothesize can support broad task adaptability. To test this, we design a controlled evaluation in which both a pretrained LLM and a pretrained VDM are equipped with lightweight adapters and presented with tasks in their natural modalities. Across benchmarks including ARC-AGI, ConceptARC, visual games, route planning, and cellular automata, VDMs demonstrate higher data efficiency than their language counterparts. Taken together, our results indicate that video pretraining offers inductive biases that support progress toward visual foundation models. † † footnotetext: Code available at https://github.com/PabloAcuaviva/visual-intelligence 1 Introduction Figure 1: Radar plot showing ConceptARC competencies between VDMs and LLMs , GPT-4 [IC] is added for additional reference. Foundation models have reshaped natural language processing by showing that large-scale pretraining can equip models with broad knowledge and strong inductive priors. This foundation allows models to adapt quickly and effectively to new tasks through techniques like in-context learning ( Brown et al. 2020 ) and parameter-efficient fine-tuning ( Liu et al. 2022 ) , achieving strong performance with minimal supervision. The success of Large Language Models (LLMs) illustrates how scale and pretraining can create systems that generalize across diverse problems. Achieving a similar level of versatility in vision, however, remains largely unexplored and a major challenge. Despite recent breakthroughs in image and video generation ( Labs 2025 ; Polyak et al. 2024 ; Qin et al. 2024 ) , vision models are not yet on par with LLMs when it comes to compositional skills, sample efficiency, and versatility in problem solving. Video Diffusion Models (VDMs) represent an exciting direction for narrowing this gap. Pretraining on rich spatiotemporal data endows them with strong inductive biases for spatial structure and temporal dynamics ( Blattmann et al. 2023 ; Google DeepMind 2025 ; Wu et al. 2025 ) , which we hypothesize can be harnessed for structured visual understanding. We move beyond treating videos as mere generative artifacts and instead regard them as a natural representation for problem solving, where tasks are expressed as transformations unfolding over time. Building on this perspective, we introduce a simple and general framework for adapting VDMs to a broad class of visual tasks and evaluate them head-to-head with equally adapted LLMs (see Figure 1 ). This setup allows us to test whether large-scale video pretraining offers a complementary foundation for structured visual problem-solving, contrasting the strengths of visually grounded models with those of symbolically trained language models. Each task is represented consistently but adapted to each model family’s modality: LLMs operate in a text-to-text setting, where inputs and outputs are serialized into structured text, while VDMs receive an image-to-image formulation, where input–output pairs are rendered as short videos to model the task as a temporal transformation. Both model families use identical LoRA-based Hu et al. 2022 adaptation: adapters are inserted at corresponding layers, pretrained backbones remain frozen, and only lightweight parameters are updated. This symmetry provides a controlled basis for comparison and isolates the impact of video pretraining on structured visual understanding. Our contributions are as follows: 1. A unified framework for adapting VDMs to image-to-image visual tasks by reframing examples as temporal sequences. 2. A controlled evaluation setting where both VDMs and LLMs are fine-tuned with LoRA-based adaptation, enabling direct comparison. 3. Empirical evidence that VDMs benefit from video pretraining for visual intelligence, hinting at a path toward flexible visual foundation models with both generative and problem-solving strengths. 2 Related Work Language Foundation Models. LLMs have demonstrated remarkable generalization and adaptability to new tasks with minimal supervision, mainly due to their large-scale pretraining on diverse text corpora Brown et al. 2020 ; Chowdhery et al. 2023 . Their extensive pretraining equips LLMs with rich knowledge and strong inductive biases, enabling them to perform few-shot learning Brown et al. 2020 and in-context learning Coda-Forno et al. 2023 , where models learn new tasks only by observing a handful of examples. Parameter-efficient finetuning methods like LoRA Hu et al. 2022 extend this adaptability allowing LLMs to specialize to new domains while the backbone is completely frozen Liao et al. 2025 . Together, these capabilities make LLMs highly flexible and scalable problem solvers. In this paper, we leverage this adaptability to compare the data efficiency of LLMs and VDMs across diverse visual tasks. Video Diffusion Models. Diffusion-based generative models have recently achieved remarkable progress in video synthesis. Pioneering approaches such as CogVideo Hong et al. 2022 and Villegas et al. 2022 introduced scalable architectures for text-to-video generation. More recent models like Sora Qin et al. 2024 , MovieGen Polyak et al. 2024 , Veo 3 Google DeepMind 2025 , and CogVideoX Yang et al. 2024 set new standards for quality and realism. Recent work has investigated controllable video generation NVIDIA et al. 2025 ; Hassan et al. 2025 ; Kanervisto et al. 2025 , with the goal of producing realistic, high-quality videos while allowing precise control over motion and dynamics. These methods emphasize modeling dynamic environments and predicting plausible future states conditioned on past observations and control inputs. Visual Foundation Models Recent work has investigated the use of generative models as generalist vision models. Methods such as image inpainting for visual prompting Bar et al. 2022 and image-based in-context learning Wang et al. 2023a demonstrate that structured inputs can enable these models to solve diverse tasks. Diffusion models have further been extended to in-context learning Wang et al. 2023b , instruction following across heterogeneous tasks Geng et al. 2024 , and broader computer vision problem solving Zhao et al. 2025 . Sequential modeling has been proposed as a unified interface for scaling vision models Bai et al. 2024 . Building on this line of work, Lin et al. 2025 train CogVideoX1.5 with temporal in-context prompts for multi-task learning, but their focus remains on broad computer vision benchmarks rather than visual intelligence, and their method requires extensive training 1 1 1 We add qualitative results on standard computer vision tasks in the Appendix to show that our framework can also be extended to this setting. . Our approach does not attempt to build a foundation model from scratch. Instead, we investigate whether a pretrained VDM, pretrained extensively on next-frame prediction, can begin to exhibit the properties expected of visual foundation models by leveraging inductive biases gained through spatiotemporal pretraining. 3 Methodology 3.1 Setup and Comparison Protocol We adopt the definition of intelligence proposed by Chollet 2019 : The intelligence of a system is a measure of its skill acquisition efficiency over a scope of tasks with respect to priors, experience, and generalization difficulty. This perspective motivates our evaluation design. We focus not only on absolute accuracy but also on how quickly

Read the complete text