Intuitive physics understanding emerges from self-supervised pretraining on natural videos
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun · 2025
Read Intuitive physics understanding emerges from self-supervised pretraining on natural videos by Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11831.txt.
Introduction
Quentin Garrido and seven co-authors show that intuitive physics understanding emerges in video models trained by self supervised prediction on natural videos. The models learn to predict masked regions of video in a learned representation space. A violation of expectation framework then probes whether the models grasp object permanence, shape consistency, and like properties.
Representation space predictors show clear understanding across physics tests. Pixel space predictors and multimodal language models that reason in text score near chance by contrast. Jointly learning an abstract space while predicting missing input appears sufficient for this core knowledge. Even one week of unique video yields above chance performance. The result challenges claims that such knowledge must be hardwired from birth.
Catalog record
| CREATOR | Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11831
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Intuitive physics understanding emerges from self-supervised pretraining on natural videos Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun Abstract We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regions in natural videos. Leveraging the violation-of-expectation framework, we find that video prediction models trained to predict outcomes in a learned representation space demonstrate an understanding of various intuitive physics properties, such as object permanence and shape consistency. In contrast, video prediction in pixel space and multimodal large language models, which reason through text, achieve performance closer to chance. Our comparisons of these architectures reveal that jointly learning an abstract representation space while predicting missing parts of sensory input, akin to predictive coding, is sufficient to acquire an understanding of intuitive physics, and that even models trained on one week of unique video achieve above chance performance. This challenges the idea that core knowledge — a set of innate systems to help understand the world — needs to be hardwired to develop an understanding of intuitive physics. † † date: August 24, 2026 † † correspondence: Quentin Garrido at [email protected] † † Code and data: https://github.com/facebookresearch/jepa-intuitive-physics Figure 1: Video prediction in representation space (V-JEPA) achieves an understanding of intuitive physics. (A) Video models are evaluated on three intuitive physics datasets using the Violation of Expectation paradigm (IntPhys, GRASP, and InfLevel). V-JEPA is significantly more ‘surprised’ by implausible videos. Random initializations of V-JEPA (untrained networks) show near-chance performance, and state-of-the-art video models based on text or pixel prediction are much closer to chance. Confidence intervals at 95% are obtained via bootstrapping, except for untrained networks ( n=20 ) which use a normal distribution assumption. (B) V-JEPA is trained to ’inpaint’ natural videos in a learned representation space. Starting from a video and a corrupted version, representations are first extracted. The goal is then to predict the representation of the original video from the representation of the corrupted ones. (C) From a trained V-JEPA, we compute a surprise metric by predicting representations of N future frames based on M past ones and comparing the predictions to the representations of observed events. The surprise metric is then used to decide which of the two videos contains a physical violation. An intuitive understanding of physics is fundamental to human cognition: we expect objects to behave predictably, i.e., not to appear or disappear abruptly, move through obstacles, or change shape or color arbitrarily. This basic grasp of the physical world has been documented not only in human infants ( Piaget, 1954 ; Baillargeon and DeVos, 1991 ; Baillargeon et al., 1992 ; Baillargeon and Hanko-Summers, 1990 ; Spelke et al., 1995 ) , but also in primates ( Cacchione and Krist, 2004 ; Mendes et al., 2007 ) , marine mammals ( Singer and Henderson, 2015 ; Herman, 2010 ) , corvids ( Bird and Emery, 2009 ; Taylor et al., 2012 ) , and chicks ( Vallortigara, 2012 ; Wood, 2013 ) . This has been taken as evidence for the core knowledge (or core systems) hypothesis, according to which humans are equipped with a set of innate or early developing evolutionary, ancient computational systems specialized to represent and reason about basic properties of the world: objects, space, numbers, geometry, agents, etc. ( Baillargeon, 2008 ; Spelke and Kinzler, 2007 ; Spelke, 2000 ; Carey, 2000 ) . In the pursuit of building machines with advanced human-level intelligence, rapid progress has produced artificial intelligence (AI) systems that often surpass human performance on high-level cognitive tasks like language, coding or mathematics ( OpenAI, 2024 ) , but paradoxically struggle in common sense physical understanding ( Riochet et al., 2022 ; Weihs et al., 2022 ; Jassim et al., 2024 ; Bisk et al., 2020 ; Benchekroun et al., 2023 ; Bansal et al., 2024 ; Bear et al., 2021 ) , illustrating Moravec’s paradox ( Moravec, 1988 ) , namely, that tasks trivial for biological organisms can be remarkably difficult for artificial systems, and vice versa. Previous work developing AI models with the aim of improving intuitive physics understanding can be sorted into two classes: structured models and pixel-based generative models . Structured models leverage hand-coded abstract representations of objects and their relationships in an Euclidean 3D space ( Battaglia et al., 2013 ; Watters et al., 2017 ) , yielding a powerful mental “game engine” able to capture human’s physical intuitions ( Ullman et al., 2017 ) . This class of models can be seen as a possible computational implementation of the core knowledge hypothesis ( Spelke and Kinzler, 2007 ; Spelke, 2000 ) . 1 1 1 Weaker versions of structured models use object masks and depth cues instead of a full 3D reconstruction, e.g., ( Riochet et al., 2020 ) . Pixel-based generative models take a radically opposite view and deny the need for any hard-coded abstraction. Instead, they propose a general purpose learning mechanism consisting of reconstructing future sensory inputs (e.g., images) based on past ones ( Lerer et al., 2016 ; Goyal et al., 2017a ; Finn et al., 2016 ) . Here, we explore a third class of models that occupies a middle ground between these opposing views, integrating features from both: Joint Embedding Predictive Architectures (JEPAs) ( LeCun, 2022 ; Bardes et al., 2024 ) . As structured models, JEPAs posit that prediction of future world states should be done in the model’s learned abstract, internal representation, and not in terms of low-level, pixel-based prediction or generation.. However, unlike structured models, JEPAs leave it to the algorithm to learn its own representation rather than hand-coding it. The mechanism consisting of predicting in representation space is congruent with the predictive coding hypothesis of cognitive neuroscience ( Hohwy, 2013 ; Rao and Ballard, 1999 ; Clark, 2013 ) . Here we study a video version of this architecture, V-JEPA ( Bardes et al., 2024 ) , which learns to represent video frames by reconstructing masked portions of the video in representation space. We rely on the violation-of-expectation framework to probe for intuitive physics understanding without requiring any task-specific training or adaptation ( Smith et al., 2019 ; Riochet et al., 2022 ; Piloto et al., 2022 ; Riochet et al., 2020 ) . By prompting the model to imagine the (representation of the) future of a video and comparing its predictions with the actual observed future of the video, we obtain a quantitative measure of surprise that can be used to detect violations of intuitive physics concepts. We find that V-JEPA accurately and consistently distinguishes between videos that follow the laws of physics and those that violate them. Specifically, when tasked with classifying the physical plausibility of video pairs, where one video is plausible and the other is not, a V-JEPA model trained on natural videos achieves 98% zero-shot accuracy on the IntPhys benchmark ( Riochet et al., 2022 ) and 62% zero-shot accuracy on the InfLevel benchmark ( Weihs et al., 2022 ) . 2 2 2 Here “zero-shot” refers both to the fact that the V-JEPA models were not trained specifically for the task of distinguishing between physically plausible and implausible videos, and