Grounding Social Perception in Intuitive Physics

Lance Ying, Aydan Y. Huang, Aviv Netanyahu, Andrei Barbu, Boris Katz, [0.3em] Joshua B. Tenenbaum, Tianmin Shu, [1.5em] Massachusetts Institute of Technology Harvard University · 2026

Read Grounding Social Perception in Intuitive Physics by Lance Ying, Aydan Y. Huang, Aviv Netanyahu, Andrei Barbu, Boris Katz, [0.3em] Joshua B. Tenenbaum, Tianmin Shu, [1.5em] Massachusetts Institute of Technology Harvard University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27410.txt.

Introduction

Ying and co-authors ask how people read social meaning from motion. They built PHASE, a large set of short two-agent cartoon scenes. Each scene varies walls, goals, and friendly or hostile ties. Their model, SIMPLE, blends planning with physics simulation. It infers goals and ties from paths alone. SIMPLE matches human judgments with high accuracy. Strong vision-language models fall short of that level. Physics-free planning models also fail. The work backs a physics-grounded view of social perception.

Catalog record

CREATORLance Ying, Aydan Y. Huang, Aviv Netanyahu, Andrei Barbu, Boris Katz, [0.3em] Joshua B. Tenenbaum, Tianmin Shu, [1.5em] Massachusetts Institute of Technology Harvard University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59397
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27410

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grounding Social Perception in Intuitive Physics Lance Ying, Aydan Y. Huang, Aviv Netanyahu, Andrei Barbu, Boris Katz, [0.3em] Joshua B. Tenenbaum, Tianmin Shu, [1.5em] Massachusetts Institute of Technology Harvard University Abstract People infer rich social information from others’ actions. These inferences are often constrained by the physical world: what agents can do, what obstacles permit, and how the physical actions of agents causally change an environment and other agents’ mental states and behavior. We propose that such rich social perception is more than visual pattern matching, but rather a reasoning process grounded in an integration of intuitive psychology with intuitive physics. To test this hypothesis, we introduced PHASE (PHysically grounded Abstract Social Events), a large dataset of procedurally generated animations, depicting physically simulated two-agent interactions on a 2D surface. Each animation follows the style of the Heider and Simmel movie, with systematic variation in environment geometry, object dynamics, agent capacities, goals, and relationships (friendly/adversarial/neutral). We then present a computational model, SIMPLE, a physics-grounded Bayesian inverse planning model that integrates planning, probabilistic planning, and physics simulation to infer agents’ goals and relations from their trajectories. Our experimental results showed that SIMPLE achieved high accuracy and agreement with human judgments across diverse scenarios, while feedforward baseline models—including strong vision-language models—and physics-agnostic inverse planning failed to achieve human-level performance and did not align with human judgments. These results suggest that our model provides a computational account for how people understand physically grounded social scenes by inverting a generative model of physics and agents. 1 1 footnotetext: These authors contributed equally to this work. 2 2 footnotetext: Corresponding author. E-mail: [email protected] Keywords: social perception \mid intuitive physics \mid intuitive psychology \mid inverse planning 1 Introduction Human social perception is richly grounded in our understanding of physics. As illustrated in Figure 1 A , when we infer that one agent is chasing, blocking, or helping another, we rely not only on intuitive psychology (e.g., agents act purposefully) but also on intuitive physics (e.g., bodies move continuously, forces have effects, and obstacles constrain what is possible). Classic demonstrations such as the Heider and Simmel movie ( Heider and Simmel 1944 ) show that even movements of simple geometric shapes can elicit rich social narratives (as depicted in Figure 1 B ). Critically, those anthropomorphic narratives of social behavior rely on viewers’ interpretation of the physical interactions (e.g., contact, collision, and impeded motion). This raises a core cognitive science question: how do people infer rich social information from physically grounded observations? There have been many computational accounts of social perception that capture aspects of rational action understanding. One prominent type of computational approaches is inverse planning ( Baker et al. 2009 ; Baker et al. 2017 ; Jara-Ettinger et al. 2020 ; Ying et al. 2025 ) , which posits that people assume others are rational planners that select the most optimal action to achieve their goal, minimizing costs and maximizing utilities. The observer can then invert this generative model to compute the most likely mental states that can best explain the observed action. However, existing models of inverse planning ( Baker et al. 2009 ; Baker et al. 2017 ; Jara-Ettinger et al. 2016 ; Ullman et al. 2009 ; Zhi-Xuan et al. 2020 ) often abstract away physical dynamics, treating actions as symbolic state transitions. Conversely, recent end-to-end neural models have emerged as powerful approaches to model social perception by training on large corpora of diverse social scenes ( Rabinowitz et al. 2018 ; Hovaidi-Ardestani et al. 2018 ; Malik and Isik 2023 ) . Although these models can learn predictive representations from large corpora, they typically rely on visual features and also do not explicitly show sophisticated physical reasoning processes. In this paper, we study how people flexibly interpret social scenes in physically grounded environments. We hypothesize that people can utilize an analysis-by-synthesis approach ( Yuille and Kersten 2006 ) on these tasks: we possess a joint physical and social generative model of agents planning in our minds and invert this generative process to infer the most likely goal, relations, and physical parameters that best explain the observation. This view builds on prior work on infants’ judgment of agents’ goal-directed actions in physical environments ( Liu et al. 2017 ; Liu et al. 2022 ; Saxe et al. 2005 ) . As prior work has demonstrated, people, starting from as early as 10 months old, can estimate the costs of actions for pursuing goals under various physical constraints, and reason about cost-utility trade-offs to interpret their behavior. Recent studies in neuroscience and computational cognitive science have also provided evidence that people’s intuitive theory of psychology builds on their intuitive theory of physics ( Liu et al. 2025 ) . In this work, we extend the prior work and operationalize this view with a dataset and a computational model. Figure 1 : ( A ) Examples of real-life social interactions in physical environments: a basketball player trying to block an opponent from shooting a ball; two people carrying a couch together. ( B ) The classic Heider-Simmel animation abstracts such real-life interactions in animated displays of simple geometric shapes. ( C ) PHASE models social interactions as physically grounded abstract events, where animated agents with limited fields of view move in a physics-based environment with objects, landmarks, and obstacles, enabling behaviors such as helping, hindering, or collaborating toward goals. Our main contributions are twofold. First, we introduce PHASE (PHysically grounded Abstract Social Events), a dataset of physically simulated two-agent interactions with controlled variation in scene geometry, physical dynamics, agent capacities, and social structure (goals and relationships), as illustrated Figure 1 C . Second, we propose SIMPLE (SIMulation, Planning, and Local Estimation), a physics-grounded Bayesian inverse planning model that combines a forward physics engine with probabilistic planning and inference over goals and relations. We show that incorporating physical dynamics improves goal and relationship inference relative to physics-agnostic inverse planning and end-to-end video baselines, and that the resulting inferences better align with human judgments. 2 A Paradigm for Physically Grounded Social Perception Figure 2 : Example PHASE animations for different scenario types. A). In this collaborative scenario, agents collaborate to move the pink circle to the yellow landmark. The green agent is too large to move past the barrier. Therefore, the red agent first moves the circle to the left of the barrier and then carries it together with the green agent to the yellow landmark. B). Two agents move towards different landmarks with no interaction. C). In this competing scenario, the agents want to move the blue circle to different landmarks, but there is only one blue circle. They therefore pull the circle in different directions. D). In this opposing scenario, the red agent tries to chase the green agent, while the green agent tries to get away from the red agent. The videos for these examples can be found at https://osf.io/fkp5m/ . Inspired by the seminal work by Heider and Simmel ( Heider and Simmel 1944 ) , we curated the PHASE dataset to study human social perception in abstract physically grounded scenes. The videos of the PHASE dataset feature two agents in the shape of trapezoids. The agents can vary in their sizes and

Read the complete text