Populate-A-Scene: Affordance-Aware Human Video Generation

Mengyi Shan, Zecheng He, Haoyu Ma, Felix Juefei-Xu, Peizhao Zhang, Tingbo Hou, Ching-Yao Chuang · 2025

Read Populate-A-Scene: Affordance-Aware Human Video Generation by Mengyi Shan, Zecheng He, Haoyu Ma, Felix Juefei-Xu, Peizhao Zhang, Tingbo Hou, Ching-Yao Chuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00334.txt.

Introduction

Mengyi Shan and six co-authors teach a video generator to place people into scenes sensibly, in a 2025 paper. Given one image of a place and a prompt describing an action, their fine-tuned model inserts a person. The figure behaves as the place invites: sitting on chairs, stepping through doors, not floating over tables.

Earlier systems needed explicit boxes or poses to place figures. Here the model must infer affordances, the action possibilities a scene offers, on its own. Studying the model's attention maps revealed that a pre-trained video model already carries an implicit sense of where people belong, which fine-tuning sharpens. The authors ask whether video generators could double as interactive world simulators. A model that understands where actions fit is a step from generating clips toward simulating places.

Catalog record

CREATORMengyi Shan, Zecheng He, Haoyu Ma, Felix Juefei-Xu, Peizhao Zhang, Tingbo Hou, Ching-Yao Chuang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00334

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Populate-A-Scene: Affordance-Aware Human Video Generation Mengyi Shan, Zecheng He, Haoyu Ma, Felix Juefei-Xu, Peizhao Zhang, Tingbo Hou, Ching-Yao Chuang Abstract Can a video generation model be repurposed as an interactive world simulator? We explore the affordance perception potential of text-to-video models by teaching them to predict human-environment interaction. Given a scene image and a prompt describing human actions, we fine-tune the model to insert a person into the scene, while ensuring coherent behavior, appearance, harmonization, and scene affordance. Unlike prior work, we infer human affordance for video generation (i.e., where to insert a person and how they should behave) from a single scene image, without explicit conditions like bounding boxes or body poses. An in-depth study of cross-attention heatmaps demonstrates that we can uncover the inherent affordance perception of a pre-trained video model without labeled affordance datasets. † † affiliation: GenAI at Meta † † affiliation: University of Washington † † contribution: Work done at Meta † † date: August 8, 2026 † † correspondence: Mengyi Shan at [email protected] Figure 1: We repurpose a text-to-video generation model as a human-world interaction simulator. Given a scene image and a prompt, our model inserts a person into the environment and generates a video of them naturally interacting with the scene. The scene can be real images (top) or synthesized by image generative models (bottom). Notably, there is no need for any mask, location bounding boxes, or pose sequences to guide the human insertion – our method takes care of affordance prediction entirely within the video model. 1 Introduction Scaling data, compute, and model parameters in video generation models presents a promising avenue for developing highly capable simulators that can accurately replicate complex physical worlds ( Polyak et al. 2024 ; Brooks et al. 2024b ) , complete with diverse objects and people that interact and coexist within them. Nevertheless, humans are not merely passive observers, but rather active participants in the world. Human understanding of affordance Koffka 1999 ; Gibson 1996 ; Norman 2013 enables purposeful engagement with surroundings and adaptive behavior by recognizing potential actions afforded by an object’s physical properties. It remains unclear whether video generation models can interpret and replicate intricate semantic aspects of the world, such as contextual understanding and dynamic behavior, beyond the capabilities of traditional graphics pipelines. Affordance, or “opportunities for interaction” ( Gibson 1996 ) , has inspired extensive research in vision and psychology. Traditional affordance prediction relies on data-driven approaches using 3D information ( Hassan et al. 2021 ) , specifically labeled datasets ( Wang et al. 2017 ; Gupta et al. 2011 ; Fouhey et al. 2012 ; Delaitre et al. 2012 ; Chen et al. 2023 ) , or one-shot large foundational models ( Li et al. 2024 ) . However, these methods rely on domain-specific annotations, which are challenging to obtain. In contrast, recent advancements in generative models offer the potential to create realistic human-scene media content using vast amounts of in-the-wild media data. ( Kulal et al. 2023 ) , for example, predicts a human’s pose and appearance in a scene but is restricted to static images with a given position mask. In this work, we demonstrate that throughout the intricate process of video generation, the model learns to generate human activities and motions that adhere to the affordance constraints dictated by the physical world. To better study affordance modeling, we propose augmenting a pre-trained text-to-video model Polyak et al. 2024 with an additional scene conditioning branch. This modification formulates the problem as a conditional video generation task: given a scene represented by an image, the model is tasked with introducing natural human motion and interactions to the scene. We discover that pre-trained video generation models can rapidly adapt to this new task by fine-tuning on a relatively small-scale scene conditioning dataset. We then validate the affordance perception abilities through an extensive study of the cross-attention feature heatmaps, a key module that enables the model to follow language prompts. Unlike prior work, our model does not require input masks, bounding boxes, or pose sequences to specify regions or patterns of human behavior, which makes it an interaction simulator that reasons about semantics and affordance properties in the scene, instead of merely a human renderer that turns given pose signals into pixels. During inference, the model can process a wide array of environment-action combinations to generate diverse interactive videos, not limited to interaction with the single, salient object in a complicated scene. Fig. 1 demonstrates results of our model without aggressive cherry-picking. In particular, the last row of Fig. 1 illustrates a ”movie studio” pipeline where input scenes are generated using a text-to-image model ( Dai et al. 2023 ) , and our model seamlessly integrates actors into these scenes without requiring 3D capture. Our results lower the barrier for amateur AI video creators by eliminating the need for explicit body poses signals, as they are required in most AI human video models but challenging to synthesize. In short, this work makes the following contributions: • We address affordance-aware human video generation, where we generate video of subject(s) interacting with a given environment image, without telling the model where the subject(s) are and how their poses look like. • We apply the dual-stream conditioning mechanism with a minimal grounding module to model affordance, and thus reveal the affordance capabilities of video generation models through in-depth analysis. • We demonstrate our model’s ability to generalize across diverse environments and actions through a synthetic benchmark created with vision-language models. 2 Related Works Text-to-video generative models. Text-to-video generation aims to synthesize plausible, temporally coherent, and optionally condition-aligned video sequences from textual prompts. Recent rapid advancements in text-to-video models have been phenomenal Ho et al. 2022 ; Singer et al. 2022 ; Ge et al. 2024 ; Blattmann et al. 2023 ; Brooks et al. 2024a ; Polyak et al. 2024 ; Wang et al. 2023a ; Bar-Tal et al. 2024 ; Chen et al. 2024 ; Esser et al. 2023 ; He et al. 2022 . Current work explores replacing the traditional U-Net with a Transformer Vaswani 2017 architecture Gupta et al. 2023 ; Ma et al. 2024 ; Brooks et al. 2024a ; Polyak et al. 2024 , inspired by the promising text-to-image generative results from DiT Peebles and Xie 2022 . We start from a pre-trained Transformer-based text-to-video model Movie Gen Polyak et al. 2024 and explore its ability to perceive affordance through minimal fine-tuning on human-scene interaction data. Some text-to-video tasks augment the model with an image as a starting frame and use prompts to describe the style or motion in the video Zeng et al. 2023 ; Gong et al. 2024 ; Ren et al. 2024 . Our task differs in that we give the model an empty scene frame that is not supposed to appear in the video, but provides a “playground” for population. Human video generation. Human video generation evolves alongside rapidly advancing generic video generative models. Generating realistic human content is inherently challenging due to complex body topology, strong priors on interaction plausibility, and audiences’ sensitivity to even minor artifacts. Existing methods use motion guidance to improve video faithfulness, leveraging signals such as OpenPose Hu et al.

Read the complete text