Target-Aware Video Diffusion Models

Taeksoo Kim, Hanbyul Joo · 2025

Read Target-Aware Video Diffusion Models by Taeksoo Kim, Hanbyul Joo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18950.txt.

Introduction

Taeksoo Kim and Hanbyul Joo published a computing paper in 2025 on video generation where an actor acts on a named target. The input is a still image, a mask marking the target, and a text prompt naming the action. Past video models lacked this target awareness. The team extends a base model to take the mask as an extra input.

A special token carries the target's place data inside the prompt. Training used a curated dataset plus an added loss that aligns attention maps with the mask, applied to the most relevant spots. Tests show the model beats rivals at making actors interact with the right target. Uses shown include 3D motion synthesis and long video creation.

Catalog record

CREATORTaeksoo Kim, Hanbyul Joo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18950

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Target-Aware Video Diffusion Models Taeksoo Kim, Hanbyul Joo Abstract We present a target-aware video diffusion model that generates videos from an input image, in which an actor interacts with a specified target while performing a desired action. The target is defined by a segmentation mask, and the action is described through a text prompt. Our key motivation is to incorporate target awareness into video generation, enabling actors to perform directed actions on designated objects. This enables video diffusion models to act as motion planners, producing plausible predictions of human-object interactions by leveraging the priors of large-scale video generative models. We build our target-aware model by extending a baseline model to incorporate the target mask as an additional input. To enforce target awareness, we introduce a special token that encodes the target’s spatial information within the text prompt. We then fine-tune the model with our curated dataset using an additional cross-attention loss that aligns the cross-attention maps associated with this token with the input target mask. To further improve performance, we selectively apply this loss to the most semantically relevant attention regions and transformer blocks. Experimental results show that our target-aware model outperforms existing solutions in generating videos where actors interact accurately with the specified targets. We further demonstrate its efficacy in two downstream applications: zero-shot 3D HOI motion synthesis with physical plausibility and long-term video content creation. † † footnotetext: Project page: https://taeksuu.github.io/tavid/ 1 Introduction Video diffusion models have demonstrated remarkable capabilities in simulating complex real-world scenes. Ideally, such models can serve as motion planners, in line with the concept of world models ( Ha & Schmidhuber, 2018 ; Bar et al., 2024 ; Kim et al., 2025a ) , by producing plausible predictions of interactions between an actor (human or robot) and target objects using priors learned from large-scale video datasets. However, existing image-to-video diffusion models ( Yang et al., 2025b ; Kong et al., 2024 ; HaCohen et al., 2024 ) , which generate videos from an input image guided by text prompts, are target-unaware. An alternative line of work attempts to explicitly control actor-target interactions using dense structural or motion cues, such as depth maps ( Esser et al., 2023 ; Zhang et al., 2024 ) , edges ( Chen et al., 2023 ; Khachatryan et al., 2023 ) , optical flow ( Ni et al., 2023 ; Burgert et al., 2025 ; Gu et al., 2025 ) , motion trajectories ( Yan et al., 2023 ; Shi et al., 2024a ) , or drag-based manipulation ( Deng et al., 2024 ; Teng et al., 2023 ; Shi et al., 2024b ) . While effective for certain tasks, these approaches do not meet our needs. Our goal is to use video diffusion models to infer plausible actor-target interactions, where action guidance for the actor is not available in advance and thus cannot be provided as input. Ultimately, we aim to leverage video generative models for high-level action planning, inferring realistic interaction cues for the actor within the current scene, as explored in recent robotics research ( Black et al., 2024 ; Du et al., 2023 ; Ajay et al., 2023 ; Ni et al., 2024 ) . In this paper, we present a target-aware video diffusion model that generates videos from an input image, where an actor performs a desired action directed at a specified target. The target is defined by a segmentation mask, and the action is described with a text prompt. We use the mask as a means to specify the target object in the scene, which can be obtained with minimal effort (e.g., a single click), or automatically from text input using off-the-shelf tools ( Ren et al., 2024 ) , and we show that our method is robust to variations in mask quality. By providing an explicit way to designate the target, our model can serve as an effective motion planner, enabling the actor to perform diverse interactions with the specified object. While our training uses videos with human actors, we also demonstrate that the model generalizes seamlessly to other agents, including animals and robotic hands. To integrate spatial information of the target mask, we extend a base image-to-video diffusion model ( Yang et al., 2025b ) to take the mask as an additional input, and fine-tune it on our newly curated dataset. However, simply fine-tuning the model with the extra mask input does not ensure the target awareness of the model. To address this, we introduce a special token, [TGT], into the text prompt to describe the target and enforce an alignment between the [TGT] token’s cross-attention maps and the input target mask by applying a loss on the model’s cross-attention. This cross-attention loss enables the model to associate the [TGT] token with the spatial information of the target, improving the precision of the generated interactions with the target by injecting spatial grounding into the text-conditioning mechanism of the model. We selectively apply this loss to specific attention regions and transformer blocks that are most semantically relevant for effective supervision. Experimental results demonstrate that our target-aware video diffusion model outperforms existing solutions in synthesizing videos where actors precisely engage with designated targets. To further demonstrate the strength of our target-aware model, we apply our model to two downstream applications: (1) zero-shot 3D human-object interaction (HOI) motion synthesis with physical plausibility, simulating physical agents performing plausible actions in a given environment, and (2) video content creation, generating long-term videos covering navigations and interactions with minimal user input. Our contributions can be summarized as follows: (1) We present a target-aware video diffusion model that generates videos of interactions between the actor and the target using a segmentation mask and a text prompt; (2) We propose to utilize a cross-attention loss to enable the base model to effectively incorporate the mask input and achieve target awareness, and provide a comprehensive analysis of its effects across different parts of the model; (3) We present a newly curated dataset specifically designed to train and evaluate our target-aware model; and (4) We demonstrate two real-world applications of our target-aware model: zero-shot 3D HOI motion synthesis for controlling physical agents and video content creation. 2 Related Work Controllable Video Generation Building on early work ( Ho et al., 2022 ; Esser et al., 2023 ; Guo et al., 2024 ) that extends text-to-image diffusion models to video generation, the community has shown significant interest in producing videos with enhanced controls. Several methods, inspired by ControlNet ( Zhang et al., 2023b ) , have been adapted for videos, where structural cues, such as depth maps ( Esser et al., 2023 ; Zhang et al., 2024 ) , edge information ( Chen et al., 2023 ; Khachatryan et al., 2023 ) , optical flow ( Ni et al., 2023 ; Gu et al., 2025 ) , or motion ( Yan et al., 2023 ; Shi et al., 2024a ; Cha et al., 2026 ) , are integrated into the generation process via additional modules to produce structure-consistent outputs. Another line of research focuses on manipulating the internal representations of diffusion models to achieve controls without extra modules. Attention modulation approaches ( Yang et al., 2024 ; Wu et al., 2024a ;

Read the complete text