HECTOR: Hybrid Editable Compositional Object References for Video Generation
Guofeng Zhang, Angtian Wang, Jacob Zhiyuan Fang, Liming Jiang, Haotian Yang, Alan Yuille, Chongyang Ma · 2026
Read HECTOR: Hybrid Editable Compositional Object References for Video Generation by Guofeng Zhang, Angtian Wang, Jacob Zhiyuan Fang, Liming Jiang, Haotian Yang, Alan Yuille, Chongyang Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08850.txt.
Introduction
Guofeng Zhang and colleagues give video generation fine control over each object's path. Most models paint whole scenes at once, with no levers for composition. Their HECTOR pipeline mixes still-image and video references, lets users set each element's trajectory, place, size, and speed, and beats rivals on visual quality, reference faithfulness, and motion control.
Catalog record
| CREATOR | Guofeng Zhang, Angtian Wang, Jacob Zhiyuan Fang, Liming Jiang, Haotian Yang, Alan Yuille, Chongyang Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40344 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08850
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HECTOR: Hybrid Editable Compositional Object References for Video Generation Guofeng Zhang, Angtian Wang, Jacob Zhiyuan Fang, Liming Jiang, Haotian Yang, Alan Yuille, Chongyang Ma Abstract Real-world videos naturally portray complex interactions among distinct physical objects, effectively forming dynamic compositions of visual elements. However, most current video generation models synthesize scenes holistically and therefore lack mechanisms for explicit compositional manipulation. To address this limitation, we propose HECTOR, a generative pipeline that enables fine-grained compositional control. In contrast to prior methods, HECTOR supports hybrid reference conditioning, allowing generation to be simultaneously guided by static images and/or dynamic videos. Moreover, users can explicitly specify the trajectory of each referenced element, precisely controlling its location, scale, and speed (see Figure 1 ). This design allows the model to synthesize coherent videos that satisfy complex spatiotemporal constraints while preserving high-fidelity adherence to references. Extensive experiments demonstrate that HECTOR achieves superior visual quality, stronger reference preservation, and improved motion controllability compared with existing approaches. † † affiliationnotice: This work was done while Guofeng Zhang was an intern at ByteDance. Angtian Wang is the Project Lead. Figure 1 : We propose HECTOR, a compositional, reference-guided video generation architecture. HECTOR supports conditioning on heterogeneous reference inputs (static images and/or dynamic videos) while enabling precise control over each referenced element’s location, scale, and speed. Beyond that, HECTOR also accommodates diverse operations, including multi-object composition, camera-motion control (e.g., zoom-in/zoom-out), and reference-driven video editing such as object insertion, replacement as shown in the above. 1 Introduction The recent proliferation of diffusion-based generative models has revolutionized video synthesis, with Text-to-Video (T2V) ( 20 ; 33 ; 40 ; 29 ) and Image-to-Video (I2V) ( 3 ; 47 ; 2 ; 29 ) paradigms enabling the creation of high-fidelity dynamic content for diverse applications, from entertainment to content creation. Despite these advancements, the practical utility in professional settings remains constrained by a lack of precise controllability. Standard approaches typically generate scenes holistically, where the user provides a high-level prompt but lacks the agency to dictate specific object behaviors or interactions. To address this limitation, recent research has begun to explore fine-grained control mechanisms. Methods including Motion Prompting ( 9 ) , Tora ( 49 ) , TGT ( 45 ) , ATI ( 31 ) , and Wan-Move ( 5 ) have demonstrated the ability to guide motion through trajectories. However, these approaches often operate on video as a single entity. In this work, we push this exploration a step further by asking: Can we generate video in a fundamentally compositional manner? By decomposing a scene into distinct visual references, we aim to grant users explicit control over the appearance and motion of each individual component, including background, within the generated video. Recent works have explored this along two main fronts. The first category focuses on instance-level customization, where methods such as DreamVideo ( 34 ) and MotionBooth ( 36 ) enable a single object to be localized via bounding box constraints. While effective for maintaining identity, these approaches typically rely on test time optimization for each specific reference, a process that is computationally expensive and difficult to scale to complex scenes with multiple interacting references. A second line of research finetunes existing video generation models to integrate control signals. Notable examples include Tora2 ( 50 ) , DreamVideo2 ( 35 ) and VACE ( 16 ) . These method integrate various conditional inputs in to existing models without requiring costly test-time optimization. While more efficient, these models face inherent difficulties in achieving true compositionality. In particular, they often exhibit degraded performance when processing multiple entities, frequently struggling to maintain precise boundaries or identity consistency as the scene complexity increases. Furthermore, these approaches lack explicit support for independent background conditioning and dynamic video references. While they can preserve identity from a static image, they are not designed to ingest video priors to maintain both the identity and the specific gestures of a subject. To bridge this gap, we introduce Hybrid Editable Compositional Object References (HECTOR), a framework supporting both image and video-based references. To handle data, we adopt the Video Decompositor, which serves as both a curation and inference engine. Departing from rigid bounding box heuristics, it aggregates tracking points to derive precise motion paths and scales. This ensures superior temporal smoothness and stability, maintaining target consistency even during challenging occlusion events. We adapt a DiT-based architecture to inject these signals via the Spatio-Temporal Alignment Module (STAM). STAM encodes diverse references into VAE latents, spatially organizing them to align strictly with trajectory-defined locations and timesteps. Specifically, it fuses image-based latents (for identity) and video-based latents (for gestures) into a unified conditioning tensor, gated by confidence masks and concatenated with the latent noise. This mechanism effectively binds visual identities and motion priors to precise spatial regions, enabling genuine compositional generation of coherent foreground and background entities. Extensive experiments demonstrate that HECTOR generates video with strict identity and structural consistency. Uniquely, the framework supports dynamic object entry and exit without disrupting global temporal flow. Beyond generation, HECTOR unlocks powerful editing capabilities, including high-fidelity object replacement, addition, and background modification. By decoupling identity from motion, it further enables localized manipulations—such as altering an entity’s speed or scale—while preserving the integrity of the surrounding scene (see Figure 1 ). In summary, our contributions are as follows: • We propose HECTOR, the first framework for fully compositional video generation. It enables precise, independent control over each element. • We introduce the Spatio-Temporal Alignment Module (STAM), which processes both static and dynamic references spatially and temporally in the latent space. • We present the Video Decompositor, a mechanism that automatically extracts compositional structures from video data, supporting both accurate training data curation and flexible video editing during inference. 2 Related Works Foundational video generation model. Recent video generation advancements are driven by scaling diffusion transformers (DiTs), which enable high-fidelity, temporally coherent synthesis. Early latent-based approaches ( 3 ; 4 ) democratized high-resolution generation, while leading open-source models ( 30 ; 18 ; 41 ) have established robust baselines for motion and realism. Recent works extend these foundations to complex tasks like long video generation ( 21 ; 46 ; 13 ) and video editing ( 6 ; 12 ) . However, these frameworks typically generate scenes holistically, leaving fine-grained control and spatial compositionality largely underexplored. Reference-based video customization. Reference-based models synthesize videos conditioned on specific subjects, primarily categorized into tuning-based ( 34 ; 37 ) and training-free ( 44 ; 51 ) methods. Similarly, I2V models ( 39 ; 11 ) and multi-concept frameworks ( 15 ; 7 ; 8 ) focus on animating static reference images. However, these methods generally restrict users to static inputs and lack granular dynamic control. In contrast, our approach uniquely supports dynamic video references to integrate complex motion priors. By providing explicit guidance for reference motion and action, we bridge the gap between identity preservation and precise spatiotemporal controllability. Trajectory-controlled video generation. Structure-based methods impose spatial layouts using bounding boxes or masks for precise alignment ( 22 ; 23 ; 14 ) , but these signals are rigid and labor-intensive. Zero-shot alternatives ( 43 ; 42 ) reduce manual effort but often suffer from degraded controllability or quality ( 28 ) . Alternatively,