HEIR: Learning Graph-Based Motion Hierarchies
Cheng Zheng, William Koch, Baiang Li, Felix Heide · 2025
Read HEIR: Learning Graph-Based Motion Hierarchies by Cheng Zheng, William Koch, Baiang Li, Felix Heide (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26786.txt.
Introduction
Cheng Zheng, William Koch, Baiang Li, and Felix Heide teach computers to learn how motion is organised in layers. Complex dynamics, from a swinging arm to a deforming scene, arise from simpler motion parts acting together. Existing methods rely on hand-defined hierarchies with fixed motion pieces, which limits how widely they apply.
Zheng and his three co-authors present HEIR, which learns a graph-based hierarchy straight from data. Observed motion is split into patterns inherited from a parent motion plus local residuals. Vertices hold elemental motions, and directed edges, learned by graph neural networks, hold parent-child links, all in a differentiable setup. Tests on one-dimensional translation, two-dimensional rotation, and dynamic three-dimensional scenes rebuilt the true hierarchy in the simple cases. On three-dimensional scenes, the deformations looked more realistic and interpretable than the baseline's.
Catalog record
| CREATOR | Cheng Zheng, William Koch, Baiang Li, Felix Heide |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26786
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HEIR: Learning Graph-Based Motion Hierarchies Cheng Zheng, William Koch, Baiang Li, Felix Heide Abstract Hierarchical structures of motion exist across research fields, including computer vision, graphics, and robotics, where complex dynamics typically arise from coordinated interactions among simpler motion components. Existing methods to model such dynamics typically rely on manually-defined or heuristic hierarchies with fixed motion primitives, limiting their generalizability across different tasks. In this work, we propose a general hierarchical motion modeling method that learns structured, interpretable motion relationships directly from data. Our method represents observed motions using graph-based hierarchies, explicitly decomposing global absolute motions into parent-inherited patterns and local motion residuals. We formulate hierarchy inference as a differentiable graph learning problem, where vertices represent elemental motions and directed edges capture learned parent-child dependencies through graph neural networks. We evaluate our hierarchical reconstruction approach on three examples: 1D translational motion, 2D rotational motion, and dynamic 3D scene deformation via Gaussian splatting. Experimental results show that our method reconstructs the intrinsic motion hierarchy in 1D and 2D cases, and produces more realistic and interpretable deformations compared to the baseline on dynamic 3D Gaussian splatting scenes. By providing an adaptable, data-driven hierarchical modeling paradigm, our method offers a formulation applicable to a broad range of motion-centric tasks. Project Page: https://light.princeton.edu/HEIR/ 1 Introduction Many natural and artificial motion systems comprise simple motion primitives that coordinate to produce complex behaviors. Understanding these structured relationships — commonly called motion hierarchies — is fundamental across multiple research areas, from action recognition in computer vision gaidon2013activity to verifiable prediction in robotics benton2024verifiable ; takano2011prediction . By capturing multi - scale dependencies, hierarchical models tame the combinatorial explosion when generating, predicting, or controlling motion. In computer vision , hierarchical representations decompose raw trajectories into semantically meaningful layers. Early work introduced fixed hierarchies for activity recognition and motion capture gaidon2013activity ; benton2024verifiable ; takano2011prediction , while more recent methods learn multi - level encodings that bridge low - level displacements and high - level actions kulal2021hierarchical ; leong2021joint . These approaches identify repeated motion patterns and enable reasoning over sub - actions, improving tasks like video generation and pose estimation. In robotics , hierarchical structures guide both planning and execution. Global objectives — such as navigating to a goal — are decomposed into coordinated limb or joint movements via modular controllers. Hierarchical reinforcement learning architectures separately optimize high - level navigation and local motor policies jain2019hierarchical , and modular schemes coordinate limb - specific controllers under whole - body objectives yuan2023hierarchical . This multi - scale decomposition enhances adaptability and robustness in dynamic environments. Recent advances move beyond manually defined templates toward data - driven hierarchy discovery. Hierarchical VAEs introduce multi - level latent spaces that explicitly model coarse global trajectories and fine local adjustments li2021task . Diffusion - based frameworks employ semantic graphs to generate motions at successive abstraction levels, offering fine control without hand - crafted priors jin2023act . These methods aim to learn interpretable, generalizable motion representations directly from data, closing the gap between fixed kinematic models and the rich variability of real - world behaviors. Despite differences in objectives and formulations, these research areas still face common technical challenges in modeling motion hierarchies. Most approaches represent motion structures either with hand-crafted heuristics or non-interpretable neural modules liang2025himor ; tao2015moving ; xu2020hierarchical ; kulal2021hierarchical , making them difficult to transfer across tasks. These models also struggle to find the appropriate coarseness, i.e., balancing the granularity of motion primitives against the expressiveness needed for the task. As such, automatically discovering interpretable, transferable representations of motion hierarchies is an open challenge, requiring adaptively selecting the right level of abstraction for diverse downstream applications. To address these challenges, we propose a general-purpose hierarchical motion modeling method that learns interpretable, structured motion relationships through a graph-based network. We introduce a learnable motion graph, where vertices correspond to subsets of discrete motion elements (e.g., Gaussian splats, tracked keypoints), and directed edges define a learned hierarchy of motion dependencies. To enable this, we learn edge weights on a proximity graph to infer parent-child relationships: the resulting edge weights define the parent probability distribution for any motion element, from which we sample a discrete motion hierarchy. This allows us to model both global constraints and local variations in a unified, data-driven way. We validate the generality and effectiveness of our method on three challenging tasks: two synthetic benchmarks involving structured point motion to show expressiveness (one translational and one rotational), as well as dynamic 3D Gaussian splatting for high-fidelity scene deformation. In all settings, the proposed approach effectively captures and exploits the underlying hierarchical structures. Across representative dynamic scenes, our method achieves improvements over existing methods for all scenarios in scene deformation for pixel error and perceptual metrics, highlighting its effectiveness in producing perceptually and structurally faithful deformations. 2 Related Work We review relevant work in two areas: motion representations and 3D scene deformation. Hierarchical Motion Representation and Decomposition. Motion representations capture structure in dynamic systems to support analysis, generation, and control. We existing works have explored explicit or implicit motion models. Conventional explicit approaches impose predefined structure on motion to improve control, analysis or generation. In humanoid settings, skeletons define natural hierarchical relationships which have been used for various downstream tasks such as SDF learning or motion re-targeting biswas2021hierarchical . For video understanding, motion programs kulal2021hierarchical represent human behavior as sequences of symbolic motion primitives. The FineGym dataset shao2020finegym takes another approach and develops a three-layer semantic hierarchy of pre-defined actions, which has been shown to be useful in action recognition shao2020finegym ; leong2021joint . These methods are largely limited to a domain, however, as defining an explicit structure relies on domain-specific assumptions. On the implicit side, unsupervised methods such as spectral clustering gaidon2013activity and Moving Poselets tao2015moving learn task-specific motion patterns that improve recognition without relying on predefined semantics. More closely related to our work, in physics-based settings, relationships emerge by learning how entities interact with each other grigorev2022hood ; sanchezgonzalez2020learning . Similarly, Neural Relational Inference pmlr-v80-kipf18a recovers the underlying interaction graph even when the connectivity is unknown a priori. At the intersection, we find methods that rely on explicit structures, but learn the relevant components. MovingParts yang2023movingparts factorizes dynamic NeRF scenes into rigidly moving segments for part-level editing, and an unsupervised video approach clusters pixels by principal motion axes to animate articulated objects siarohin2021motion . Both recover meaningful parts but also do not infer parent–child dependencies, leaving the full hierarchy unspecified. Unlike prior work, we do not assume any prior domain , dimensionality or naturally occurring structures in the data. Our method is capable of decomposing the motion on its own - at the same time, it provides an interpretable and controllable structure to the motion. 3D Scene Deformation and Editing. Traditional methods for scene deformation explicitly define geometric transformations using mesh-based deformation energies or cage-based constraints. Approaches such as Laplacian coordinates sorkine2005laplacian ; sorkine2007rigid preserve local geometric details by formulating deformation as energy minimization. Cage-based methods nieto2012cage ; ju2008reusable ; yifan2020neural encapsulate an object within a coarse control mesh, allowing intuitive deformation through sparse user manipulation. Although effective for explicit geometric editing, these methods typically assume structured mesh representations. Recent progress in 3D scene deformation and editing has