Rodrigues Network for Learning Robot Actions
Jialiang ZhangPeking University, Haoran GengUC Berkeley, Yang YouStanford UniversityCongyue DengStanford UniversityPieter AbbeelUC BerkeleyJitendra MalikUC BerkeleyLeonidas GuibasStanford University · 2025
Read Rodrigues Network for Learning Robot Actions by Jialiang ZhangPeking University, Haoran GengUC Berkeley, Yang YouStanford UniversityCongyue DengStanford UniversityPieter AbbeelUC BerkeleyJitendra MalikUC BerkeleyLeonidas GuibasStanford University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02618.txt.
Introduction
Jialiang Zhang and co-authors design a robot learning network in 2025. Robots must grasp how jointed bodies move, yet common networks ignore that built in structure. The team built a learnable version of the classic forward kinematics step. Their operator bakes motion structure into the network itself.
The full network, called RodriNet, is tuned for action data. On synthetic motion tasks it beat standard backbones by a wide margin. It also lifted imitation learning on robot benchmarks when paired with a diffusion policy. A single image hand rebuilding task showed gains as well. Structure aware design may help robots learn moves faster.
Catalog record
| CREATOR | Jialiang ZhangPeking University, Haoran GengUC Berkeley, Yang YouStanford UniversityCongyue DengStanford UniversityPieter AbbeelUC BerkeleyJitendra MalikUC BerkeleyLeonidas GuibasStanford University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02618
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rodrigues Network for Learning Robot Actions Jialiang ZhangPeking University, Haoran GengUC Berkeley, Yang YouStanford UniversityCongyue DengStanford UniversityPieter AbbeelUC BerkeleyJitendra MalikUC BerkeleyLeonidas GuibasStanford University Abstract Understanding and predicting articulated actions is important in robot learning. However, common architectures such as MLPs and Transformers lack inductive biases that reflect the underlying kinematic structure of articulated systems. To this end, we propose the Neural Rodrigues Operator , a learnable generalization of the classical forward kinematics operation, designed to inject kinematics-aware inductive bias into neural computation. Building on this operator, we design the Rodrigues Network (RodriNet) , a novel neural architecture specialized for processing actions. We evaluate the expressivity of our network on two synthetic tasks on kinematic and motion prediction, showing significant improvements compared to standard backbones. We further demonstrate its effectiveness in two realistic applications: (i) imitation learning on robotic benchmarks with the Diffusion Policy, and (ii) single-image 3D hand reconstruction. Our results suggest that integrating structured kinematic priors into the network architecture improves action learning in various domains. * * footnotetext: Equal contribution $\ddagger$ $\ddagger$ footnotetext: Work performed while visiting Stanford University. Figure 1 : We introduce the Neural Rodrigues Operator , a learnable extension of the classical Rodrigues’ Rotation Formula from robot control, where the original coefficients are replaced with trainable weights and joint angles are generalized to abstract features. Built upon this operator, the Rodrigues Network leverages the kinematic structure of articulated systems to advance a wide range of action-learning tasks. 1 Introduction We study the problem of understanding and predicting the actions of articulated actors. Articulated actors ( e.g. robots [ 1 ] and animated characters [ 40 , 31 ] ) are systems that use multiple rotational joints to connect moving links. Their actions, including poses, motions, or control commands, are usually represented as values associated with all joints. Learning with articulated actors usually involves predicting their actions while processing diverse sensory inputs. Such a problem lies in a wide spectrum of intelligent systems, from whole-body controllers [ 34 , 27 ] , to dexterous grasp detectors [ 18 , 53 , 50 , 56 ] , to motion retargetting networks [ 2 , 20 ] . To learn the actions of articulated actors, we seek a neural network architecture that can both understand robot structures and enforce structural constraints in its predictions. However, most existing network architectures, such as MLPs and Transformers [ 48 ] , treat actions as unstructured tokens, ignoring the kinematic relationships that govern joint interactions. A few prior works have incorporated structural priors like robot link connectivity, using graph convolutions [ 54 , 16 , 2 ] or masked attention [ 41 ] . Yet, the design of network architectures that fully capture and enforce the articulated kinematic structure of these actors remains underexplored. This leads us to our central question: Can we design a neural network for action learning that naturally integrates articulated kinematics as an inductive bias in its architecture? To answer this question, we reflect on how CNNs [ 26 ] led to a deep learning framework that achieved remarkable success in computer vision by generalizing classical convolution filters from image processing [ 6 ] . Convolution filters naturally respect image structure through translation equivariance and spatial locality, but when used with handcrafted features, they were limited to low-level image processing tasks. Building on this foundation, CNNs extend these filters into neural operators with optimizable weights, non-linearities, and high-dimensional feature channels. This results in a deep learning framework capable of learning high-level semantic features from images while preserving the key structural properties of classical image filters. In a similar vein, we identify Rodrigues’ rotation formula as the basic operator in articulated forward kinematics and generalize it into a learnable form, which we call the Neural Rodrigues Operator . Concretely, we separate the entries in the Rodrigues’ rotation formula into state-dependent parameters conditioned on the joint angles, and state-independent coefficients that only rely on the articulated structure. We then extend it into a neural operator by treating the state-dependent parameters as input features, and relaxing the state-independent coefficients into optimizable weights. We further extend it into a multi-channel operator that applies to higher-dimensional features instead of 1D joint angles. With our neural operator as the key component, we construct a complete network architecture, named the Rodrigues Network (RodriNet) , for encoding, understanding, and synthesizing articulated actions. It comprises three key components: a Rodrigues Layer for passing information from joints to links, built up our neural operator; a Joint Layer for passing information from links to joints; and finally, a Self-Attention Layer for global information exchange. We also introduce a global token for processing other task variables, such as perception inputs. Although starting with an operator in classical control theory, we end up with a deep neural network with modern designs while maintaining the structural bias in articulated kinematics, as illustrated in Figure 1 . We evaluate our approach through three sets of experiments: First, we demonstrate the strong expressivity of our network in representing forward kinematics and motions through toy experiments. Second, we showcase our effectiveness in realistic robot-learning scenarios with imitation learning on 5 robot manipulation tasks, where our network achieves state-of-the-art results. Finally, we also show our network achieving state-of-the-art results in human hand pose estimation from images, where the articulated actor is no longer a robot, highlighting its applications in computer vision and graphics. 2 Related work: articulation-aware robot learning Articulated robots can be naturally modeled as graphs, making graph convolution [ 4 , 36 ] a popular approach for processing articulated data. This has been widely applied in character animation for tasks like skeleton-based action recognition [ 54 , 13 , 45 , 46 , 12 , 5 ] , pose estimation [ 16 , 15 , 55 , 8 ] , and motion retargeting [ 2 ] . Graph convolution effectively captures link connectivity and spatial locality, but it does not explicitly incorporate articulated kinematics, which is an essential aspect of articulated action understanding. In contrast, our work introduces a novel operator derived from forward kinematics, providing networks with kinematics-informed inductive bias. Transformer-based architectures [ 48 ] are widely used in policy learning [ 3 , 57 , 9 , 39 , 43 ] , and recent work has explored incorporating structural bias via graph-aware positional embeddings [ 22 ] or masked attention [ 41 ] . However, these modifications do not fundamentally adapt the self-attention mechanism to suit kinematics. We instead use standard self-attention layers for network capacity, but rely on our kinematics-inspired operator for inductive bias. Forward kinematics provides a deterministic mapping from joint space to Cartesian space, but integrating it into neural networks remains non-trivial. Prior methods have inserted analytical forward kinematics as a differentiable layer in neural networks [ 49 ] to help them reason about the Cartesian results of the robot action. While this introduces kinematic awareness, it constrains the model’s flexibility. Other methods apply Cartesian-space loss functions after computing forward kinematics on network outputs [ 38 , 23 , 30 ] , but these approaches do not alter the network architecture and are thus orthogonal to our focus. In contrast, our method derive a learnable operator from forward kinematics,