Learn Structure, Adapt on the Fly: Multi-Scale Residual Learning and Online Adaptation for Aerial Manipulators

Samaksh Ujjawal, Naveen Sudheer Nair, Shivansh Pratap Singh, Rishabh Dev Yadav, Wei Pan, Spandan Roy · 2026

Read Learn Structure, Adapt on the Fly: Multi-Scale Residual Learning and Online Adaptation for Aerial Manipulators by Samaksh Ujjawal, Naveen Sudheer Nair, Shivansh Pratap Singh, Rishabh Dev Yadav, Wei Pan, Spandan Roy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11638.txt.

Introduction

Ujjawal and co-authors build a predictive-adaptive framework for aerial manipulators whose dynamics keep shifting. These flying arms face coupled, nonlinear motion, with residuals that shift under payload jumps. The core is a Factorized Dynamics Transformer, which treats each physical variable as its own token. That split lets short-horizon inertial effects stay apart from long-horizon aerodynamic ones. A Latent Residual Adapter then retunes the model online with recursive least squares. Real flights with unseen payloads show sharper forecasts.

Catalog record

CREATORSamaksh Ujjawal, Naveen Sudheer Nair, Shivansh Pratap Singh, Rishabh Dev Yadav, Wei Pan, Spandan Roy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39118
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learn Structure, Adapt on the Fly: Multi-Scale Residual Learning and Online Adaptation for Aerial Manipulators Samaksh Ujjawal, Naveen Sudheer Nair, Shivansh Pratap Singh, Rishabh Dev Yadav, Wei Pan, Spandan Roy Abstract Autonomous Aerial Manipulators (AAMs) are inherently coupled, nonlinear systems that exhibit nonstationary and multiscale residual dynamics, particularly during manipulator reconfiguration and abrupt payload variations. Conventional analytical dynamic models rely on fixed parametric structures, while static data-driven model assume stationary dynamics and degrade under configuration changes and payload variations. Moreover, existing learning architectures do not explicitly factorize cross-variable coupling and multi-scale temporal effects, conflating instantaneous inertial dynamics with long-horizon regime evolution. We propose a predictive-adaptive framework for real-time residual modeling and compensation in AAMs. The core of this framework is the Factorized Dynamics Transformer (FDT), which treats physical variables as independent tokens. This design enables explicit cross-variable attention while structurally separating short-horizon inertial dependencies from long-horizon aerodynamic effects. To address deployment-time distribution shifts, a Latent Residual Adapter (LRA) performs rapid linear adaptation in the latent space via Recursive Least Squares, preserving the offline nonlinear representation without prohibitive computational overhead. The adapted residual forecast is directly integrated into a residual-compensated adaptive controller. Real-world experiments on an aerial manipulator subjected to unseen payloads demonstrate higher prediction fidelity, accelerated disturbance attenuation, and superior closed-loop tracking precision compared to state-of-the-art learning baselines, all while maintaining strict real-time feasibility. I Introduction Autonomous Aerial Manipulators (AAMs) integrate a quadrotor or multirotor drone with a robotic arm. This setup boosts the drone’s dexterity and versatility, enabling tasks from basic payload transport to advanced operations like pick-and-place, contact inspection, grasping, and assembly [ 18 , 24 ] . Nevertheless, this integration introduces fundamentally challenging dynamics that defy classical analytical treatment such as configuration-dependent inertia variations as the arm moves or engages the environment (e.g., during grasping or dropping, see Fig. 1 ), nonlinear coupling forces between the drone and arm, and aerodynamic phenomena, such as rotor downwash and manipulator-induced flow interference [ 14 , 12 ] . These forces collectively dominate system behavior, making it diverge from assumed mathematical model and giving rise to residual dynamics . Conventional AAM control methods [ 14 , 12 , 10 ] rely on analytical knowledge of the underlying system model. Unfortunately, it is difficult, if at all possible, to accurately represent configuration-dependent inertial coupling forces analytically [ 14 , Ch.5.3] . Therefore, these traditional control frameworks struggle to compensate for residual dynamics. Data-driven methods [ 22 , 6 ] have recently found footprints in robotics literature to tackle the issues that arise from model-based methods. However, these existing methods are not directly suitable for the residual dynamics in an AAM system. In the following, we steer our discussion in this direction. Fig. 1: Aerial manipulation platform consisting of a quadrotor and a 2-DOF robotic arm transporting different payloads. I-A Related Works Offline data-driven models such as Deep Neural Networks (DNNs) [ 23 , 19 ] , Gaussian Processes (GPs) [ 25 , 1 ] , Physics-Informed Temporal Convolutional Networks (TCNs) [ 21 ] , Recurrent Neural Networks (RNNs) [ 11 ] , NeuralSDEs [ 4 ] and Diffusion [ 3 ] have been found effective for quadrotors. While expressive beyond analytic models, offline models trained on static datasets perform poorly for AAM under time-varying dynamics caused by manipulator reconfiguration and payload changes, leading to accumulating prediction errors and degraded control without online adaptation [ 22 ] . Recent work addresses nonstationary and time-varying disturbances by adapting [ 15 ] or updating the dynamics model [ 20 , 6 , 5 , 8 , 17 ] . Several deep learning–based approaches update only the final layer via gradient descent [ 20 ] requiring careful optimizer tuning and exhibiting poor sample efficiency under limited data. Others periodically retrain the full network and merge parameters using an exponential moving average [ 6 ] , incurring substantial computational overhead. Meta-learning methods [ 15 , 8 ] interpolate among pre-trained models, thereby limiting extrapolation to previously unseen dynamics. While GP regression [ 5 ] , Bayesian last-layer adaptation [ 8 ] , and adaptive control schemes [ 17 ] enable online updates, they do not explicitly separate cross-variable coupling from multi-scale temporal effects, such as fast configuration-dependent inertial interactions and slower aerodynamic or payload-induced transients. Moreover, these methods typically assume gradual and sufficiently data-rich adaptation regimes and do not explicitly address multi-scale residual dynamics. The structural requirements motivating our design are discussed below. Multiscale Dynamics Residual dynamics in AAMs exhibit a multi-scale structure: fast components from configuration-dependent inertial coupling and Coriolis effects, and slower components from aerodynamic wake interactions and payload-induced inertia redistribution [ 13 , 12 ] . The former depend on the instantaneous state, whereas the latter evolve over multiple control cycles due to physical time constants. Capturing both requires short-horizon history (e.g., 1 – 5 samples) for fast coupling and longer windows (e.g., \geq 5 – 20 samples) for slowly varying regimes. Standard Transformers do not explicitly separate these temporal scales. Spatial Tokenization Conventional time-tokenized Transformers emphasize dependencies across time steps, whereas AAM residual dynamics are primarily governed by cross-variable coupling across degrees of freedom. Consequently, temporal attention conflates instantaneous spatial interactions with long-horizon regime memory, lacking inductive bias aligned with the underlying physics. Online Adaptation Payload changes and aggressive maneuvers require adaptation to streaming data without retraining the entire network or compromising high-frequency control stability. This demands decoupling offline nonlinear representation learning from lightweight online parameter identification, which generic Transformer frameworks do not inherently support. Taken together, effective residual modeling for aerial manipulators demands structured factorization across variables, temporal scales, and adaptation modes. To address these requirements, we propose a predictive–adaptive framework (cf. Fig. 3 ) with the following contributions: 1. Structure-Aligned Residual Modeling: We introduce the Factorized Dynamics Transformer (FDT), a dual-stream architecture that treats physical variables as tokens rather than time steps. This induces attention across physical variables while structurally separating short-horizon interactions from long-horizon regime memory. 2. Latent Linear Online Identification: We propose a Latent Residual Adapter (LRA) that restricts online adaptation to a linear correction in latent space, updated via Recursive Least Squares. This preserves the offline nonlinear representation while enabling rapid adaptation without modifying the backbone network. 3. Residual-Compensated Adaptive Control Integration: The adapted residual prediction is incorporated into an adaptive control law to compensate modeling error in closed loop. The resulting controller avoids full knowledge or identification of system model and adapts for a single parameter irrespective of system complexity, avoiding any computational burden. The proposed framework aligns architectural structure with the physical characteristics of aerial manipulator residual dynamics, enabling accurate prediction and principled online adaptation. We validate the approach on a real aerial manipulator platform under unseen payload conditions, demonstrating improved prediction fidelity, faster disturbance recovery, and real-time feasibility in closed-loop operation. II Methodology We consider a quadrotor-based AAM composed of a 6-degrees-of-freedom (DOF) floating base and an n_{arm} -DOF robotic arm. Let n\triangleq 6+n_{arm} denote the total number of generalized coordinates, and let \chi\in\mathbb{R}^{n} be the full configuration vector with control input \tau\in\mathbb{R}^{n} . The system dynamics follow the Euler–Lagrange formulation [ 14 , 13 ] M(\chi)\ddot{\chi}+C(\chi,\dot{\chi})\dot{\chi}+g(\chi)+d(\dot{\chi},t)=\tau, (1) where M(\chi)\in\mathbb{R}^{n\times n} is the inertia matrix, C(\chi,\dot{\chi})\dot{\chi} represents Coriolis and centripetal effects, g(\chi) denotes gravity,

Read the complete text