Sample-Efficient Online Control Policy Learning with Real-Time Recursive Model Updates
Zixin Zhang, James Avtges, Todd D. Murphey · 2025
Read Sample-Efficient Online Control Policy Learning with Real-Time Recursive Model Updates by Zixin Zhang, James Avtges, Todd D. Murphey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08241.txt.
Introduction
Robots learning control from very little data, with models updated live, are the subject of a 2025 paper by Zixin Zhang and 2 co-authors. Hardware experiments are slow and costly, so data-driven control must be sample-efficient and light, yet modern methods usually demand large datasets and cannot refresh their models in real time.
Koopman theory recasts a nonlinear system as a linear model over well-chosen observables, and such representations can be fitted in an optimisation-friendly way that supports rapid updates. The authors build a recursive, sample-efficient learning pipeline around that idea for online policy learning. A controller that relearns its plant model on the fly can follow equipment as it wears, warms, or changes load, instead of trusting yesterday's calibration.
Catalog record
| CREATOR | Zixin Zhang, James Avtges, Todd D. Murphey |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-08241
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sample-Efficient Online Control Policy Learning with Real-Time Recursive Model Updates Zixin Zhang, James Avtges, Todd D. Murphey Abstract Data-driven control methods need to be sample-efficient and lightweight, especially when data acquisition and computational resources are limited—such as during learning on hardware. Most modern data-driven methods require large datasets and struggle with real-time updates of models, limiting their performance in dynamic environments. Koopman theory formally represents nonlinear systems as linear models over observables, and Koopman representations can be determined from data in an optimization-friendly setting with potentially rapid model updates. In this paper, we present a highly sample-efficient, Koopman-based learning pipeline: Recursive Koopman Learning (RKL). We identify sufficient conditions for model convergence and provide formal algorithmic analysis supporting our claim that RKL is lightweight and fast, with complexity independent of dataset size. We validate our method on a simulated planar two-link arm and a hybrid nonlinear hardware system with soft actuators, showing that real-time recursive Koopman model updates improve the sample efficiency and stability of data-driven controller synthesis—requiring only < < 10% of the data compared to benchmarks. The high-performance C++ codebase is open-sourced. Website: https://www.zixinatom990.com/home/robotics/corl-2025-recursive-koopman-learning . Keywords: Koopman Operator, Control 1 Introduction Figure 1: Recursive Koopman Learning Pipeline Designing controllers for nonlinear, dynamic robotic systems is challenging. Conventional model-based approaches rely on first principles to derive system models and then compute control inputs by solving constrained optimal control problems, often through trajectory optimization. While these methods are mathematically rigorous and have been successfully deployed on robotic systems [ 1 , 2 , 3 , 4 ] , they exhibit inherent limitations: they are difficult to apply to systems that cannot be easily modeled using first principles, such as soft robots; they require significant time and effort to be adapted for highly dynamic tasks or complex environments with significant uncertainties; they struggle with designing controllers for hybrid dynamical systems, especially those involving multiple contact modes; and as the number of constraints increases, these methods impose escalating demands on computational resources and solver performance. Data-driven approaches have advantages in dealing with uncertainties and complex dynamics. Over the past decade, these approaches—especially Reinforcement Learning (RL)—have made significant advances. RL algorithms [ 5 , 6 , 7 ] have been widely used to design controllers for various robotic systems and have achieved remarkable performance in challenging tasks, including soft robot control [ 8 ] , dexterous manipulation [ 9 , 10 ] , vehicle control [ 11 ] , and dynamic locomotion [ 12 , 13 ] . However, RL typically require large amounts of data and have low sample efficiency, and their training demands substantial computational resources. These factors make it difficult to perform RL training solely on hardware—an important consideration for systems that cannot be simulated with high fidelity, such as soft robots. Additionally, due to the stochastic processes involved and the use of neural networks, RL often lack convergence guarantees and mathematical interpretability. Given the disadvantages of conventional model-based control and RL, there is a need for sample-efficient and computationally lightweight data-driven control policy learning methods. A natural idea is to use linear model-based methods—linearity brings convexity and makes computations lighter and faster, and model-based methods tend to have higher sample efficiency. Standard linearization—which is performed locally and in the original state space—can only produce accurate local models. In contrast, Koopman theory can help us find exact linear models for nonlinear systems in infinite dimensional spaces. With advances in computational capabilities, it has become feasible to estimate Koopman operators effectively using data-driven methods such as Dynamic Mode Decomposition (DMD) and Extended Dynamic Mode Decomposition (EDMD) [ 14 , 15 ] . As a result, Koopman theory has been successfully applied in various fields, including fluid dynamics [ 16 ] , physics [ 17 ] , power systems [ 18 ] , and robotics [ 19 , 20 , 21 , 22 , 23 ] . To improve sample efficiency, a key strategy is to identify and use the most informative data. A widely accepted but rarely formalized hypothesis—referred to here as the Attempting Control Goal (ACG) hypothesis—suggests that data collected while attempting a control objective with an ideal policy is particularly informative for that task. An ideal policy is a control policy that is dynamically feasible and globally optimizes a specified performance criterion—such as minimizing a cost function or maximizing reward—under the true system dynamics. Although such a policy is unattainable, researchers aim to approximate it and use the resulting demonstration data to train control policies. In imitation learning [ 24 , 25 ] , for example, human or animal instructors are treated as noisy surrogates of the ideal policy, and their successful demonstrations are used for training. However, certain systems are so challenging that even skilled humans struggle to provide consistently successful demonstrations, and the additional resources required to collect these data cannot be ignored. This motivates a weak ACG hypothesis: when a high-quality approximation of the ideal policy is unavailable, does the iterative loop of (i) computing the best policy attainable with the data so far and (ii) repeatedly attempting to control the system still yield data that are highly informative? This motivates us to adopt real-time online model updates. Researchers have attempted updating NN models with online data [ 11 , 26 ] , but these model-updating methods are heavy, slow, far from achieving real-time performance, and sample-inefficient. Linear models are particularly well-suited for this purpose, and researchers have explored updating Koopman models with new data. While a few papers mention model updates [ 21 , 23 ] , they primarily rely on retraining or iterative numerical solvers to update the parameters, which can be computationally intensive and slow. Additionally, some works have proposed faster online updating algorithms for linear models inspired by the Recursive Least Squares (RLS) algorithm [ 25 , 27 , 28 ] ; however, these approaches lack comprehensive convergence and complexity analyses, and no experimental validation has been conducted on real-world robotic systems. [ 28 ] also uses RLS and MPC, but its update termination condition—which hinders convergence analysis—and MPC operate on the state space rather than the observable, questionable for Koopman-based methods. Several studies have analyzed the convergence properties of EDMD. For instance, [ 29 , 30 , 31 ] have provided detailed discussions on the convergence of EDMD for both stochastic and deterministic systems, under i.i.d. and ergodic sampling assumptions. However, no existing work formally analyzes the convergence of EDMD on Markov chains—a key property of data generated by dynamical systems. The original EDMD paper [ 15 ] conceptually discusses approximations of the stochastic Koopman operator for Markov processes, but does not present a formal analysis. In addition, it omits the assumptions necessary to apply the Strong Law of Large Numbers for Markov chains [ 32 ] , and its treatment of the sampling density does not impose any assumptions on dataset structure. [ 29 ] derives probabilistic error bounds on finite data assuming i.i.d. or ergodic sampling with the exponentially stable Koopman semigroup—a strong assumption which presupposes all non-constant observables decay exponentially toward their equilibrium values over time—without considering data correlation that is critical when using data from trajectories. In this paper, we introduce Recursive Koopman Learning (RKL), a sample-efficient, Koopman-based