Control of Microrobots with Reinforcement Learning under On-Device Compute Constraints
Yichen Liu 1 , Kesava Viswanadha 1 , Zhongyu Li, Nelson Lojo, Kristofer S. J. Pister · 2025
Read Control of Microrobots with Reinforcement Learning under On-Device Compute Constraints by Yichen Liu 1 , Kesava Viswanadha 1 , Zhongyu Li, Nelson Lojo, Kristofer S. J. Pister (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24740.txt.
Introduction
Yichen Liu, Kesava Viswanadha, Zhongyu Li, and co-authors train a sub-centimeter four-legged microrobot to walk using reinforcement learning run on the robot itself. Tiny robots face tight limits on compute, memory, and power, so control must be low latency and on device. Training used a compact neural policy in massively parallel simulation on graphics hardware.
Domain randomization improved robustness before deployment on an ultra-small chip with a slow microcontroller. Readers in microrobotics see a full path from simulated learning to real locomotion under severe hardware limits. The result shows learned walking need not depend on a powerful off-board computer.
Catalog record
| CREATOR | Yichen Liu 1 , Kesava Viswanadha 1 , Zhongyu Li, Nelson Lojo, Kristofer S. J. Pister |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-24740
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Control of Microrobots with Reinforcement Learning under On-Device Compute Constraints Yichen Liu 1 , Kesava Viswanadha 1 , Zhongyu Li, Nelson Lojo, Kristofer S. J. Pister Abstract An important function of autonomous microrobots is the ability to perform robust movement over terrain. This paper explores an edge ML approach to microrobot locomotion, allowing for on-device, lower latency control under compute, memory, and power constraints. This paper explores the locomotion of a sub-centimeter quadrupedal microrobot via reinforcement learning (RL) and deploys the resulting controller on an ultra-small system-on-chip (SoC), SC μ \mu M-3C, featuring an ARM Cortex-M0 microcontroller running at 5 MHz. We train a compact FP32 multilayer perceptron (MLP) policy with two hidden layers ( [ 128 , 64 ] [128,64] ) in a massively parallel GPU simulation and enhance robustness by utilizing domain randomization over simulation parameters. We then study integer (Int8) quantization (per-tensor and per-feature) to allow for higher inference update rates on our resource-limited hardware, and we connect hardware power budgets to achievable update frequency via a cycles-per-update model for inference on our Cortex-M0. We propose a resource-aware gait scheduling viewpoint: given a device power budget, we can select the gait mode (trot/intermediate/gallop) that maximizes expected RL reward at a corresponding feasible update frequency. Finally, we deploy our MLP policy on a real-world large-scale robot on uneven terrain, qualitatively noting that domain-randomized training can improve out-of-distribution stability. We do not claim real-world large-robot empirical zero- shot transfer in this work. Electrical Engineering and Computer Science Dept., University of California, Berkeley, USA Email: [email protected] * * footnotetext: Both authors contributed equally to this paper. I Introduction Intelligent crawling microrobots based on biomimetic mechanisms have received increasing attention in recent years. Due to their small size and versatility, microrobots have been proposed for applications ranging from planetary exploration [ 1 ] to minimally invasive surgery [ 2 ] . While significant progress has been made in hardware development, including actuation, fabrication, and power efficiency, reliable locomotion control for sub-centimeter terrestrial microrobots remains challenging, particularly on complex terrain. Physical experimentation is constrained by the difficulty of constructing representative test environments and the high cost of iterative testing. Therefore, simulation-based approaches provide a scalable and cost-effective framework for accelerating controller development and experimental iteration. A major challenge regarding microrobot control is the on-device autonomy. Sophisticated control algorithms for larger walking robots can be implemented on high-performance CPUs with lower concern for power constraints [ 31 ] . In contrast, the control algorithm for micro-scale walking robots is limited by the size, computing performance, and power efficiency of the controller hardware. Microrobots must often rely on off-board computation or open-loop gait patterns triggered by external stimulation. Recent work has moved towards microrobot autonomy by performing computation on-device (e.g., on-board digital control for microscopic and millimeter systems [ 20 , 21 , 22 , 23 ] ), but on-device, low-latency, RL-based locomotion for terrestrial microrobots remains under-explored relative to macro-scale quadrupeds. Deploying a gait controller approach on a micro-scale robot is restrictive, aside from the challenges of training and optimizing the controller. This has typically driven researchers towards off-board processing [ 4 ] [ 5 ] or embedding gait in hardware that is activated using external stimulation [ 6 ] . Yet, on-board control remains more responsive and versatile. A truly autonomous micro-robot will be able to maneuver across complex terrain independently without the restriction of a wireless connection. Our group has designed the single-chip micro mote, SC \mu M, a compact microrobot control system as shown in Figure 1 [ 9 ] . Our group has also analyzed a central pattern generator (CPG)-based algorithm for micro-scale crawlers, which has shown promising results in simulation [ 3 ] . Fig. 1 : Single-Chip Micro Mote, SC \mu M (left) and SOI-based microrobots (right). Reinforcement learning (RL) has demonstrated strong results for robust locomotion and Sim2Real transfer on quadrupeds [ 10 ] [ 11 ] . This paper explores RL-based locomotion for sub-centimeter quadrupedal microrobots trained in simulation and executed under strict on-device constraints on the SC \mu M microchip [ 7 ] [ 9 ] . We focus not on whether microrobots can perform sufficient on-board computation (evidenced by [ 20 , 21 , 22 , 23 ] ), but on connecting the learned locomotion reward and policy robustness, the update-frequency dependent reward of gaits, and device-level power and computation constraints into a gait-selection methodology that supports deployment decisions. Our paper focuses on the following: 1. Training an MLP policy for sub-centimeter quadrupedal microrobot locomotion in massively parallel GPU simulation, improving robustness via domain randomization. 2. Analyzing reward trends under changes in inference update frequency and quantization while validating on-device feasibility with SC \mu M-3C. 3. Deriving cycle and power models for Int8 inference on a Cortex-M0-class core and proposing a resource-aware gait-selection method that selects the gait maximizing expected reward under given hardware constraints. II Related Work II-A Microscale Terrestrial Walkers and Controller Design Walking silicon microrobots and MEMS-based terrestrial microrobots have a long history (e.g. early silicon walking microrobots [ 12 ] ). In many insect-scale legged microrobots, locomotion has often been demonstrated in tethered settings using off-device power and control electronics, but high-fidelity wireless locomotion has also recently been demonstrated [ 24 ] . Our group has previously demonstrated the use of multi-objective optimization of the locomotion controller on a hexapod microrobot [ 3 ] . However, the presented works either don’t use fully on-device control or utilize open-loop controls, making the system vulnerable to interference and noise in practice. II-B Learning-Based Control for Microrobot Locomotion Learning-based gait design has been studied for microrobot-inspired platforms [ 3 ] [ 25 ] [ 26 ] . Yang et al. [ 3 ] formulate microrobot locomotion as a data-efficient controller-search problem, using contextual optimization to learn reusable locomotion primitives for a silicon microrobot and then using these primitives for higher-level behaviors like maze navigation. Complementary to this work, Lambert et al. [ 26 ] propose avenues for improving model-based RL control for microrobots by clustering and filtering out redundant data, improving data and power efficiency. Lambert et al. also propose MicroBotNet, a <10^{6} MAC network that provides visual Simultaneous Localization and Mapping (SLAM) for navigation. Together, these works motivate learning-based locomotion at microrobot scales while highlighting hardware constraints as a primary design consideration. II-C Sim2Real and Domain Randomization for Legged Locomotion Simulation-based optimization has proven to be effective in generating training data for locomotion control on macro-scale legged robots. The degradation in solution performance between simulation and reality has also been well-characterized [ 27 ] . Despite the success in macro-scale Sim2Real control, it is difficult to accurately adapt simulation to reality on the sub-centimeter scale. At these length scales, the error in empirical model approximations of quantities like dry friction is amplified due to the complicated mechanics of contact at such a small scale. Physically accurate simulation at this scale is very computationally expensive. To address the issues of error and noise, researchers often resort to analyzing and simulating simplified robot configurations or using scaled-up models to enhance robustness. These approaches are typically not adequate for more intricate small-scale mechanisms, as this oversimplification creates disparity between simulation and real-world performance. We