Reinforcement Learning Enables Autonomous Microrobot Navigation and Intervention in Simulated Blood Capillaries
Jannik Drotleff, Samuel Tovey, Paul Hohenberger, Christoph Lohrmann, Julian Hoßbach, Konstantin Nikolaou, Christian Holm, [0.7em] Institute for Computational Physics, University of StuttgartAllmandring 3, 70569 Stuttgart, Baden-Württemberg, Germany · 2026
Read Reinforcement Learning Enables Autonomous Microrobot Navigation and Intervention in Simulated Blood Capillaries by Jannik Drotleff, Samuel Tovey, Paul Hohenberger, Christoph Lohrmann, Julian Hoßbach, Konstantin Nikolaou, Christian Holm, [0.7em] Institute for Computational Physics, University of StuttgartAllmandring 3, 70569 Stuttgart, Baden-Württemberg, Germany (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26154.txt.
Introduction
Jannik Drotleff and colleagues train microrobots to swim through blood capillaries, in simulation. Tiny robots could one day deliver drugs to exact spots or dissolve clots. Real capillaries are hostile classrooms: twisting flow fields, branching walls, and crowds of red blood cells. Their simulator includes all three, with branching geometry drawn from anatomy. Deep reinforcement learning agents learn navigation there by chemotaxis, following chemical gradients.
Physical limits map out clearly. Across robot sizes and swimming speeds, a forbidden zone appears where Brownian jitter and flow beat propulsion. No policy can cross it. Successful agents discover shared strategies on their own. Run-and-rotate and search-and-sit styles emerge regardless of robot parameters. One search style is notably energy-thrifty. Without retraining, agents then block and unblock capillary flow on demand. Throughput returns to healthy baseline levels, a first step toward robotic intervention in the bloodstream.
Catalog record
| CREATOR | Jannik Drotleff, Samuel Tovey, Paul Hohenberger, Christoph Lohrmann, Julian Hoßbach, Konstantin Nikolaou, Christian Holm, [0.7em] Institute for Computational Physics, University of StuttgartAllmandring 3, 70569 Stuttgart, Baden-Württemberg, Germany |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26154
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning Enables Autonomous Microrobot Navigation and Intervention in Simulated Blood Capillaries Jannik Drotleff, Samuel Tovey, Paul Hohenberger, Christoph Lohrmann, Julian Hoßbach, Konstantin Nikolaou, Christian Holm, [0.7em] Institute for Computational Physics, University of StuttgartAllmandring 3, 70569 Stuttgart, Baden-Württemberg, Germany Abstract Autonomous microrobots navigating biological vasculature could enable targeted drug delivery and thrombolysis, yet training control policies for realistic environments remains an open challenge. Prior reinforcement learning (RL) studies of microrobotic navigation have been limited to idealized geometries that omit complex hydrodynamic flow fields, confined branching structures, and dense cellular obstacles found in vivo. Here, we develop a physically grounded simulation of a blood capillary network, incorporating realistic hydrodynamic flow fields, explicit red blood cell dynamics, and anatomically derived branching geometry, and train deep RL agents to navigate it via chemotaxis. We systematically map the physical limits of navigation across robot size and swimming speed, revealing a forbidden regime where Brownian motion and flow overcome propulsion. Successful agents independently discover multiple universal strategy types, including run-and-rotate and energy-efficient search-and-sit policies, regardless of robot parameters. Without retraining, these agents perform targeted blocking and unblocking of capillary flow, restoring throughput to healthy baseline levels. These results establish RL as a viable framework for developing autonomous microrobotic intervention strategies in complex biological environments. 1 Introduction The ability to precisely control robots on the micrometer scale would open a new era of medical applications, from microscale manufacturing [ 1 ] and porous media exploration [ 2 , 3 ] to targeted drug delivery, minimally invasive surgery [ 4 ] , and thrombolysis [ 5 ] . The clinical urgency is substantial: thrombosis-related conditions account for one in four deaths worldwide [ 6 ] . Realizing this potential demands solutions to two coupled challenges: building microrobots that can propel themselves at low Reynolds numbers and developing control strategies that function in the complex, noisy environments these robots will encounter. At the microscale, motion is governed by a low Reynolds number ( Re ), where viscous forces dominate inertial effects, and propulsion becomes fundamentally different from macroscopic locomotion. Despite this constraint, hardware progress has been rapid. Externally controlled magnetic microrobots have shown promising results, both as nanoparticle swarms [ 7 , 8 , 9 ] and individual colloids [ 10 ] . Bio-inspired flagella provide propulsion and orientation mechanisms [ 11 , 12 , 13 ] , while Janus particles offer asymmetric surface properties that enable control via magnetic fields and laser-driven self-thermophoresis [ 14 , 15 ] . Most recently, a fully integrated robotic surface swimmer with on-board sensing, computation, memory, communication, and locomotion was realized at the 200\text{\,}\mathrm{\SIUnitSymbolMicro m} scale [ 16 ] , demonstrating that previously assumed engineering limitations can be overcome. Yet even the most advanced designs offer only rudimentary actuation (typically translation and uncontrolled rotation) and remain confined to surface swimming, far from operating inside the body. Further complicating control is Brownian motion, a stochastic noise arising from thermal interactions with the surrounding fluid, which constantly perturbs the robot’s trajectory. Reinforcement learning (RL) has become a powerful tool for discovering effective navigation strategies in these challenging environments [ 17 ] . Initial applications focused on optimizing navigation within fluidic disturbances: Colabrese et al. [ 18 ] utilized Q-learning to enable gravitactic microswimmers to exploit vortex flows, outperforming passive strategies, while Nasiri and Liebchen [ 19 ] employed the Advantage Actor-Critic algorithm to guide active particles through complex two-dimensional flow fields. Moving beyond purely synthetic settings, Muiños-Landin et al. [ 20 ] demonstrated RL-based microrobotic control under real experimental conditions. Other work has focused on emergent behaviors: Xiong et al. [ 21 ] and Tovey et al. [ 22 ] showed that RL agents trained via actor-critic architectures can autonomously rediscover chemotactic behaviors, the oriented motion toward higher chemical concentrations, recovering the same run-and-tumble strategies used by bacteria [ 23 , 24 , 25 ] . The mechanics underlying these learned policies have been characterized [ 26 ] , revealing how neural network responses to environmental inputs produce the observed motion patterns. To facilitate such studies, Tovey et al. [ 27 ] introduced SwarmRL, an open-source platform that integrates RL with molecular dynamics simulations for active matter research. However, a significant gap remains between these idealized simulations and the application-specific environments needed for microrobotic deployment. Current studies focus on understanding control algorithms and swimming mechanisms in open domains, simple channels, or uniform flow fields, with little attention given to the realistic environments required to model actual deployment scenarios. One such environment is that of a blood capillary network: a complex geometry of branching paths under the influence of flow fields and filled with passive blood cells. Biological capillaries present severe challenges from both simulation and learning perspectives, as robots must navigate confined geometries where flow fields are further complicated by the dense presence of dynamic obstacles such as red blood cells (RBCs) [ 28 , 29 ] . Here, we bridge this gap by developing a physically grounded simulation of a blood capillary network to train and evaluate deep RL agents. Rather than aiming for a perfect biological representation, we employ a geometry that qualitatively captures essential properties—such as branching morphology and confined flow. Our environment incorporates realistic branching geometry derived from anatomical illustration, steady-state hydrodynamic flow fields, explicit RBC dynamics, and Brownian fluctuations. Within this framework, we map the physical boundaries of successful chemotactic navigation across robot size and speed, analyze the emergent control strategies, and demonstrate that trained agents can perform targeted blocking and unblocking of capillary flow without retraining—capabilities directly relevant to embolotherapy and thrombolysis. 2 Results Capillary environment and navigation challenge Figure 1 : Simulated capillary environment. a) Capillary geometry and Lattice-Boltzmann flow field, derived from an anatomical capillary illustration [ 30 ] . b) Static concentration field for a chemical source at \vec{r}_{s}=($119\text{\,}\mathrm{\SIUnitSymbolMicro m}$,$175\text{\,}\mathrm{\SIUnitSymbolMicro m}$)^{T} , constrained by capillary walls. To bridge the gap between idealized simulations and the complexity of biological environments, we established a realistic simulation environment modeled after human blood capillary structures. We derived the simulation boundaries from an anatomical capillary illustration from the Encyclopedia Britannica [ 30 ] , mapping it into a domain of $200\text{\,}\mathrm{\SIUnitSymbolMicro m}$\times$184.13\text{\,}\mathrm{\SIUnitSymbolMicro m}$ based on physiological capillary diameters of 5\text{\,}\mathrm{\SIUnitSymbolMicro m} to 10\text{\,}\mathrm{\SIUnitSymbolMicro m} [ 31 ] . A steady-state flow field was computed using the Lattice-Boltzmann method, with the vessel geometry dictating local flow velocities and creating a training environment where peak velocities reach approximately 3.5\times the mean flow ( Fig. 1 a ). Central to our approach is chemotaxis, a directed motion along a chemical concentration gradient toward a source. To represent the chemical signal, we applied a static concentration field that weakens as it spreads. We constrained this field to the inside of the capillaries, ensuring that the gradient accurately traces the anatomical branching structure ( Fig. 1 b ). To maximize the navigation challenge for the RL agents, the chemical source was placed at \vec{r}_{s}=($119\text{\,}\mathrm{\SIUnitSymbolMicro m}$,$175\text{\,}\mathrm{\SIUnitSymbolMicro m}$)^{T} , in a region of comparatively high flow. This steady-state approximation is justified by a Péclet number Pe<1 throughout the domain [ 32 ] , ensuring that diffusion dominates over advective transport of the chemical signal (see Appendix A for the full Péclet analysis). Physical limits of navigation