From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning
Honglin He, Yukai Ma, Wayne Wu, Bolei ZhouUniversity of California, Los Angeles https://metadriverse.github.io/s2e · 2025
Read From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning by Honglin He, Yukai Ma, Wayne Wu, Bolei ZhouUniversity of California, Los Angeles https://metadriverse.github.io/s2e (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22028.txt.
Introduction
Honglin He and co-authors present a learning framework for navigation models in a 2025 arXiv paper. Navigation foundation models learn from huge video collections and generalize across settings. Yet offline training alone leaves them blind to the effects of their own actions.
Their Seeing-to-Experiencing framework adds reinforcement learning in simulation after video pretraining. One innovation steadies offline learning with anchor-based matching of motion patterns. A second, a residual-attention module, adds reactive behavior without erasing pretrained knowledge. The goal is safe, interactive movement in real city settings, such as dodging obstacles and pedestrians. The team also builds a full end-to-end evaluation.
Catalog record
| CREATOR | Honglin He, Yukai Ma, Wayne Wu, Bolei ZhouUniversity of California, Los Angeles https://metadriverse.github.io/s2e |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22028
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning Honglin He, Yukai Ma, Wayne Wu, Bolei ZhouUniversity of California, Los Angeles https://metadriverse.github.io/s2e Abstract Navigation foundation models trained on massive web-scale data enable agents to generalize across diverse environments and embodiments. However, these models, which are trained solely on offline data, often lack the capacity to reason about the consequences of their actions or adapt through counterfactual understanding. They thus face significant limitations in the real-world urban navigation where interactive and safe behaviors, such as avoiding obstacles and moving pedestrians, are critical. To tackle these challenges, we introduce the Seeing-to-Experiencing (S2E) learning framework to scale the capability of navigation foundation models with reinforcement learning. S2E combines the strengths of pre-training on offline videos and post-training through reinforcement learning. It maintains the model’s generalizability acquired from large-scale real-world videos while enhancing its interactivity through reinforcement learning in simulation environments. Specifically, we introduce two innovations: 1) an Anchor-Guided Distribution Matching strategy for offline pretraining, which stabilizes learning and models diverse motion patterns through anchor-based supervision; and 2) a Residual-Attention Module for reinforcement learning, which obtains reactive behaviors from simulation environments without erasing the model’s pretrained knowledge. Moreover, we establish a comprehensive end-to-end evaluation benchmark, NavBench-GS, built on photorealistic 3D Gaussian Splatting reconstructions of real-world scenes that incorporate physical interactions. It can systematically assess the generalizability and safety of navigation foundation models. Extensive experiments show that S2E mitigates the diminishing returns often seen when scaling with offline data alone. We perform a thorough analysis of the benefits of Reinforcement Learning (RL) compared to Supervised Fine-Tuning (SFT) in the context of post-training for robot learning. Our findings emphasize the crucial role of integrating interactive online experiences to effectively scale foundation models in Robotics. * * footnotetext: Equal contribution. Keywords: Urban Navigation, Foundation Models, Reinforcement Learning Figure 1: Real-world deployments of S2E. S2E achieves zero-shot generalization across environments and embodiments. We demonstrate its effectiveness on wheeled and quadruped robots in diverse urban scenarios. 1 Introduction Foundation models have demonstrated transformative capabilities across various domains, including language understanding [ 1 , 2 ] , generation [ 3 , 4 ] , and visual recognition [ 5 , 6 , 7 ] . Through training on massive data, these models significantly enhance the generalizability and adaptability of downstream tasks. However, applying foundation models to robot navigation presents unique challenges [ 8 ] due to the complex nature of sequential decision-making in dynamic real-world environments. For example, in a hustling urban space, navigation foundation models must make real-time decisions to avoid collision with obstacles like trash bins and safely maneuver through ever-changing crowds. Recent work on navigation foundation models has harnessed large-scale web videos and human demonstrations for pretraining [ 9 , 10 , 11 ] . These approaches primarily rely on passive visual learning [ 12 , 13 ] , where models are trained to imitate behaviors observed in massive video data. Such data captures diverse visual observations of the real world; yet, it lacks explicit information on physics and cause-effect relations, which are crucial for decision-making in the real world. As a result, navigation policies trained solely on offline data often exhibit limited reactivity to the surroundings and struggle to adapt to dynamic objects and motions in the environment. While offline video data helps build proper perceptual prior for the model, it only captures statistical correlations, not grounded causality [ 14 ] . Visual imitation [ 15 , 16 ] teaches an agent what actions look like, but not how to adapt, recover, or reason about counterfactual outcomes when the environment changes. Thus, navigation foundation models must move from seeing to experiencing : actively interact with the world, receive feedback, and refine behaviors through trial and error. As shown in Figure 2 , similar to humans learning skateboarding, where experience with balance, falling, and correction is irreplaceable, agents must interact with the world to acquire true adaptability. Reinforcement Learning (RL) enables agents to close the gap between observations and actions and offers a scalable interactive learning paradigm to enrich model capabilities beyond the behavior cloning of static datasets. However, RL alone has shown limited success in building generalizable navigation models. Prior approaches [ 17 , 18 , 19 , 20 , 21 ] have trained agents in narrow synthetic environments through RL; however, due to the poor sampling efficiency and lack of inductive priors, the resulting models are struggling to achieve scalable and generalizable navigation capabilities in the real world. Figure 2: Motivation. Like humans, AI agents must also go through interactive practices and learn from feedback to obtain actionable skills. To this end, we propose a new learning framework, S eeing- to - E xperiencing ( S2E ), to scale navigation foundation models with reinforcement learning in simulated environments while maintaining the generalizable visual representation acquired through pretraining on offline videos. This framework comprises two crucial technical components. First, we introduce an Anchor-Guided Distribution Matching (AGDM) strategy for pre-training on real-world video data. It is designed as an anchor-guided model architecture to learn complex multi-modal distributions in normalized motion trajectory space, enabling efficient representation of diverse behaviors across scenarios. This design significantly mitigates learning uncertainty and provides a more reliable backbone to ease subsequent online adaptation. Moreover, the anchor-based architecture naturally supports the cross-embodiment deployment of the foundation model. Second, we propose a Residual-Attention Module (RAM) for RL post-training in simulation environments. It is designed as a residual architecture by copying the pretrained attention block and learning a residual component that selectively captures knowledge acquired from online interactions. This design allows agents to obtain novel capabilities, such as obstacle avoidance and movement anticipation, through reinforcement learning while preserving the generalizable visuomotor representations learned from offline pretraining. We further introduce NavBench-GS, a comprehensive end-to-end evaluation benchmark built on realistic 3D Gaussian Splatting environments with accurate physics and interactive dynamics. Unlike prior evaluations that rely on offline 2D video-based testing [ 9 , 10 , 11 ] , NavBench-GS enables the reactive simulation to facilitate closed-loop policy assessment in photo-realistic 3D scenes. These scenes, combining realistic visual appearance and physical interaction with reproducibility, address a longstanding challenge in robotics: the difficulty of replicating real-world environments for end-to-end evaluation. It enables standardized, reproducible evaluation of navigation foundation models in terms of generalization and safety in unseen settings. Extensive experiments show that reinforcement learning substantially enhances the policy performance and alleviates the diminishing returns associated with scaling solely on offline data. We analyze the effectiveness of Reinforcement Learning (RL) versus Supervised Fine-Tuning (SFT) in post-training for robot learning. Although both methods are widely discussed in relation to large language models (LLMs) [ 22 ] , they remain underexplored in the field of scaling robot learning. Additionally, we demonstrate the generalizability of the proposed S2E framework through real-world evaluations in challenging scenarios. 2 Related Work Goal conditioned navigation. Goal-conditioned navigation is the most common setting for robotic navigation. Prior works have developed diverse approaches to represent navigation goals, which can be categorized into three main paradigms: 1) image-goal navigation, where target images serve as visual references [ 23 , 24 , 25 ] ; 2) position-goal navigation, which directly encodes destination coordinates [ 26 , 27 , 28 ]