Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation
Hong Chen (✉), Daqi Liu (✉), Zehan Zhang (✉), Haiguang Wang, Tianhao Lu, Longfei Yan, Haiyang Sun, Fangzhen Li, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye (✉), Yihua Tan (✉) · 2026
Read Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation by Hong Chen (✉), Daqi Liu (✉), Zehan Zhang (✉), Haiguang Wang, Tianhao Lu, Longfei Yan, Haiyang Sun, Fangzhen Li, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye (✉), Yihua Tan (✉) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29908.txt.
Introduction
Chen and twelve colleagues propose SWAM, a world action model for robot navigation from camera images. A world model predicts how the scene will look after moving, and older planners separate choosing a goal from generating the path, which costs time and can mismatch actions with predicted views.
Given a start image and a goal image, SWAM generates in one pass both the intermediate depth-augmented views and the action trajectory, keeping the two consistent. It learns from depth pseudo-labels during training but needs only an ordinary colour camera when running. A refinement module aligns motion with visual cues, and a scale loss steadies predictions over different distances. In experiments SWAM beats leading two-stage planners in success rate, accuracy, and speed, and generalises to unseen places.
Catalog record
| CREATOR | Hong Chen (✉), Daqi Liu (✉), Zehan Zhang (✉), Haiguang Wang, Tianhao Lu, Longfei Yan, Haiyang Sun, Fangzhen Li, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye (✉), Yihua Tan (✉) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29908
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation Hong Chen (✉), Daqi Liu (✉), Zehan Zhang (✉), Haiguang Wang, Tianhao Lu, Longfei Yan, Haiyang Sun, Fangzhen Li, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye (✉), Yihua Tan (✉) Abstract Existing world model-based planners for visual navigation typically follow a verification-centric paradigm, decoupling goal intent from trajectory synthesis. This approach suffers from candidate dependence, heavy computational overhead, and inconsistencies between sampled actions and predicted visuals. To address these issues, we propose SWAM ( S patial-perceiving W orld A ction M odel), a task-centric joint observation-action generation framework. Given start and goal RGB observations, SWAM performs single-pass inference to simultaneously generate intermediate RGB-D sequences and corresponding action trajectories, promoting goal-consistent trajectory generation and improved spatial feasibility. While SWAM leverages depth pseudo-labels during training to internalize spatial priors, it requires only monocular RGB input at inference time. We further introduce a visual-guided action refinement module and a trajectory-scale regularization loss to enforce fine-grained alignment between motion and visual cues while stabilizing predictions across varying distances. Extensive experiments show that SWAM significantly outperforms state-of-the-art two-stage planners in success rate, trajectory accuracy, and inference efficiency, while demonstrating robust zero-shot generalization to unseen environments. Figure 1 : Comparison of different paradigms for visual navigation. Left: Nomad predicts trajectories from historical RGB frames and a goal image. Middle: NWM generates future RGB videos conditioned on trajectories. Right (Ours): Our model jointly predicts trajectories and RGBD paths from the current RGB observation and the goal image in one forward pass, enabling efficient planning. 1 Introduction Visual goal-conditioned navigation [ 4 , 51 ] is a core challenge for intelligent systems operating in the physical world. The task demands that an agent, given a visual goal and its current observation, must infer where to move next, which paths are traversable, and when the goal has been successfully reached. This requires tight integration of visual perception, spatial reasoning, and decision-making [ 8 , 13 ] . Direct online policy methods learn observation-to-action mappings [ 29 , 31 , 32 , 36 , 37 , 41 ] . While inference-efficient, they lack explicit visual horizons and struggle with error recovery. Offline world model-based planners [ 2 , 45 ] adopt a two-stage pipeline: sample candidate action sequences from an external policy [ 36 , 6 ] , rollout visual trajectories via an action-conditioned video predictor, and select the trajectory best matching the goal. While this enables simulation-evidence-based evaluation, its modular design constrains the world model to verification rather than trajectory synthesis, leading to three limitations: (i) candidate dependence, decision quality is bounded by candidate coverage; (ii) computational inefficiency, exhaustive sampling and rollout incur prohibitive inference costs, especially in complex or longer planning settings; (iii) geometric inconsistency, absence of explicit spatial constraints can produce abrupt viewpoint changes or infeasible trajectories, reducing the reliability of simulation evidence. We argue that these limitations stem from decoupling the path proposal and environment prediction. Verification-centric pipelines treat the world model [ 40 , 43 , 28 ] as a passive evaluator, decoupling goal intent from trajectory generation and relegating spatial reasoning to implicit visual patterns, which neglects the core requirement of navigation. To address this, we propose a Spatial-perceiving World Action Model (SWAM), a joint observation–action generation framework. Given a start RGB observation and a goal RGB observation, our model performs single-pass inference to jointly generate an intermediate RGBD observation sequence and the corresponding action trajectory. Unlike unidirectional world models [ 2 , 54 ] , our approach co-generates "how to act" and "what to perceive" within a unified process. This approach jointly constrains actions and observations throughout generation, ensuring goal alignment, temporal consistency, and spatial feasibility. Unlike verification-centric world models, SWAM directly synthesizes control trajectories through joint latent denoising. SWAM offers three key advantages: (1) goal alignment is maintained throughout generation, producing structured, target-directed paths; (2) inference cost is reduced by eliminating candidate expansion and repeated evaluation; (3) explicit injection of spatial cues enhances the constraints on actions. Technically, we leverage DepthAnything v3 [ 21 ] to provide per-frame depth pseudo-labels during training, leveraging spatial priors while requiring only monocular RGB at test time. Estimated depth supplies sufficient navigable cues, avoiding reliance on scarce ground-truth depth. We further introduce a Visual-Guided Action Refinement (VGAR) module and Trajectory-Scale Regularization (TSR) loss to enhance the alignment between predicted actions and generated visual cues, and to stabilize trajectory predictions over various distances. Experiments show that SWAM outperforms strong policy-based and two-stage baselines in trajectory error, goal success rate, visual fidelity, and practical inference efficiency, while demonstrating more stable long-distance and cross-dataset behavior. In conclusion, our contributions are threefold: • We introduce SWAM, a joint observation–action generation framework for visual navigation, replacing candidate-based pipelines; • We propose a visual-guided action refinement and trajectory-scale regularization loss to better leverage visual cues for better action prediction and reduce trajectory drift error; • We validate SWAM across multiple datasets, showing improved accuracy, success rate, visual quality, inference efficiency, and robust cross-scene generalization. 2 Related Work 2.1 Visual Goal-Conditioned Navigation Visual goal-conditioned navigation aims to learn policies that guide an agent to reach a target location specified by a goal observation. Early approaches formulate this problem as learning a mapping from the current observation and a goal image to control actions, typically using reinforcement learning or imitation learning [ 56 , 27 , 11 ] . These methods learn deterministic policies that directly predict actions from visual observations and have demonstrated strong performance in simulated indoor environments. Subsequent work improves policy learning through representation learning and memory mechanisms, enabling agents to reason over partial observations and long-horizon dependencies [ 24 , 25 ] . However, deterministic policy learning often struggles with the multi-modality of navigation behaviors, where multiple valid trajectories can reach the same goal. To address this limitation, recent works explore generative formulations of navigation policies. In particular, diffusion-based policies have been proposed to model distributions over action trajectories, enabling more diverse and robust navigation strategies [ 36 , 6 ] . While these approaches improve the expressiveness of goal-conditioned policies, they typically generate action sequences alone, without explicitly modeling the intermediate visual states along the trajectory. In contrast, our approach formulates navigation as joint video–action generation conditioned on the goal observation, allowing the model to explicitly capture the visual transitions connecting the start and goal states. 2.2 Action-Conditioned World Models for Planning and Navigation World models [ 1 , 14 , 20 ] aim to learn predictive models of environment dynamics that allow agents to plan by imagining future observations under candidate actions. Early works, such as visual model predictive control, predict future images conditioned on candidate action sequences and optimize actions by minimizing perceptual distance to a goal observation [ 16 , 7 ] . Recent advances in generative modeling [ 17 , 22 , 34 ] have significantly improved the capacity of world models. A number of works adopt video prediction or video diffusion models to model environment dynamics for planning [ 12 ] . More recently, large-scale generative models have been used to construct action-conditioned video diffusion world models [ 23 , 39 , 52 ] capable of generating realistic imagined rollouts [ 2 ,