Multimodal embodiment - aware navigation transformer

Louis Dezons 1 2 , Quentin Picard 1 , Rémi Marsal 2 , François Goulette 2 , David Filliat 1 · 2026

Read Multimodal embodiment - aware navigation transformer by Louis Dezons 1 2 , Quentin Picard 1 , Rémi Marsal 2 , François Goulette 2 , David Filliat 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19267.txt.

Introduction

Dezons and colleagues propose ViLiNT, a navigation policy for ground robots that blends camera images, 3D laser scans, a goal embedding, and a description of the robot's own body shape. A transformer merges those cues and conditions a diffusion model that draws candidate routes. A clearance head then scores and ranks those routes against the robot's dimensions. The design aims to keep collision avoidance working when environment, robot, or sensors change, and it improves success rates across three simulated settings.

Catalog record

CREATORLouis Dezons 1 2 , Quentin Picard 1 , Rémi Marsal 2 , François Goulette 2 , David Filliat 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36501
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19267

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal embodiment - aware navigation transformer Louis Dezons 1 2 , Quentin Picard 1 , Rémi Marsal 2 , François Goulette 2 , David Filliat 1 Abstract Goal-conditioned navigation models for ground robots trained using supervised learning show promising zero-shot transfer, but their collision-avoidance capability nevertheless degrades under distribution shift, i.e. environmental, robot or sensor configuration changes. We propose ViLiNT, a multimodal, attention-based policy for goal navigation, trained on heterogeneous data from multiple platforms and environments, which improves robustness with two key features. First, we fuse RGB images, 3D LiDAR point clouds, a goal embedding and a robot’s embodiment descriptor with a transformer architecture to capture complementary geometry and appearance cues. The transformer’s output is used to condition a diffusion model that generates navigable trajectories. Second, using automatically generated offline labels, we train a path clearance prediction head for scoring and ranking trajectories produced by the diffusion model. The diffusion conditioning as well as the trajectory ranking head depend on a robot’s embodiment token that allows our model to generate and select trajectories with respect to the robot’s dimensions. Across three simulated environments, ViLiNT improves Success Rate on average by 166% over equivalent state-of-the-art vision-only baseline (NoMaD). This increase in performance is confirmed through real-world deployments of a rover navigating in obstacle fields. These results highlight that combining multimodal fusion with our collision prediction mechanism leads to improved off-road navigation robustness 1 1 1 Code link: https://github.com/LARIAD/ViLiNT . I Introduction Robust autonomous navigation in off-road environments remains challenging due to the complexity and high variability of the environment, partial observation, dynamic obstacles, and sensing degradations (e.g., dust, rain, grass). Classical pipelines (e.g., [ 7 ] ) rely on handcrafted or learned traversability maps and local planners. They offer interpretability and a good capacity to detect situations when motion is impossible. However, they may struggle when the deployment configuration (robot or sensor) changes, requiring at least sensor calibration or even new training data. End-to-end control trained by imitation (e.g., [ 21 ] , [ 22 ] ) has shown some robustness to the robot embodiment thanks to training on data from different robots and direct regression of controls or waypoints from onboard sensing. However, single-modality policies (typically RGB vision) may strongly degrade under environmental changes when other modalities like LiDAR may bring more robustness. These end-to-end local planners also often lack explicit mechanisms to deal with deadlocked situations, i.e., to exit local minima where no feasible trajectory to the goal is available. Finally, these approaches offer some robustness and adaptation to different embodiments, but they do not explicitly take the robot’s dimensions into account and cannot easily be adapted to a new platform without fine tuning using data from this platform. We therefore propose ViLiNT , a multimodal local motion planning architecture based on a transformer that fuses LiDAR, RGB images, 2D goal and embodiment data into a shared token space as shown in Fig. 1 . This approach targets both navigation quality by proposing efficient goal-reaching paths and safety by predicting robot-specific safety margins. Trajectories are generated with a goal-conditioned diffusion policy [ 5 ] and fed to the clearance prediction head conditioned by the robot embodiment. At inference, the system generates multiple candidate trajectories and selects the one with the minimum predicted risk (i.e. largest clearance), which is sent to the low-level controller. If no safe trajectory is found, an alternative sampling of exploratory trajectories is used to exit these local minima. Fig. 1 : Overall architecture of our navigation model. Highlighted blocks in red indicate our main contributions. See text for details. This design combines three complementary ideas: (i) multimodal token-level fusion to improve robustness; (ii) generative trajectory sampling to capture multi-modality in feasible plans; and (iii) an explicit clearance predictor trained with automatically generated labels from the training datasets. Both the generator and the clearance predictor are parameterized by the robot size, the result is a motion-planner that is both expressive and selective: it can propose diverse options adapted to the robot embodiment and then reject those likely to collide, without requiring dense human labels. Our key contributions are: • A LiDAR–RGB tokenization and cross-modal Transformer that integrates recent image frames, a LiDAR scan, goal coordinates, and platform description to produce robot’s dynamics and scene description tokens. • A Navigation data augmentation based on LiDAR data that enables clearance prediction learning without requiring data from all target platforms. • A Cross-attention clearance head that uses per-waypoint queries (with robot-size conditioning) over LiDAR tokens to rank diffusion-generated trajectories to select the safest one. II Related Work II-A Perception and planning Many state-of-the-art traversable-path generation systems comprise two main components: (i) perception–mapping pipelines that estimate a traversability/cost field from onboard sensing, and (ii) model-based planners that optimize trajectories with respect to that field under kinematic/dynamic constraints. Observation and mapping pipelines infer traversability from RGB-D/LiDAR — often fusing proprioception for self-supervision — and recent works learn off-road costmaps and visual traversability that integrate with standard navigation stacks (e.g., [ 3 ] ). On the planning side, reactive local planners such as TEB [ 19 ] , as well as trajectory-optimization/Model Predictive Control (MPC) methods (e.g., MPPI [ 27 ] ), remain strong baselines for smooth, constraint-aware motion. While such modular stacks are effective and interpretable, their performance typically hinges on careful platform-specific parameterization (e.g., dynamic limits, and sensor-specific parameters). As sensing setups and robot dimensions change, these parameters often need substantial tuning, and robustness can degrade under appearance/geometry shifts despite good nominal performance. Motivated by these limitations, recent work has explored learning richer priors from data to reduce manual cost design while keeping compatibility with classical controllers [ 10 ] . Our approach follows this trend: we encode observations, robot embodiment, and goals with a Transformer and generate trajectories via a diffusion policy, while remaining compatible with a robot-specific low-level controller. II-B End-to-end navigation models Vision-only end-to-end driving systems such as DAVE-2 [ 1 ] demonstrated that imitation from raw monocular imagery can produce lane-following and steering on-road, establishing the paradigm of mapping pixels to controls without intermediate costmaps. Building on this idea, goal-conditioned policies trained on broad, heterogeneous datasets (e.g., GNM/ViNT and follow-ups [ 21 , 22 , 25 ] ) show cross-robot/environment generalization and can output feasible trajectories, thereby reducing manual cost engineering while staying compatible with low-level controllers. However, purely visual policies degrade when observations lack salient features or contain unobserved regions due to limited field of view, which impacts collision-avoidance robustness under domain shift. LiDAR-centric approaches such as DTG [ 13 ] alleviate some visual brittleness but face sparsity and semantic ambiguity (e.g., thin obstacles). Both NoMaD [ 25 ] and DTG [ 13 ] employ diffusion models [ 5 ] to generate trajectories: DTG primarily uses diffusion as a distribution estimator, whereas NoMaD ranks candidate trajectories (e.g., by goal proximity) to select long-range plans—implicitly assuming generated trajectories are safe to track. We adopt diffusion for trajectory generation but pair it with an explicit clearance prediction head conditioned on multimodal tokens and embodiment, scoring candidates to improve robustness. II-C Multimodal navigation models For multimodal sensing in off-road navigation, Bird’s-Eye-View (BEV) fusion networks trained with self-supervision predict traversability and geometry at low latency; e.g., RoadRunner [ 7

Read the complete text