Visual Prompt Based Reasoning for Offroad Mapping and Navigation using Multimodal LLMs
Abdelmoamen Nasser, Yousef Baba’a, Murad Mebrahtu, Nadya Abdel Madjid, Jorge Dias, Majid Khonji · 2026
Read Visual Prompt Based Reasoning for Offroad Mapping and Navigation using Multimodal LLMs by Abdelmoamen Nasser, Yousef Baba’a, Murad Mebrahtu, Nadya Abdel Madjid, Jorge Dias, Majid Khonji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04564.txt.
Description
Nasser and colleagues present a zero-shot off-road approach using SAM2 for environment segmentation and a vision-language model to reason about drivable areas. The team passes the original and segmented images, annotated with numeric labels per mask, to the vision-language model, which identifies drivable regions by label. With planning and control modules, the unified framework removes terrain-specific models, relying on the vision-language model's inherent reasoning. It surpasses trainable models on high-resolution segmentation datasets and enables full navigation in an Isaac Sim off-road environment.
Catalog record
| CREATOR | Abdelmoamen Nasser, Yousef Baba’a, Murad Mebrahtu, Nadya Abdel Madjid, Jorge Dias, Majid Khonji |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38924 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04564