Visual Prompt Based Reasoning for Offroad Mapping and Navigation using Multimodal LLMs

Abdelmoamen Nasser, Yousef Baba’a, Murad Mebrahtu, Nadya Abdel Madjid, Jorge Dias, Majid Khonji · 2026

Read Visual Prompt Based Reasoning for Offroad Mapping and Navigation using Multimodal LLMs by Abdelmoamen Nasser, Yousef Baba’a, Murad Mebrahtu, Nadya Abdel Madjid, Jorge Dias, Majid Khonji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04564.txt.

Description

Nasser and colleagues present a zero-shot off-road approach using SAM2 for environment segmentation and a vision-language model to reason about drivable areas. The team passes the original and segmented images, annotated with numeric labels per mask, to the vision-language model, which identifies drivable regions by label. With planning and control modules, the unified framework removes terrain-specific models, relying on the vision-language model's inherent reasoning. It surpasses trainable models on high-resolution segmentation datasets and enables full navigation in an Isaac Sim off-road environment.

Catalog record

CREATORAbdelmoamen Nasser, Yousef Baba’a, Murad Mebrahtu, Nadya Abdel Madjid, Jorge Dias, Majid Khonji
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38924
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04564

Browse the full library · About the Tower of Babel