LLM-Based Agentic Exploration for Robot Navigation & Manipulation with Skill Orchestration

Abu Hanif Muhammad Syarubany, Farhan Zaki Rahmani, Trio Widianto · 2026

Read LLM-Based Agentic Exploration for Robot Navigation & Manipulation with Skill Orchestration by Abu Hanif Muhammad Syarubany, Farhan Zaki Rahmani, Trio Widianto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00555.txt.

Introduction

Abu Hanif Muhammad Syarubany, Farhan Zaki Rahmani, and Trio Widianto describe a shopping robot in a 2026 paper. The robot works indoors and takes orders in plain speech. As it moves, it builds a light map of the place. The map notes signs at each fork and which way each shop lies. Printed tags on the walls give the robot fixed points to line up on.

A large language model reads each order and picks the robot's next move. At each fork it picks a way to turn, and whether to enter a shop. A control program then runs the move with set motion skills. Those skills dodge blocks, line up on tags, enter shops, and grasp goods. The team tested the stack in a computer model of a store. They also tested it in a real hall. The robot ran whole tasks, from order to goods in hand.

Catalog record

CREATORAbu Hanif Muhammad Syarubany, Farhan Zaki Rahmani, Trio Widianto
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00555

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-Based Agentic Exploration for Robot Navigation & Manipulation with Skill Orchestration Abu Hanif Muhammad Syarubany, Farhan Zaki Rahmani, Trio Widianto Abstract This paper presents an end-to-end LLM-based agentic exploration system for an indoor shopping task, evaluated in both Gazebo simulation and a corresponding real-world corridor layout. The robot incrementally builds a lightweight semantic map by detecting signboards at junctions and storing direction-to-POI relations together with estimated junction poses, while AprilTags provide repeatable anchors for approach and alignment. Given a natural-language shopping request, an LLM produces a constrained discrete action at each junction (direction and whether to enter a store), and a ROS finite-state main controller executes the decision by gating modular motion primitives, including local-costmap-based obstacle avoidance, AprilTag approaching, store entry, and grasping. Qualitative results show that the integrated stack can perform end-to-end task execution from user instruction to multi-store navigation and object retrieval, while remaining modular and debuggable through its text-based map and logged decision history. I Introduction Recent progress in visual perception, semantic mapping, and large language models (LLMs) is enabling mobile robots to follow high-level instructions rather than fixed waypoints. The key challenge is integration: a robot must perceive discrete cues, maintain a semantic environment, and translate language-level goals into safe, executable motions in real-time. In this project, we study an agentic exploration task in a corridor-based shopping environment with multiple store categories and a pickup point, evaluated in both Gazebo [ 7 ] and a corresponding real-world setup. Information about store locations is not given as ground-truth coordinates; instead, it is distributed across junction signboards containing directional arrows, store icons, and AprilTags [ 12 ] . The robot must explore, build a semantic map from these cues, and decide which direction to take at each junction to satisfy a user-provided shopping order. We implement an end-to-end ROS stack [ 13 ] that couples an LLM decision layer with modular low-level controllers. ORB-SLAM3 [ 3 ] provides global pose, AprilTags [ 12 ] act as stable anchors for approach behaviors, and perception nodes write a lightweight JSON semantic map of junction-to-POI relations. Given the mission, current junction context, and a history log, an LLM outputs a constrained discrete command <direction>|||<store_action> , which a finite-state MainController executes by gating motion primitives (wall avoidance, tag approach, store entry, and grasping). II Related Work Metric localization and mapping are core requirements for indoor mobile robots. Feature-based visual SLAM systems such as ORB-SLAM provide accurate monocular pose tracking with keyframe mapping and loop closure [ 10 ] . ORB-SLAM2 extends this pipeline to stereo and RGB-D sensing while maintaining real-time performance [ 11 ] . ORB-SLAM3 further improves robustness by supporting visual inertial sensing and multi-map operation, which is useful in re-localization after tracking loss [ 3 ] . In our stack, we rely on SLAM to provide a continuous global pose estimate that downstream modules can use for logging motion primitives. Fiducial markers are widely used to complement SLAM with strong, repeatable geometric anchors. AprilTag provides a robust, open fiducial system that enables accurate pose estimation from a single camera view [ 12 ] , and AprilTag 2 improves detector efficiency and robustness in practical settings [ 14 ] . In our scenario, AprilTags embedded in signboards serve as reliable junction/entrance references that help stabilize the semantic map and support precise approach behaviors. Beyond purely geometric maps, semantic representations associate places with symbolic meaning (e.g., store categories) so that tasks can be solved in terms of “what” and “where” rather than only metric coordinates. Surveys on semantic mapping in mobile robotics summarize common formulations and the trade-offs between metric, topological, and hybrid representations [ 8 , 4 ] . Our approach follows this line by maintaining a lightweight, text-based semantic map of junctions, directions, and store categories, which is convenient for both debugging and for prompting an LLM with structured context. Recent work also explores large language models as high-level planners or policies for embodied agents. One direction is to use language models to infer feasible action sequences or subgoals from structured state descriptions [ 5 ] . Other approaches explicitly ground language to robot affordances (e.g., mapping instructions into executable skills under constraints) [ 1 ] , or generate code-like policies that call lower-level controllers [ 9 ] . In contrast to end-to-end learned navigation, our project uses an LLM only as a discrete decision layer over a junction-store semantic graph; low-level motion is executed by separate ROS controllers (wall following, tag approaching, store entry, recovery), enabling modularity and safer execution. Fig. 1 : Camera setup and responsibilities used in the proposed system. The USB camera is used for signboard understanding and store-entry centering, whereas the RealSense depth camera supports local costmap construction, AprilTag detection, and object tracking. Fig. 2 : Map layout of the corridor-based shopping environment. Junction signboards (arrows, store icons, and AprilTags) support semantic mapping and navigation, and the shaded region on the right marks the pickup point. III Methodology (a) Simulated map (b) Real map Fig. 3 : Environment layouts used in this work. (a) Gazebo simulation and (b) the corresponding real-world setup. Both share the same corridor structure and junction signboards for evaluation. We developed our agentic exploration pipeline in a corridor-based shopping environment with consistent junction structure, using both a Gazebo simulation [ 7 ] and a corresponding real-world layout (Fig. 2 & Fig. 3 ). The system is designed to build a semantic map from signboard cues and execute language-conditioned tasks through an LLM decision layer coupled with ROS motion primitives. Our sensing setup uses a two-camera configuration (Fig. 1 ), where an RGB USB camera supports signboard/store-icon perception and entry centering, while a RealSense RGB-D camera provides depth for local costmap construction, AprilTag perception [ 12 ] , and grasp-target tracking. TABLE I : Distribution of training and validation images per class for the YOLO dataset. Class Train Validation Total Negative 1335 (4.6%) 340 (4.6%) 1675 (4.6%) Cup 3258 (11.3%) 838 (11.2%) 4096 (11.3%) Mug 3287 (11.4%) 853 (11.4%) 4140 (11.4%) Hamburger 4851 (16.8%) 1258 (16.9%) 6109 (16.8%) Box 3283 (11.4%) 850 (11.4%) 4133 (11.4%) Direction-Left 1081 (3.7%) 278 (3.7%) 1359 (3.7%) Direction-Right 1105 (3.8%) 286 (3.8%) 1391 (3.8%) Direction-Straight 1142 (4.0%) 292 (3.9%) 1434 (3.9%) Pickup point 1581 (5.5%) 409 (5.5%) 1990 (5.5%) Cafe 1644 (5.7%) 428 (5.7%) 2072 (5.7%) Convenience store 1574 (5.5%) 406 (5.4%) 1980 (5.5%) Hamburger store 1575 (5.5%) 408 (5.5%) 1983 (5.5%) Pharmacy 1602 (5.5%) 413 (5.5%) 2015 (5.5%) Apriltag 1557 (5.4%) 395 (5.3%) 1952 (5.4%) Total 28875 (100%) 7454 (100%) 36329 (100%) Fig. 4 : System-level pipeline of our agentic exploration stack, including perception, localization, mapping, and an LLM reasoning core. The controller layer uses the semantic map and decision history to trigger low-level skills through a finite-state orchestrator. (a) Architecture flow (b) LLM agent flow Fig. 5 : Software pipeline of the proposed system. (a) ROS architecture flow from sensors and perception. (b) LLM agent flow, where a natural-language order to produce a constrained action <direction>|||<store_action> for robot execution. Algorithm 1 Gated AprilTag approaching with final forward advance. 1: Input: tag detections (x,z,\psi) , gate g , targets (x^{\ast},d^{\ast}) , tolerances (\tau_{x},\tau_{z},\tau_{\psi}) 2: Params: gains (k_{x},k_{z},k_{\psi}) , limits (v_{x,\max},v_{y,\max},\omega_{\max}) , timeout T , advance

Read the complete text