Distilling On-device Language Models for Robot Planning with Minimal Human Intervention

Zachary Ravichandran, Ignacio Hounie, Fernando Cladera · 2025

Read Distilling On-device Language Models for Robot Planning with Minimal Human Intervention by Zachary Ravichandran, Ignacio Hounie, Fernando Cladera (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17486.txt.

Introduction

Zachary Ravichandran and co-authors present a framework that shrinks large language model planners for robots into small models that run on the robot itself, in a 2025 paper. The catalog names three authors, while the head byline lists six, so the work is best read as a team product. Cloud models give robots strong reasoning and a natural chat interface. Field and factory sites, though, often lack steady links to the cloud.

Their framework makes its own training data. It invents tasks and settings, asks the big model for plans, and trains a compact model on those plans as a drop-in swap. Applied to three planners, it lifted a small model from a tenth of the big model's score to over nine tenths. The shrunk planners also crossed robot types, from ground to air, and room to field. All code, models, and data are shared.

Catalog record

CREATORZachary Ravichandran, Ignacio Hounie, Fernando Cladera
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17486

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distilling On-device Language Models for Robot Planning with Minimal Human Intervention Zachary Ravichandran, Ignacio Hounie, Fernando Cladera Abstract Large language models (LLMs) provide robots with powerful contextual reasoning abilities and a natural human interface. Yet, current LLM-enabled robots typically depend on cloud-hosted models, limiting their usability in environments with unreliable communication infrastructure, such as outdoor or industrial settings. We present prism , a framework for distilling small language model (SLM)-enabled robot planners that run on-device with minimal human supervision. Starting from an existing LLM-enabled planner, prism automatically synthesizes diverse tasks and environments, elicits plans from the LLM, and uses this synthetic dataset to distill a compact SLM as a drop-in replacement of the source model. We apply prism to three LLM-enabled planners for mapping and exploration, manipulation, and household assistance, and we demonstrate that prism improves the performance of Llama-3.2-3B from 10-20% of GPT-4o’s performance to over 93% — using only synthetic data . We further demonstrate that the distilled planners generalize across heterogeneous robotic platforms (ground and aerial) and diverse environments (indoor and outdoor). We release all software, trained models, and datasets at https://zacravichandran.github.io/PRISM . Keywords: LLM-enabled robots, LLM Distillation 1 Introduction Large language Models (LLMs) have emerged as powerful tools for robot planning by providing contextual reasoning and a natural human interface. LLM-enabled robots have been successfully deployed for mobile manipulation [ 1 , 2 , 3 ] , autonomous driving [ 4 , 5 ] , service robots [ 6 ] , navigation [ 7 , 8 ] , and inspection [ 9 , 10 ] . Being trained on internet-scale data, LLMs contain rich semantic knowledge and are highly adaptable. With only a few in-context examples, they can attune to task-relevant information and generalize to novel robotic scenarios. However, the scale of LLMs brings significant computational demands. Current frontier models contain hundreds of billions of parameters and require significant cloud computing infrastructure [ 11 , 12 ] . Although small language models (SLMs) can run on-device, their relatively poor performance in robot planning domains has precluded widespread adoption by the community [ 13 ] . Consequently, nearly all LLM-enabled robots rely on models such as GPT or PALM [ 14 , 1 , 3 , 9 , 4 , 15 , 16 , 6 ] , which tethers them to external compute. While this dependence may be feasible in structured environments such as homes or offices, it severely limits deployment in large-scale, unstructured settings — domains critical for environmental monitoring, industrial inspection, disaster response, and a host of similar problems. To address these limitations, we introduce prism (Planning with Robotic dIstilled Small language Models), a framework for distilling SLMs that run on-device with minimal human intervention. Starting from an existing LLM-enabled planner (termed “source”), prism generates diverse tasks and textual environment representations — such as scene graphs or object sets — and uses these generations to elicit plans from the source LLM. This interaction produces high-quality planning demonstrations without manual annotations, simulators, or dataset curation. The user simply provides the LLM-enabled planner along with its action and observation spaces. prism then uses the generated dataset to fine-tune an SLM that shares the original planner’s interface, enabling it to directly replace the source LLM. We outline prism in Fig. 1 , we summarize prism ’s contributions : • Effectiveness. prism distills SLMs that rival the planning performance of a source LLM-enabled planner, such as GPT-4o, as measured by planning success rate. • On-device models. prism effectively distills language models that are small enough to run onboard robot compute. In this work, we consider models with a memory footprint of under 5GB. • Minimal human interaction. prism requires a high-level configuration comprising the source planner’s observation space, action space, etc. , but it does not require manual dataset construction, simulators, details about the deployment environment, or other labor intensive information. We evaluate prism across three LLM-enabled planners — SPINE on real-world air and ground robots [ 9 ] , LLM-Planner in the ALFRED simulator [ 16 , 17 ] , and SayCan in a manipulation simulator [ 18 ] . While undistilled SLMs achieve only 10-20% of GPT-4o’s success rate, prism boosts performance to over 93% across all three settings. Moreover, the distilled SLMs compute efficiency enables them to plan onboard a robot in closed-loop control settings. We also release datasets, models, and code in order to promote reproducibility. The rest of the paper is structured as follows. We overview related work in § 2 , describe our framework in § 3 , present experimental results in § 4 , provide concluding remarks in § 5 , and discuss limitations in § 6 . Figure 1: PRISM distills on-device language models for robot planning with minimal human intervention. Given an existing LLM-enabled planner, prism synthesizes diverse task and textual environments, elicits plans from the LLM, then uses the resulting dataset to fine-tune an SLM that rivals the performance of the original planner, while being extremely compute efficient. We demonstrate prism -distilled models across platforms and environments. 2 Related work LLM-Enabled Robots. LLMs have emerged as a powerful enabler of contextually-aware robots, as evidenced by their deployment in mobile manipulation [ 1 , 3 , 2 , 18 , 19 ] , service robots [ 6 ] , task and motion planning [ 20 ] , autonomous driving [ 21 , 4 ] , anomaly detection [ 22 , 23 ] , navigation [ 14 , 8 , 7 , 24 ] , exploration and mapping [ 9 , 10 ] , and multi-robot teaming [ 25 ] . These works forego any model training; instead, they configure a pre-trained LLM via in-context examples. While convenient and effective, this approach requires the use of extremely large models (over 100B parameters) — all the above works use GPT-4 [ 11 ] , PALM [ 26 ] , or similar — which are far too large to run onboard a robot and therefore tethers the robot to high-quality network infrastructure. Distilling LLMs. LLM distillation aims to retain the contextual reasoning abilities of LLMs while overcoming their compute limitations. These methods train an SLM to emulate the generations of a source LLM via supervised fine tuning (SFT) [ 27 , 28 , 29 ] , reinforcement learning [ 30 , 31 ] , and ranking optimization [ 32 , 33 ] , among other methods. By finetuning a target SLM rather than training from scratch, these methods enjoy significant compute and data efficiency [ 34 ] . While traditional distillation methods curate a dataset of queries with which to elicit LLM generations, an emerging line of self-distillation research uses an LLM to generate both queries and responses [ 35 , 36 , 37 ] , thereby reducing the need for dataset curation. Although self-distillation research has focused on training models in large-scale virtual domains, our work builds upon its insights in order to synthesize data that results in efficient planners that are grounded in the physical world . Distilling LLMs for Planning. Distillation has been applied to virtual planning domains, including web search or video games [ 38 , 39 , 40 , 41 , 42 ] . These methods typically use

Read the complete text