RT-Cache: Training-Free Retrieval for Real-Time Manipulation

Owen Kwon, Abraham George, Alison Bartsch, Amir Barati Farimani · 2025

Read RT-Cache: Training-Free Retrieval for Real-Time Manipulation by Owen Kwon, Abraham George, Alison Bartsch, Amir Barati Farimani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09040.txt.

Introduction

Owen Kwon, Abraham George, Alison Bartsch, and Amir Barati Farimani replace robot thinking with robot remembering in a 2025 robotics paper. Their system, RT-Cache, stores many recorded image-and-action trajectories in one searchable memory. Facing a new scene, the robot embeds what its camera sees, retrieves the closest stored episodes, and replays their action snippets. No training and no per-step model calls are needed.

A hierarchical search keeps lookups under a second even at million scale, trading compute for storage. On real robots and large public logs it roughly doubled success rates and cut task time by about thirty percent against strong retrieval rivals. One anchoring episode even bootstrapped a harder, contact-rich task.

Catalog record

CREATOROwen Kwon, Abraham George, Alison Bartsch, Amir Barati Farimani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-09040

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RT-Cache: Training-Free Retrieval for Real-Time Manipulation Owen Kwon, Abraham George, Alison Bartsch, Amir Barati Farimani Abstract Real robots are expected to repeat the same behavior in new environments with very little new data, yet modern controllers either incur heavy per‑step inference or require deployment‑time fine‑tuning. We propose RT‑Cache , a training‑free retrieval‑as‑control pipeline that caches diverse image–action trajectories in a unified vector memory and, at test time, embeds the current frame to retrieve and replay multi‑step snippets, replacing per‑step model calls. A hierarchical search keeps lookups sub‑second at million scale, shifting cost from compute to storage and enabling real‑time control on modest GPUs. Across real‑robot tasks and large open logs, RT‑Cache achieves higher success and lower completion time than strong retrieval baselines (approximately 2 × 2\times higher success and ∼ 30 % \sim 30\% faster in our settings), and a single‑episode anchoring study shows immediate adaptation to a more complex, contact‑rich task without fine‑tuning. RT‑Cache turns experience into an append‑only memory, offering a simple, scalable path to few‑shot deployment today and a foundation for multimodal keys and optional integration with high‑level policies. Project page: https://rt-cache.github.io/ . Keywords: Big Data, Robot System, Learning from Experience Fig. 2: RT‑Cache retrieval—from terabytes to real time. (1) Trajectory Database. Heterogeneous logs are standardized (sample‑rate normalization, EEF Cartesian normalization, data filtering). Each image–action pair is stored in a Trajectory DB and embedded into a Vector DB . (2) Trajectory Retrieval. For each new observation, we embed the image and run a hierarchical search: dataset‑centroid filtering \rightarrow sampled local index \rightarrow cosine k ‑NN over candidates. The selected snippet’s action vectors are replayed for the next N steps to command the robot, then the system re‑queries, forming a closed loop. This training‑free design replaces per‑step inference and keeps lookups sub‑second at million‑scale while adapting by simply appending new episodes to memory. I INTRODUCTION Robots are increasingly asked to repeat the same behavior in new environments with very little new data . How can we deliver this in practice without heavy per‑task training or fine‑tuning, while keeping latency low enough for real hardware? A natural candidate is the family of Vision–Language–Action (VLA) models, which offer a unified interface for perception and control across embodiments [ 1 , 2 , 3 , 4 , 5 ] . Yet deployment remains costly: each control step often requires a large forward pass, inflating wall‑clock time and GPU demand; and adapting to unseen scenes typically requires task‑ or site‑specific fine‑tuning with non‑trivial data and compute. Even with speedups such as parallel decoding [ 5 ] , substantial computation at every step hampers real‑time use on modest hardware or multi‑camera humanoid setups. A complementary direction is to reuse past experience via retrieval [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . However, imitation‑style and policy‑retrieval methods typically assume a trained policy or adapters and focus on single‑step selection on relatively small corpora; at deployment they often still require task‑specific fine‑tuning, and they rarely exploit multi‑step reuse from large, heterogeneous datasets. As robot logs grow by orders of magnitude [ 16 ] , a system that scales retrieval to millions of states and replays meaningful action segments— without extra training—would bridge an important gap. Our RT-Cache system reframes deployment as remember and replay : cache successful real‑robot experience at scale and retrieve multi‑step snippets on the fly. RT‑Cache maintains a large Memory , millions of image–action pairs, in a unified vector database built from diverse trajectories. At runtime, the current camera frame is embedded into a shared space and compared against this memory; the top match (or an aggregate of the top- K ) provides the next N actions. In effect, retrieval becomes the controller , the robot executes the replayed sequence rather than generating each step with a large policy. Crucially, RT‑Cache is training‑free at deployment : incorporating new scenes requires only adding their embeddings to the memory, no backpropagation, no per‑task fine‑tuning. To keep lookups tractable on very large corpora, RT‑Cache uses a multi‑stage retrieval strategy . We first narrow the search by dataset‑level centroids (or equivalent partitions), then query a small, sampled local index before a final k ‑NN over candidates. This hierarchical filter turns exhaustive, minutes‑long global searches into sub‑second queries while preserving accuracy, enabling real‑time control . Because the memory grows with use, even a few in‑domain examples can immediately anchor retrieval in a new environment, enabling few‑shot adaptation without model updates. This retrieval‑as‑control paradigm achieves low‑latency, practical deployment without any task‑specific optimization or encoder fine‑tuning. Contributions. • RT-Cache , a deployment‑time, training‑free pipeline that retrieves and replays multi‑step actions from a large repository of real‑robot experience, effectively turning retrieval into the controller. • A scalable, multi‑stage retrieval procedure (dataset‑centroid selection + small‑subset indexing + final k ‑NN) that keeps lookup times sub‑second at million‑scale. • A unified trajectory memory (vector database) spanning heterogeneous sources (Open‑X and additional real‑world logs) via standardized action representation and embeddings, enabling immediate few‑shot coverage by simply inserting new samples. • Real‑robot experiments demonstrating that RT‑Cache cuts wall‑clock operation time while maintaining or improving success; adding even a handful of in‑domain examples can flip zero‑shot failures into successes. II RELATED WORKS II-A Vision–Language–Action (VLA) Models in Robotics Generalist VLA models have demonstrated strong cross‑task and cross‑embodiment generalization [ 1 , 2 , 3 , 4 , 5 ] , typically by pretraining on large, heterogeneous robot/vision corpora and then adapting to downstream settings. Despite this promise, three deployment pain points are common: (1) Per‑step compute : each control step usually requires a large forward pass; even with engineering such as parallel decoding [ 5 ] , substantial computation occurs at every timestep, constraining control rates and increasing wall‑clock latency. (2) Unseen‑environment fine‑tuning : to reach reliable performance in new scenes (camera pose, lighting, workspace geometry), VLAs are often adapted with LoRA adapters or full fine‑tuning, which demands non‑trivial numbers of labeled steps and incurs additional iteration time. (3) Resource footprint : deployment frequently requires tens of GB of GPU memory and tight I/O budgets, making continuous per‑step inference expensive on modest hardware or in multi‑camera humanoid settings. A further concern is continual‑learning side effects : repeated task‑specific updates can interfere with previously acquired skills [ 17 ] . These factors have motivated complementary directions that lessen online compute and reduce (or eliminate) deployment‑time training while retaining broad task coverage. II-B Retrieval‑Based Action Reuse and Planning Retrieval methods reuse prior experience by matching the current observation to stored trajectories and either copying actions or selecting data for a learned controller [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . Image‑based approaches such as VINN perform per‑timestep RGB k ‑NN and copy neighbor actions, yielding simple, training‑light deployment but suffering from single‑step drift and sensitivity to viewpoint/scale; most reports target modest index sizes rather than million‑scale, multi‑step replay [ 6 ] . Behavior Retrieval instead uses retrieval to weight/select demonstrations for offline behavior cloning, leveraging large corpora but requiring task‑specific training/fine‑tuning and providing no test‑time retrieval, which can leave deployment mismatch uncorrected [ 7 ] . Language‑guided retrieval fetches demos or skills from

Read the complete text