Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action Models
Rokas Bendikas Centre for Artificial Intelligence, UCL [email protected], Daniel DijkmanQualcomm AI Research [email protected] Markus PeschlQualcomm AI ResearchSanjay HareshQualcomm AI ResearchPietro MazzagliaQualcomm AI Research · 2025
Read Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action Models by Rokas Bendikas Centre for Artificial Intelligence, UCL [email protected], Daniel DijkmanQualcomm AI Research [email protected] Markus PeschlQualcomm AI ResearchSanjay HareshQualcomm AI ResearchPietro MazzagliaQualcomm AI Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23655.txt.
Introduction
Robot vision systems spend most of their tokens on background that the task ignores. Rokas Bendikas and 4 co-authors cut that waste with Oat-VLA, a tokenization scheme centered on objects and the agent itself. Vision-language-action models normally inherit heavy visual token loads from their parent models. The new scheme builds in a bias toward scene objects and the robot's own appearance.
With only a few visual tokens left, performance held up. Training converged at least twice as fast as OpenVLA on the LIBERO suite. The method also beat OpenVLA on varied real-world pick-and-place tasks, which the authors cite as evidence that leaner visual input need not mean weaker control.
Catalog record
| CREATOR | Rokas Bendikas Centre for Artificial Intelligence, UCL [email protected], Daniel DijkmanQualcomm AI Research [email protected] Markus PeschlQualcomm AI ResearchSanjay HareshQualcomm AI ResearchPietro MazzagliaQualcomm AI Research |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23655
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Focusing on What Matters: Object-Agent-centric Tokenization for Vision Language Action Models Rokas Bendikas Centre for Artificial Intelligence, UCL [email protected], Daniel DijkmanQualcomm AI Research [email protected] Markus PeschlQualcomm AI ResearchSanjay HareshQualcomm AI ResearchPietro MazzagliaQualcomm AI Research Abstract Vision-Language-Action (VLA) models offer a pivotal approach to learning robotic manipulation at scale by repurposing large pre-trained Vision-Language-Models (VLM) to output robotic actions. However, adapting VLMs for robotic domains comes with a high computational cost, which we attribute to the tokenization scheme of visual inputs. In this work, we aim to enable more efficient VLA training by proposing Oat-VLA, an Object-Agent-centric Tokenization for VLAs. Building on the insights of object-centric representation learning, our method introduces an inductive bias towards scene objects and the agent’s own visual information. As a result, we find that Oat-VLA can drastically reduce the number of visual tokens to just a few tokens without sacrificing performance. We reveal that Oat-VLA converges at least twice as fast as OpenVLA on the LIBERO suite, as well as outperforms OpenVLA in diverse real-world pick and place tasks. Keywords: visual-language-action models, object-centric representations, robotic manipulation, imitation learning Figure 1: Comparison of our method Oat-VLA to OpenVLA on the LIBERO dataset, full fine-tuning. Left : Action token accuracy relative to training time. Oat-VLA converges more than 2\times faster . Right : Average success rate across the four LIBERO tasks suites relative to training steps. 1 Introduction Large language models (LLMs) [ 1 , 2 ] have made it possible to tackle increasingly difficult tasks by leveraging self-supervised learning on large-scale datasets. Furthermore, the combination of the vast amount of knowledge present in modern pre-trained LLMs and the large amount of parameters have opened up many possibilities for adapting LLMs to a variety of applications, such as coding [ 3 ] , reinforcement learning [ 4 ] and imitation learning [ 5 ] . A key ingredient thereof lies in the appropriate tokenization of the input data, which can be straightforward for natural language [ 6 ] , but is generally significantly more powerful when leveraging the structure of the input [ 7 ] . While the latter has become a common strategy for language-valued inputs, the former choice of a simple tokenizer which is agnostic to input semantics has remained prevalent when training LLMs to additionally process visual information [ 8 , 9 ] . Multimodal capabilities are the cornerstone of real-world applications including robotics [ 10 ] . Recently, Vision-Language-Action models (VLAs) have gained significant traction [ 11 , 5 , 10 , 12 , 13 ] due to their ability to adapt general knowledge to an imitation-learning setting. VLAs typically take images from cameras in the environment and text, e.g., language instructions, to predict robotic actions, allowing multimodal prompting of agents for robotic manipulation. Despite the success of VLAs, they rely on expensive pre-training on specialized robotics datasets such as Open X-Embodiment [ 12 , 10 , 13 ] to generalize to new tasks. Unfortunately, training VLAs at this scale requires large amounts of compute resources, unavailable to many research laboratories. We identify the visual processing of current state-of-the-art VLAs [ 10 ] as one of the main bottlenecks that significantly increase the compute requirements of training VLAs. This process generally consists of dividing the image into (hundreds of) patches [ 14 ] , processing them through a visual encoder model, and then feeding the outputs as visual tokens to the LLM. As we show in Figure 1 , this visual tokenization scheme, which so far is the most popularly adopted, can be made fundamentally more efficient without compromising performance on robotics tasks. Our key insight is that task execution given a language instruction requires focus on specific parts of the input observation such as objects of interest, while background information is less relevant and can therefore be discounted. In this work, we present an Object-Agent-centric Tokenization scheme for VLAs (Oat-VLA), which adaptively tokenizes important parts of the scene into a reduced set of visual tokens to feed to the VLA. By producing fewer, but more meaningful tokens, Oat-VLA significantly reduces memory and compute requirements. Furthermore, we designed Oat-VLA’s architecture to be able to leverage the pre-trained knowledge already contained in OpenVLA [ 10 ] to maintain easy adaptation capabilities and high fine-tuning performance, which match or even outperform heavier VLA training schemes. Our contributions can be summarized as follows: • Inspired by object-centric representations, we design a method to condense visual information about objects into a reduced set of tokens; • We design a simple approach to identify agent-related visual patches and concatenate them with object-centric tokens to ensure precision of the VLA when predicting actions; • We show that the overall model, Oat-VLA, is modular and scalable, so that it can be efficiently adapted by reusing existing VLA checkpoints. We assess the performance of our method on the LIBERO benchmark and demonstrate a 2x speedup in terms of convergence according to the action token accuracy and success rate on tasks; • Finally, we tested Oat-VLA on real-world pick and place tasks and find that it executes more robustly than OpenVLA, translating into a higher success rate. 2 Related work Vision-language-action models. There has been an increasing trend in leveraging generalization capabilities of Vision-Language-Models (VLMs) for robotic control by fine-tuning these VLMs on robotics tasks [ 15 , 10 , 16 ] . RT-2 [ 15 ] utilizes frontier closed-sourced VLMs by co-fine-tuning them on visual and robotics tasks to achieve a general purpose robotics policy. At the same time, OpenVLA [ 10 ] and \pi 0 [ 16 ] embody similar open-source efforts. Preceding these, there have been multiple efforts at using the transformer architecture trained on large robotics dataset to achieve general purpose robotics policies. Gato [ 17 ] presented the first transformer trained on a multitude of language, vision and robotics tasks whereas RT-1 [ 11 ] presented a first concentrated effort at using large transformer architectures trained on a large robotics dataset. Octo [ 13 ] and JAT [ 18 ] present open-source transformer policies designed for efficient fine-tuning on down-stream tasks. Consequently, there have been multiple efforts at incorporating spatial awareness [ 19 ] , 3D information [ 20 ] , generalization to new objects [ 21 ] , recurrence and streaming ability [ 22 , 23 , 24 , 25 ] and embodied chain-of-thought [ 26 , 27 ] to VLAs to improve generalization. In our work, we focus on improving the visual representation of VLAs for compute efficiency reasons, while orthogonal works [ 28 , 29 ] have shown significant progress by improving the action prediction component. Efficient VLMs, VLAs. Concurrent to improving generalization capability of VLAs, a number of works have explored improving efficiency of these models and the VLMs on which they build. Related work TokenLearner [ 30 ] introduces a generic learned method to reduce the number of tokens, while other works [ 31 , 32 ] increase the efficiency of the attention layers in transformers. TinyVLA [ 33 ] presents a VLA with a smaller backbone and diffusion-based action heads for faster training and inference speeds. Similarly, MiniVLA [ 34 ] trains a performant VLA using a 1B parameters models. VLA-Cache [