OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction

Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, Pieter Abbeel · 2025

Read OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction by Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, Pieter Abbeel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03734.txt.

Introduction

Huang Huang and co-authors present OTTER, a robot control model that reads instructions with its eyes open in a 2025 paper. Robot policies often take vision and language as separate streams, which dulls the match between words and sights. Fine-tuning to fix that can erase pre-trained knowledge.

OTTER instead pulls only the visual features that match the instruction, using alignments the base model already learned. Its vision and language encoders stay frozen, keeping their broad knowledge intact. Simulations and real robot tests show strong wins over rival policies. Fresh objects and rooms were handled without new training, a prized zero-shot trait.

Catalog record

CREATORHuang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, Pieter Abbeel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, Pieter Abbeel Abstract Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions. Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies, degrading the pre-trained semantic alignments. We propose OTTER, a novel VLA architecture that leverages these existing alignments through explicit, text-aware visual feature extraction. Instead of processing all visual features, OTTER selectively extracts and passes only task-relevant visual features that are semantically aligned with the language instruction to the policy transformer. This allows OTTER to keep the pre-trained vision-language encoders frozen. Thereby, OTTER preserves and utilizes the rich semantic understanding learned from large-scale pre-training, enabling strong zero-shot generalization capabilities. In simulation and real-world experiments, OTTER significantly outperforms existing VLA models, demonstrating strong zero-shot generalization to novel objects and environments. Video, code, checkpoints, and dataset: https://ottervla.github.io/ . † † affiliationnotice: Equal contribution 1 Introduction Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have inspired the exploration of scaling datasets and computational resources for vision-language-action (VLA) models ( Collaboration et al. 2024 ; Khazatsky et al. 2024 ; Octo Model Team et al. 2024 ; Kim et al. 2024 ) . Different input modalities are usually encoded into separate tokens: multi-view images encoded via visual feature extractors, along with tokenized language instructions, optionally with the robot’s proprioceptive states, are fed into a transformer-based robot policy for end-to-end action generalization. This approach requires the policy network to connect the vision and language information and conduct precise robot control, which often presents significant challenges, especially in unseen environments. Figure 1 : (Top) Different feature extraction approaches in VLA models. (a) Direct Feature Passing: existing approaches, exemplified by Octo and OpenVLA, extract and pass visual and text tokens independently to the policy network. (b) Text-Aware Feature Extraction: the proposed approach, OTTER, extracts visual tokens that correspond to the text tokens, and then feeds them into the policy. (Bottom) Real-world Robot Experiments: OTTER demonstrates higher success rates on both training and unseen real-world robot pick-and-place tasks compared to Octo and OpenVLA. OTTER exhibits better zero-shot generalization to unseen objects, maintaining strong performance across a variety of novel tasks. Existing works such as RT-2 ( Brohan et al. 2023 ) and OpenVLA ( Kim et al. 2024 ) have demonstrated the benefits of directly fine-tuning pre-trained VLMs on robotic datasets to map vision and language to control. While these approaches leverage rich visual and language features from pre-trained encoders, this fine-tuning may interfere with pre-trained vision and language features, particularly since robot datasets are far less semantically diverse compared to large vision-language datasets ( Schuhmann et al. 2022 ) where these VLMs are trained on, often leading to a noticeable drop in performance on unseen objects or environments compared to seen tasks ( Brohan et al. 2023 ) . This finding is consistent with observations in vision-language research ( Kerr et al. 2023 ; Lan et al. 2024 ) , where fine-tuning language-aligned vision encoders, such as CLIP ( Radford et al. 2021 ) , has been shown to overfit and degrade generalization and long-tail classification performance. Yet, models like CLIP and SigLIP ( Zhai et al. 2023 ) already exhibit strong vision-language alignment and impressive zero-shot performance on various downstream tasks, including fine-grained tasks like open-vocabulary segmentation ( Rao et al. 2022 ; Lan et al. 2024 ; Dong et al. 2023 ) . This raises the question of whether a more effective strategy is to extract and utilize the pre-trained alignment rather than risk weakening it through fine-tuning. Figure 2 : OTTER Model architecture. At each timestep t , text-aware visual features f_{vl} are extracted from a pre-trained CLIP model (see Figure 3 ). Then f_{vl} and the text tokens f_{l} are further compressed through separate attention pooling layers, producing two representations f^{\prime}_{vl} and f^{\prime}_{l} , respectively. The robot’s proprioception is encoded by an MLP layer to generate the embodiment representation f_{e} . The tokens f^{\prime}_{l} , f^{\prime}_{vl} , and f_{e} are then concatenated to form f_{t} , which serves as input to a causal transformer. With a context window of T steps, the model autoregressively predicts the future actions ( a_{t} ) at each step. We seek to preserve the generalizability of VLMs for effective performance in unseen scenarios. To this end, we propose OTTER, a novel VLA architecture that freezes pre-trained vision and language encoders and extracts task-relevant visual features guided by language instructions. Instead of passing all visual features to the policy network, OTTER selectively extracts those semantically aligned with the task description. We use CLIP for its zero-shot capabilities and wide adoption, employing a lightweight selection mechanism to preserve its pre-trained vision-language understanding while adapting it for robotic control. Figure 2 illustrates the architecture of OTTER, which incorporates text-aware feature extraction into the vision-language-action (VLA) pipeline. At each timestep, OTTER first processes visual and textual inputs to pinpoint task-relevant visual tokens. These selected features, along with proprioceptive data, enable the policy network to concentrate specifically on action planning. By keeping the pre-trained vision-language encoders frozen, OTTER effectively decouples task planning (selecting relevant visual features) from robot action planning (predicting appropriate actions). Both physical and simulation experiments demonstrate that OTTER outperforms existing VLA models, showing strong generalization to novel objects and environments with less performance degradation ( Figure 1 ). To summarize, our contributions are: 1. We propose OTTER, a VLA model that leverages the semantic alignment capabilities of pre-trained VLMs for better generalization. By extracting text-aware visual features that are semantically aligned with task instructions, OTTER preserves and utilizes the rich visual-language understanding from pre-training for effective robotic task execution. 2. OTTER significantly outperforms state-of-the-art VLA models on unseen robot manipulation tasks through its zero-shot generalization capabilities. By preserving the frozen pre-trained vision-language model rather than fine-tuning, OTTER effectively leverages the semantic understanding from large-scale pre-training to handle novel objects and environments. 3. Empirical results suggest that OTTER’s performance on unseen tasks scales along multiple axes: through larger pre-trained vision-language encoders, increased policy network capacity, and pre-training on larger robot datasets. 2 Related Work 2.1 Vision Language Pre-training Vision-language pre-training (VLP) seeks to improve the performance of downstream tasks that involve both vision and language by training models on extensive datasets of image-text pairs. A prominent class of vision-language models leverages contrastive learning ( Alayrac et al. 2020 ; Cherti et al. 2023 ; Jia et al. 2021 ; Radford et al. 2021 ; Yao et al. 2021 ; Yuan et al. 2021 ; Zhai et al. 2023 ) . Among them, CLIP ( Radford et al. 2021 ) , which was trained on a private WIT-400M dataset of image-text pairs, demonstrates impressive zero-shot capabilities across various downstream tasks, including image-text retrieval and image classification through text prompts. Furthermore, CLIP shows potential for application in broader fields such as decision making and robotics, where robots are required to perform language-specified tasks based on visual inputs. Recent multimodal foundation models, such Qwen-VL ( Bai et al. 2023 ) and Llama 3-V ( Dubey et al. 2024 ) , all follow

Read the complete text