Pixel Motion as Universal Representation for Robot Control

Kanchana Ranasinghe, Xiang Li, E-Ro Nguyen, Cristina Mata, Jongwoo Park · 2025

Read Pixel Motion as Universal Representation for Robot Control by Kanchana Ranasinghe, Xiang Li, E-Ro Nguyen, Cristina Mata, Jongwoo Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07817.txt.

Introduction

Kanchana Ranasinghe and co-authors present a robot control framework that speaks in pixel motion. Their 2025 paper, on a system called LangToMo, splits control into two layers. The upper layer is an image diffusion model that, given one starting frame and a text instruction, forecasts how the pixels should move. Pixel motion serves as a universal middle language: readable, focused on movement, and easy to pull from ordinary videos with weak supervision.

The lower layer turns the forecast motion into robot actions, through mappings that can be hand-written or learned with little supervision. The upper policy runs at sparse intervals, steering; the lower runs densely, executing. Because the motion language does not depend on any one robot body, training can draw on any video-and-caption data. The authors argue the split gives control that is flexible, scalable, and quick to generalize.

Catalog record

CREATORKanchana Ranasinghe, Xiang Li, E-Ro Nguyen, Cristina Mata, Jongwoo Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07817

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pixel Motion as Universal Representation for Robot Control Kanchana Ranasinghe, Xiang Li, E-Ro Nguyen, Cristina Mata, Jongwoo Park Abstract We present LangToMo, a vision-language-action framework structured as a dual-system architecture that uses pixel motion forecasts as intermediate representations. Our high-level System 2 , an image diffusion model, generates text-conditioned pixel motion sequences from a single frame to guide robot control. Pixel motion—a universal, interpretable, and motion-centric representation—can be extracted from videos in a weakly-supervised manner, enabling diffusion model training on any video-caption data. Treating generated pixel motion as learned universal representations , our low level System 1 module translates these into robot actions via motion-to-action mapping functions, which can be either hand-crafted or learned with minimal supervision. System 2 operates as a high-level policy applied at sparse temporal intervals, while System 1 acts as a low-level policy at dense temporal intervals. This hierarchical decoupling enables flexible, scalable, and generalizable robot control under both unsupervised and supervised settings, bridging the gap between language, motion, and action. Checkout kahnchana.github.io/LangToMo Figure 1: Dual-System VLA Framework, LangToMo, with pixel motion representations. 1 Introduction Translating open-ended natural language instructions into robot actions is a cornerstone of flexible robot control. We identify two key requirements to enable this: (i) universal representations that support operating diverse embodiments ( Nair et al., 2022 ; Ren et al., 2025 ; Zheng et al., 2025 ) , and (ii) benefiting from video-language data without action labels ( Du et al., 2023b ; Gu et al., 2023 ; Black et al., 2023 ; Ko et al., 2023 ; Cheang et al., 2025 ; Lee et al., 2025 ) . We explore their intersection, proposing LangToMo, a vision–language–action framework structured as a dual-system architecture , inspired by dual-process theories of cognition ( Kahneman, 2011 ) and recent hierarchical robotics frameworks ( Belkhale et al., 2024 ; Black et al., 2024 ; Shi et al., 2025b ; Nvidia et al., 2025 ; Intelligence et al., 2025 ) . In our high level System 2 module, we use pixel motion as the robot action representation. We use image diffusion to learn to predict pixel motion from a single image (observation) conditioned on a language described action. Subsequently, our embodiment-aware low level System 1 deterministically projects these action representations into executable robot actions. We adopt pixel motion—the apparent motion of pixels between frames—as our universal motion representation , because it is agnostic to embodiments, viewpoints, and tasks. By predicting pixel motion instead of full RGB images, LangToMo captures essential motion patterns more efficiently (i.e. with less training data, see Section 4.1 ) than text-to-video generation ( Du et al., 2023b ; Ko et al., 2023 ; Gu et al., 2023 ; Black et al., 2023 ) . In contrast to operating with sparse point tracks ( Yuan et al., 2024a ; Wen et al., 2023 ; Xu et al., 2024 ; Bharadhwaj et al., 2024b ) , our dense pixel motion representation can capture both manipulator and object movements (see Figure 2 ). Our pixel motion features also retain the inherent 2D structure of the visual domain unlike prior work modeling pixel trajectories as 1D point tracks ( Wen et al., 2023 ; Xu et al., 2024 ) . Moreover, dense pixel motion can be freely computed from videos using off-the-shelf algorithms like RAFT ( Teed & Deng, 2020 ) , enabling scalable, weakly supervised training on large video-caption datasets, similar to prior work on predictive world models ( Gu et al., 2023 ; Black et al., 2023 ; Zhang et al., 2025 ) . Optical flow, a measure of pixel motion (PM) between consecutive frames, has been leveraged to enhance motion-focused video generation ( Liang et al., 2024 ; Koroglu et al., 2024 ) , including in the robotics domain ( Gao et al., 2025 ) . PM calculated from current and future frames is used for robot control in Ko et al. (2023) ; Bharadhwaj et al. (2024a) , further establishing the promise of this direction. In contrast, we directly generate PM from language and a single current frame (without access to future frames) using our System-2 module, offering greater data efficiency and performance (see Tables 4 , 3 and 3 ). Our predicted PM serves as an interpretable intermediate representation for downstream systems (e.g., our System-1), enabling even unsupervised control via hand-crafted mappings. Alternate motion signals in image-space are used in works like Sudhakar et al. (2024) ; Shridhar et al. (2024) ; Huang et al. (2024) ; Shi et al. (2025a) , but they rely on explicit dense annotations limiting training scalability, unlike our System-2 formulation. Generating PM from a single image has also been explored ( Walker et al., 2015 ; Gao et al., 2018 ; Aleotti et al., 2021 ) , but has been limited to the visual domain with no language conditioning. In contrast, our System-2 module generates PM conditioned on both visual and textual cues with no access to future frames. Sequences of PM generated by our System 2 are then be transformed into robot actions via System 1 , a fast and deterministic controller. Specifically, System 1 consists of motion to action mappings that are embodiment aware . We explore two instantiations of System 1: (a) learning mappings directly from limited expert demonstrations, and (b) hand-crafting mappings by leveraging the interpretable nature of pixel motion (motivated by Ko et al. (2023) ). Connecting System 1 and System 2 forms our overall language-conditioned robot control framework, LangToMo. This hierarchical formulation allows operating the expensive high-level System 2 at sparse temporal intervals while invoking the lightweight low-level System 1 at dense temporal intervals for efficient inference. This also allows independent training of each system, leading to better overall training efficiency. In summary, our contributions are as follows: • Universal Action Representation: 2D structured dense pixel motion as a learnable, interpretable, and manipulator-motion focused representation for robot control tasks. • Simple & Scalable Learning: mapping natural language actions to motion representations (pixel motion sequences) with a history-aware conditional diffusion model trained on any video-caption data, without requiring pixel-level or action trajectory annotations. • Robotics Application: conversion of learned action representations into action policies with minimal supervision, enabling operation under zero-shot and even unsupervised settings. We evaluate LangToMo on both simulated and real-world environments, highlighting its effectiveness and generality across diverse robot control tasks. 2 Related Work Learning from Videos: Robot learning has a rich history of leveraging videos to extract sub-goal information, learn strong representations, or build dynamics models for planning ( Lee & Ryoo, 2017 ; Finn & Levine, 2017 ; Sun et al., 2018 ; Kurutach et al., 2018 ; Pari et al., 2022 ; Nair et al., 2022 ; Shao et al., 2021 ; Chen et al., 2021 ; Bahl et al., 2022 ; Sharma et al., 2019 ; Du et al., 2023b ; Sivakumar et al., 2022 ; Sudhakar et al., 2024 ; Ko et al., 2023 ; Hu et al., 2025 ; Ren et al., 2025 ) . Several recent works learn representations connected to language modality from video-caption data ( Du et al., 2023b ; Sudhakar et al., 2024 ; Ko et al., 2023 ; Hu et al., 2025 ) , but depend

Read the complete text