TransMASK: Masked State Representation through Learned Transformation Journal: Autonomous Robots

Sagar Parekh, Preston Culbertson, and Dylan P. Losey · 2026

Read TransMASK: Masked State Representation through Learned Transformation Journal: Autonomous Robots by Sagar Parekh, Preston Culbertson, and Dylan P. Losey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05670.txt.

Introduction

Parekh and team propose TransMASK, a self-taught method learning a mask that transforms observed image features to retain only task-relevant ones. Adding a learned mask during training creates rival pressure among image features, forcing the policy to attend only to always task-relevant features. The masks are easy to read and reject fake features such as decoy spots and background color. Compared with other copy learning form methods, TransMASK rules prove more robust to shifts in off-topic features, gaining at least 30% over base models in new settings.

Catalog record

CREATORSagar Parekh, Preston Culbertson, and Dylan P. Losey
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74171
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05670

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TransMASK: Masked State Representation through Learned Transformation Journal: Autonomous Robots Sagar Parekh, Preston Culbertson, and Dylan P. Losey Abstract Humans train robots to complete tasks in one environment, and expect robots to perform those same tasks in new environments. As humans, we know which aspects of the environment (i.e., the state) are relevant to the task. But there are also things that do not matter; e.g., the color of the table or the presence of clutter in the background. Ideally, the robot’s policy learns to ignore these irrelevant state components. Achieving this invariance improves generalization: the robot knows not to factor irrelevant variables into its control decisions, making the policy more robust to environment changes. In this paper we therefore propose a self-supervised method to learn a mask which, when multiplied by the observed state, transforms that state into a latent representation that is biased towards relevant elements. Our method — which we call TransMASK — can be combined with a variety of imitation learning frameworks (such as diffusion policies) without any additional labels or alterations to the loss function. To achieve this, we recognize that the learned policy updates to better match the human’s true policy. This true policy only depends on the relevant parts of the state; hence, as the gradients pass back through the learned policy and our proposed mask, they increase the value for elements that cause the robot to better imitate the human. We can therefore train TransMASK at the same time as we learn the policy. By normalizing the magnitude of each row in TransMASK, we force the mask to align with the Jacobian of the expert policy: columns that correspond to relevant states have large magnitudes, while columns for irrelevant states approach zero magnitude. We compare our approach to other methods that extract relevant states for downstream imitation learning. Across experiments with visual and non-visual states, we see that TransMASK results in policies that are more robust to distribution shifts for irrelevant features. See our project website: https://collab.me.vt.edu/TransMASK/ ∎ 1 Introduction Imitation learning enables robots to learn tasks from offline demonstrations provided by a human expert. Consider teaching a robot to pick up a block and place it at the center of the table (see Figure 1 ). The human teacher guides the robot through successive states such as reaching the block, grasping it, and placing it at the desired location. In the demonstration, the robot receives observations of the scene recorded through cameras and the actions applied by the expert. When demonstrating the task, the human expert focuses only features crucial for completing the task — the object, the goal, and the robot positions. However, the robot’s observations record information about the entire scene, capturing lighting conditions, table texture, background objects, and other task-irrelevant features. Figure 1: Robot learning to pick up a green block from a cluttered environment and place it at the center of the table. (A) The expert demonstrations are collected over a wooden table. (B) This expert’s decisions depend only on features intrinsic to the task (e.g., green block, robot position, and target position). The robot’s observations in these demonstrations, however, record information about the entire scene, capturing the table texture, background color, and task-irrelevant objects. (C) We assume a disentangled state s is extracted from these high-dimensional visual observations. In this state, some elements correspond to the task structure — positions of the green block, target, and robot, while others correspond to scene-specific factors. In the figure, relevant elements are shown in green and scene-specific elements are shown in orange and purple. (D) Standard imitation learning policy attends to the entire state. When this policy is deployed for the identical task in a different scene (e.g., over a marble table), it may fail due to spurious dependencies learned during training. (E) To learn a robust policy, we must encode the state to a representation z which masks-out task-irrelevant features. A policy that is conditioned on z is more robust to distribution shift caused by changes in the scene that do not alter the task structure, for instance, when the task is still to pick and place a green block but over a marble table instead of a wooden table. A policy trained using these observations inherits this issue: it can attend to all parts of the state, including the details a human demonstrator would ignore. This causes the policy to become increasingly brittle. Consider Figure 1 : when deploying a policy trained with demonstrations over a wood table for an identical task on a marble table, the visual observations become out-of-distribution, causing the policy to fail. Our key idea is that a robust policy is one that attends to the same features a human expert would. Thus, we aim to design policies with structures to mask out task-irrelevant information, enabling the policy to make decisions using features inherent to the task (e.g., object position, end-effector pose) instead of the scene(s) in its training data. Existing works have developed a variety of methods for extracting this relevant state. A straightforward approach augments the training data with random transformations of visual observations (e.g., random cropping, adding noise to the image) tobin2017domain ; kaidanov2024role . Nevertheless, introducing variability in the training data can degrade in-domain performance and does not guarantee robustness to large distribution shifts. Another line of work leverages vision-language model (VLM) backbones sharma2023lossless ; yang2024robot that have robust state representations due to large-scale pretraining on diverse datasets. Despite these advantages, VLMs typically require task-specific finetuning, which can lead to catastrophic forgetting of pretrained features aversely effecting the robustness. Alternatively, a different class of literature focuses on learning minimal state representations that can extract only the most relevant information from the observations. This is achieved by introducing a trade-off between i) disconnecting the latent state from the actual state (compression) and ii) ensuring the latent state can map to correct actions (performance). Examples of this approach include using variational autoencoders (VAEs) for encoding the images to states rahmatizadeh2018vision ; shang2021self , training an encoder with the Information-Bottleneck (IB) principle tishby2000information , or adding structure to the representation space using contrastive learning lee2025class ; giammarino2025visually . However, these approaches — as we will show empirically and theoretically — rely on an ill-posed optimization problem that often collapses into inaccurate state representations that lack either compression or performance. We therefore propose a new method for extracting the correct state from observations without modifying the optimization objective. We begin with the intuition that actions demonstrated by the expert are conditioned only on parts of the state that are relevant to the task. When optimizing a standard imitation learning objective, predicted actions will correlate strongly with these elements and weakly correlate with extraneous elements. Consequently, the magnitude of gradients associated with action-relevant elements will be larger. Building on this intuition, we hypothesize that: Magnitude of policy Jacobian can be a proxy for causal relevance. Put simply, we can exploit the gradients to identify and preserve the components of state that matter for control. This prevents the robot policy from learning spurious correlations between states and actions. In order to ensure that the state space is structured, we assume that the robot state is

Read the complete text