Casper : Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models

Huihan Liu, Rutav Shah, Shuijing Liu, Jack Pittenger, Mingyo Seo, [5pt] Yuchen Cui, Yonatan Bisk, Roberto Martín-Martín, Yuke Zhu, [5pt] The University of Texas at Austin The University of California, Los Angeles · 2025

Read Casper : Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models by Huihan Liu, Rutav Shah, Shuijing Liu, Jack Pittenger, Mingyo Seo, [5pt] Yuchen Cui, Yonatan Bisk, Roberto Martín-Martín, Yuke Zhu, [5pt] The University of Texas at Austin The University of California, Los Angeles (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14727.txt.

Introduction

Huihan Liu and co-authors build Casper, a robot helper that guesses what its human operator wants. In assistive teleoperation, a person steers while the robot fills in the skilled parts. Reading intent from a few seconds of steering is the hard core. Their 2025 system leans on vision-language models and their everyday common sense. An open-world perception module names new objects and scenes. A reasoning step reads the operator's goal from the input snippet.

A skill library then carries out long, multi-step mobile manipulation tasks. Human studies and ablations back the design. Casper beat plain teleoperation and rival assistive setups on task success. Operators reported lower mental load and higher satisfaction. The system aims at homes and workshops, not just lab benches.

Catalog record

CREATORHuihan Liu, Rutav Shah, Shuijing Liu, Jack Pittenger, Mingyo Seo, [5pt] Yuchen Cui, Yonatan Bisk, Roberto Martín-Martín, Yuke Zhu, [5pt] The University of Texas at Austin The University of California, Los Angeles
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-14727

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Casper : Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models Huihan Liu, Rutav Shah, Shuijing Liu, Jack Pittenger, Mingyo Seo, [5pt] Yuchen Cui, Yonatan Bisk, Roberto Martín-Martín, Yuke Zhu, [5pt] The University of Texas at Austin The University of California, Los Angeles Abstract Assistive teleoperation, where control is shared between a human and a robot, enables efficient and intuitive human-robot collaboration in diverse and unstructured environments. A central challenge in real-world assistive teleoperation is for the robot to infer a wide range of human intentions from user control inputs and to assist users with correct actions. Existing methods are either confined to simple, predefined scenarios or restricted to task-specific data distributions at training, limiting their support for real-world assistance. We introduce Casper , an assistive teleoperation system that leverages commonsense knowledge embedded in pre-trained visual language models (VLMs) for real-time intent inference and flexible skill execution. Casper incorporates an open-world perception module for a generalized understanding of novel objects and scenes, a VLM-powered intent inference mechanism that leverages commonsense reasoning to interpret snippets of teleoperated user input, and a skill library that expands the scope of prior assistive teleoperation systems to support diverse, long-horizon mobile manipulation tasks. Extensive empirical evaluation, including human studies and system ablations, demonstrates that Casper improves task performance, reduces human cognitive load, and achieves higher user satisfaction than direct teleoperation and assistive teleoperation baselines. More information is available at https://ut-austin-rpl.github.io/casper/ Keywords: Assistive Teleoperation, Mobile Manipulation 1 Introduction Deploying robots in human-centric settings like households requires balancing robot autonomy with humans’ sense of agency [ 1 , 2 , 3 , 4 , 5 , 6 ] . Full teleoperation offers users fine-grained control but imposes a high cognitive load, whereas fully autonomous robots act independently but often misalign their actions with nuanced human needs. Assistive teleoperation — a paradigm in which both the human and the robot share control [ 7 , 8 , 9 , 10 ] — has thus emerged as an ideal middle ground. By keeping the user in control of high-level decisions while delegating low-level actions to the autonomous robot, this approach both preserves user agency and enhances overall system performance. As such, assistive teleoperation is becoming a desirable paradigm for robots to serve as reliable partners in human-centric environments, such as assisting individuals with motor impairments [ 11 , 12 ] . Figure 1: Casper infers user intents and offers help when confident. Given user teleoperation input, Casper uses VLMs to predict human intent using commonsense reasoning. Upon user confirmation, Casper performs autonomous execution to fulfill the intent using a skill library. Casper ’s background reasoning runs in parallel with foreground human control to minimize disruption. While promising, assistive teleoperation in everyday environments remains challenging. A long-standing challenge in assistive teleoperation is to infer human intents from user control inputs and assist users with correct actions [ 8 ] . This challenge is amplified in real-world settings, where robots must go beyond closed-set intent prediction [ 13 , 14 ] to handle diverse, open-ended user goals across different contexts and scenes. As a result, a key capability the robot should possess is to interpret user control inputs within the visual context and infer intent through commonsense reasoning. For example, consider a user teleoperating a robot to move a jar of pasta toward both a laptop and a cooking pot. Even if the pasta jar is closer to the laptop, commonsense suggests that the user intends to pour pasta into the pot, not onto the laptop. As another example, some users push an automatic door to open it, while others want to press an accessibility button. These examples illustrate the nuanced and context-dependent nature of human intent, highlighting the level of commonsense reasoning required for robots to provide effective and satisfactory assistance. Existing assistive teleoperation systems often fall short in inferring diverse intents. Prior methods often limit the problem space to a closed set of objects [ 14 , 9 ] , or to a predefined task like picking up objects, implicitly assuming the intent type is known a priori [ 14 , 13 ] . These intent inference methods, either based on rule-driven strategies [ 15 , 13 ] or learned from demonstrations [ 16 , 17 , 14 , 10 ] , are typically limited to one single skill type or bound by the task distributions at training, struggling to generalize in new scenarios. Critically, these systems usually lack commonsense reasoning, which is essential for interpreting contextual cues and generalizing intent inference to novel scenes and behaviors in real-world environments. To address the above limitations, we introduce Casper , an assistive teleoperation system that infers diverse intents from human user control and offers assistance with long-horizon mobile manipulation tasks (Fig. 1 ). Casper builds on three core components. First, it features an open-world perception module that uses pre-trained visual language models (VLMs) to provide a generalized understanding of open-world objects and scenes without task-specific training. Second, Casper leverages VLM-powered commonsense reasoning to infer a diverse range of user intents, significantly expanding the possible intent choices compared with prior systems. Third, to realize task execution, Casper uses a flexible library of parameterized skills encompassing a range of navigation and contact-rich manipulation behaviors [ 18 ] . With this comprehensive and composable skill library, Casper can execute long-horizon tasks that go beyond the capabilities of traditional assistive teleoperation systems. Furthermore, deploying the system for long-horizon tasks introduces a user-centric consideration: offering undesirable assistance based on prematureintent inference can frustrate or disrupt the user. To avoid this, the system should determine intents only after gathering enough information from user inputs and visual contexts. Casper addresses this by shadowing the user: it observes foreground human actions and infers user intents in the background. A confidence module based on self-consistency [ 19 ] ensures that assistance is triggered only when prediction confidence is high, reducing errors and user disruption. By running VLM-based inference in parallel with user control, Casper unobtrusively predicts intent and prepares actions. To evaluate the effectiveness of Casper in assisting human users, we conduct extensive user studies on a mobile manipulator (TIAGo [ 20 ] ), involving 10 pilot study participants and 13 study participants, totaling over 80 hours of interaction across 3 long-horizon tasks. Additionally, we conduct offline experiments to test the intent inference module and perform detailed performance analyses and ablation studies. Compared with prior assistive teleoperation baselines without commonsense reasoning ability and a full teleoperation baseline, Casper achieves a higher success rate, better user satisfaction, and lower cognitive load of users across all tasks. 2 Related Work Assistive Teleoperation. Assistive teleoperation offers a promising balance between human control and robotic assistance, enhancing user agency and task efficiency [ 11 , 12 , 15 , 21 , 22 , 23 ] . Assistive teleoperation enables users to share control with the robot, injecting their intent to guide the system toward their goals [ 8 , 7 , 10 , 24 , 25 , 26 , 27 ] . Accurately predicting user intent is thus a key challenge [ 8 , 28 , 29 ] . Prior approaches typically select the most probable intent from a fixed set of

Read the complete text