Solving New Tasks by Adapting Internet Video Knowledge

Calvin Luo, Zilai Zeng, Yilun Du, Chen Sun · 2025

Read Solving New Tasks by Adapting Internet Video Knowledge by Calvin Luo, Zilai Zeng, Yilun Du, Chen Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15369.txt.

Introduction

Calvin Luo and co-authors teach robots new tricks by warming up web video models in a 2025 paper. Video models trained on internet footage speak fluent text and know how actions look, but they miss the quirks of one robot's room. Models trained only on robot demos know the room but starve for data and fail on new tasks.

The team tries several ways to blend the two diets and grades each on data cost and robustness. Their new strategy, Inverse Probabilistic Adaptation, led the pack across settings. Adapted models carried out fresh text-specified behaviors they had never seen in the room. The method stayed strong even when the in-room examples were clumsy, suboptimal demonstrations. The authors frame small local datasets as seasoning: a little, well folded in, changes what a giant model can serve.

Catalog record

CREATORCalvin Luo, Zilai Zeng, Yilun Du, Chen Sun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15369

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Solving New Tasks by Adapting Internet Video Knowledge Calvin Luo, Zilai Zeng, Yilun Du, Chen Sun Abstract Video generative models demonstrate great promise in robotics by serving as visual planners or as policy supervisors. When pretrained on internet-scale data, such video models intimately understand alignment with natural language, and can thus facilitate generalization to novel downstream behavior through text-conditioning. However, they may not be sensitive to the specificities of the particular environment the agent inhabits. On the other hand, training video models on in-domain examples of robotic behavior naturally encodes environment-specific intricacies, but the scale of available demonstrations may not be sufficient to support generalization to unseen tasks via natural language specification. In this work, we investigate different adaptation techniques that integrate in-domain information with large-scale pretrained video models, and explore the extent to which they enable novel text-conditioned generalization for robotic tasks, while also considering their independent data and resource considerations. We successfully demonstrate across robotic environments that adapting powerful video models with small scales of example data can successfully facilitate generalization to novel behaviors. In particular, we present a novel adaptation strategy, termed Inverse Probabilistic Adaptation , that not only consistently achieves strong generalization performance across robotic tasks and settings, but also exhibits robustness to the quality of adaptation data, successfully solving novel tasks even when only suboptimal in-domain demonstrations are available. 1 Introduction Recent advancements in video generative models have made promising their application to interactive problems and decision-making. Such video models trained explicitly on in-domain demonstrations have demonstrated accurate encoding of environment-specific visual details and dynamics, and have been popular choices to utilize for robotic learning ( Du et al., 2024b ; Huang et al., 2023 ; Yang et al., 2023b ; Ko et al., 2024 ; Liang et al., 2024 ) . When optimized on expert video demonstrations, their encoded understanding of expert behavior can be directly used to supervise the learning of high-performing policies ( Huang et al., 2023 ; Escontrela et al., 2024 ) , and applied as performant visual planners ( Du et al., 2024a ) in robotic settings. However, for arbitrary robotic environments, there is usually a severe difference in scale of tractably available expert demonstration data, especially with associated text labelling, in comparison with general internet-scale datasets of videos paired with natural language. As a result, such in-domain video generative models usually suffer from weaker generalization capability, across novel text specifications and motions of interest. Instead of training directly on in-domain demonstrations, stronger generalization performance can be obtained by using text-to-video models pretrained on internet-scale data. Having summarized powerful priors over visual styles, natural motion, and alignment with natural language from large-scale data, such models can be leveraged to supervise policies that behave flexibly conditioned on text across multiple environment visual styles without modification ( Luo et al., 2024 ) . However, interaction is often performed in a fixed environment with specific visual characteristics and potentially unique interaction dynamics, which a general environment-agnostic video model may not inherently understand or respect. Thus, directly applying a large-scale pretrained video generative model without modification comes with a potential drawback; they may not understand the intricacies of particular environments of interest to successfully facilitate high-performance behavior within them. These considerations naturally motivate the investigation of ways to mutually cover the independent deficiencies of each approach. In this work, we perform a thorough study on novel task generalization via adapting internet video knowledge; we seek to illuminate how in-domain information can be best integrated into large-scale pretrained text-to-video models, such that powerful zero-shot text-conditioned generalization capabilities are enabled while considering environment-specific knowledge pertaining to visual styles and interaction dynamics. We compare the downstream robotic performance of multiple adaptation techniques and contrast their respective requirements on in-domain data samples, which range from utilizing only a few still-frames of the agent to text-labelled video demonstrations, and training resources, which span from direct finetuning of the large-scale video model to utilizing it only for inference without any updates. Broadly, we provide this study of adaptation for facilitating action prediction ( Adapt2Act ) as valuable insight to the practitioner interested in balancing performance with resource availability. We perform standardized evaluations across both robotic manipulation tasks ( Yu et al., 2020 ) and continuous control ( Tassa et al., 2018 ) , and demonstrate that adapted video generative models are able to successfully act as accurate video planners for novel text-conditioned specifications across a variety of robotic tasks, and can also supervise the learning of novel text-conditioned policies. Furthermore, in this work we propose a novel adaptation technique termed Inverse Probabilistic Adaptation , which we highlight achieves consistently strong generalization performance across robotic environments and downstream evaluation approaches. We also discover that even when only suboptimal in-domain demonstrations are provided, it can still effectively leverage web-scale priors and text conditioning to generate coherent video plans and successfully solve novel tasks. Visualizations and code are provided at diffusion-supervision.github.io/adapt2act/ . Figure 1: Adaptation Techniques. We explore how in-domain information can be integrated into large-scale text-to-video models through three different adaptation techniques: Subject Customization, Probabilistic Adaptation, and Direct Finetuning. Subject Customization only modifies the image and text encoder, rather than the motion module, and is lightweight in terms of data requirements: it only utilizes pairs of static images and text annotated with a special identifier. Probabilistic Adaptation learns a small in-domain model from paired video data, which is then used through score composition with a large-scale video model that is kept frozen. The small in-domain model can be flexibly parameterized to consider available training resources. Direct Finetuning seeks to update the motion module of the large-scale pretrained video model with in-domain paired video data. 2 Related Work Adaptation Techniques for Diffusion Models. Although many large-scale pretrained text-to-video models ( Ho et al., 2022 ; Guo et al., 2023 ; Ramesh et al., 2022 ; Xing et al., 2023 ; Villegas et al., 2022 ; Singer et al., 2022 ; Khachatryan et al., 2023 ) have demonstrated strong capabilities of synthesizing high-quality videos following the given prompts, it is often desirable to perform adaptation for specialized tasks, such as customizing video generation to specific subjects or styles. Figure 2: Downstream Task Evaluation. We identify downstream robotic task performance as a way to achieve standardized, quantitative comparisons across adaptation techniques. We evaluate how adapted video models can enable text-conditioned generalization via two approaches: visual planning and policy supervision . For visual planning, the adapted video model synthesizes a text-conditioned video plan into the future, which is then converted into actions to follow through a separately trained inverse dynamics model. In policy supervision, the adapted video model is used in a discriminative manner to evaluate frames achieved by the policy; these are converted into text-conditioned rewards, which the policy is optimized to maximize. DreamBooth ( Ruiz et al., 2023 ) finetunes text-to-image diffusion models to connect a unique identifier to a subject of interest, using a few images of that specific subject. The subject will be implanted into the output space of the diffusion model after finetuning, enabling novel view synthesis with the subject via prompting with its corresponding identifier. In DreamVideo ( Wei et al., 2024 ) , this idea is extended to facilitate

Read the complete text