What to Do Next? Memorizing skills from Egocentric Instructional Video
Jing Bi, Chenliang Xu · 2025
Read What to Do Next? Memorizing skills from Egocentric Instructional Video by Jing Bi, Chenliang Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02997.txt.
Introduction
Jing Bi and Chenliang Xu teach an agent to plan goal steps from first person teaching videos. Learning by watching means pulling key facts about a place from raw views. Their task asks for high level next acts in a sim world seen from the agent's eyes.
A memory of what spots afford which acts, built as a web, joins a transformer brain. That memory helps pick acts that fit the scene and flags when the agent strays off plan. Tests in a rich sim show cleaner learning and firm acts even when slips occur. Watching may thus become doing.
Catalog record
| CREATOR | Jing Bi, Chenliang Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-02997
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What to Do Next? Memorizing skills from Egocentric Instructional Video Jing Bi, Chenliang Xu Abstract Learning to perform activities through demonstration requires extracting meaningful information about the environment from observations. In this research, we investigate the challenge of planning high-level goal-oriented actions in a simulation setting from an egocentric perspective. We present a novel task, interactive action planning, and propose an approach that combines topological affordance memory with transformer architecture. The process of memorizing the environment’s structure through extracting affordances facilitates selecting appropriate actions based on the context. Moreover, the memory model allows us to detect action deviations while accomplishing specific objectives. To assess the method’s versatility, we evaluate it in a realistic interactive simulation environment. Our experimental results demonstrate that the proposed approach learns meaningful representations, resulting in improved performance and robust when action deviations occur. 1 Introduction Recent advances in the field of egocentric vision have enabled the recognition and understanding of human activities in real-world scenarios, paving the way for a range of applications, such as augmented reality assistants that guide users in performing tasks and human-robot collaboration systems. Significant progress has been made in this field with the focus on recognizing What is happening? [ 36 ] , e.g . action recognition [ 9 ] and detection [ 34 , 7 , 26 ] . However, visual intelligence demands more than mere comprehension of current events but also the capacity to achieve specific task with perceived information. This generally involves two steps. 1) Task planning: plan ahead a series of steps (high-level task actions, e.g . , navigate to kitchen ). 2) Motion planning: accomplish step with a sequence of motions (low-level actions, e.g . , turn right, turn left ) based on the objectives and the state of the environment. Recent advances in visual navigation [ 3 , 18 ] go beyond recognition and equip autonomous agents with ability to understand the sorrounding environment and make decisions with visual information which is crucial for solving how to move to accomplish given step. In this study, we focus on the visual task planning: deciding what to do next to achieve the given task with only perceived visual information. As shown in Fig. 1 , with a description of the task and the visual observation of the initial location, a successful involves first navigate to kitchen, grab plate and place them on the table. Figure 1 : The problem formulation: Given the description of the goal g_{i} set up table and initial observation, the agent should need to plan a sequence of actions to achieve the task. When executing an action a^{i}_{t} , the agent will receive a stack of [o^{s}_{t},o^{e}_{t}] as the observation, where o^{s}_{t},o^{e}_{t} represent the starting/ending point of the action. Different trajectories may share the same scene; thus, understanding the environment is crucial for action planning The classic task planning problem has been extensively studied in the field of robotics, but it assumes a closed world where the algorithm is equipped with complete domain knowledge. This means that all possible states and actions are explicitly defined in logic form using description languages like Planning Domain Description Language (PDDL) [ 1 , 13 ] . For instance, in a block world, the state of the world is defined by a set of predicates such as (:init (on block-a table) (on block-b block-a)) and (:goal (on block-b table) (on block-a block-b)) . The planner then uses a search algorithm to determine the minimum number of actions required to transform the current state into the goal state. However, this method is not suitable for visual inputs, as representing and grounding visual observations in a symbolic form is still a challenging task [ 14 , 20 ] . A similar work, Visual semantic planning [ 35 ] , also focuses on task planning and involves predicting task actions in a simulation environment using pre-defined high-level actions. Their method utilizes Q-learning with successor feature vectors to encode the expected outcomes of future actions and reason about the optimal action based on future rewards. However, their approach relies on optimal planning and modeling reward functions, while our method does not have such constraints. A similar problem, referred to as procedure planning in instructional videos, has recently been proposed [ 6 , 4 ] where the observation is clip of video and the algorithm is required to plan a sequence of actions. However, evaluating the planning model’s performance with a static dataset often produces overoptimistic results since a feedback signal is only provided if the planned actions are correct. For example, in a breakfast preparation task, the correct step may be to cut an apple but if the system makes an incorrect prediction such as pick an apple the dataset will still provide a frame of cut apples instead of a complete one. The lack of reactive feedback limits the ability to predict suitable future actions and makes it difficult to identify and recover from such errors. To address the aforementioned issues, we propose a novel task called Interactive Action Planning that emphasizes the following key elements: a) learning skills offline from recorded experience without the need for interaction; b) online evaluation of performance with an interactive environment; c) comprehending the description of the given task; d) planning a sequence of actions while being aware of the environment, as the one shown in Fig. 1 . The proposed task aims to overcome the challenges posed by the limitations of current visual task planning problem, and offers a more comprehensive and effective evaluation pipeline. Unlike typical image-language translation tasks, predicting the same actions as ground truth with sequence mapping is not optimal because the same task can be executed differently based on the situation. Solving the proposed task requires the agent to reason about the objectives of the activities, determine the possible interactions in the current situation, and decide the appropriate actions [ 5 ] . As humans, it is difficult to approach new problems as if we have never faced them before. Instead, we try to find the best plan they have heard of or previously used that is closest to the problem at hand and attempt to adapt it to the current situation. Similarly, when planning in a new environment, it is crucial for the agent to extract useful information from past experiences. In light of classic case-based planning approaches that view planning as a memory task, and recent work of utilizing episodic memory for control [ 27 ] , we tackle this problem with focusing on two aspects: memorizing essential information from demonstrations with awareness of the environments and retrieve goal-directed relevant knowledge for planning. We proposed a single holistic framework that integrates a new form of memory to associate the visual information with environment affordance. Specifically, the proposed memory serves a few purposes: a) organizing the expert’s experience into a more structured representation for subsequent action policy learning; b) localizing current visual observations in the latent space and retrieving corresponding memories when encountering a similar situation to narrow down potential action selection c) enabling replanning once the action execution deviates from reaching the goal. This framework has the advantage of decomposing the action learning from memorizing the environment,