Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding
Jinghan Zhao, Yifei Huang, Feng Lu · 2025
Read Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding by Jinghan Zhao, Yifei Huang, Feng Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20073.txt.
Introduction
Zhao, Huang, and Lu teach video models to understand procedures through visible states. Their 2025 paper addresses how machines learn from instructional videos. Agents must grasp tasks as sequences of concrete steps. Prior methods aligned video with task and step descriptions. Those descriptions stay too abstract to match pixels well.
The team adds states: text snapshots of how objects look. States anchor steps to things a model can see. Their Task-Step-State framework links tasks, steps, and states hierarchically. Steps drive transitions between one state and the next. A progressive pretraining strategy unfolds that hierarchy gradually. Training cycles from task to step to state and back. Tests ran on the COIN and CrossTask video datasets. Recognizing tasks, recognizing steps, and forecasting the next step all improved. Ablations credit state supervision as the key driver. Progressive training beat plain joint training clearly. Grounding procedures in visible states is their central claim.
Catalog record
| CREATOR | Jinghan Zhao, Yifei Huang, Feng Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20073
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding Jinghan Zhao, Yifei Huang, Feng Lu Abstract Learning procedural-aware video representations is a key step towards building agents that can reason about and execute complex tasks. Existing methods typically address this problem by aligning visual content with textual descriptions at the task and step levels to inject procedural semantics into video representations. However, due to their high level of abstraction, ‘task’ and ‘step’ descriptions fail to form a robust alignment with the concrete, observable details in visual data. To address this, we introduce ‘states’, i.e. , textual snapshots of object configurations, as a visually-grounded semantic layer that anchors abstract procedures to what a model can actually see. We formalize this insight in a novel Task-Step-State (TSS) framework, where tasks are achieved via steps that drive transitions between observable states. To enforce this structure, we propose a progressive pre-training strategy that unfolds the TSS hierarchy, forcing the model to ground representations in states while associating them with steps and high-level tasks. Extensive experiments on the COIN and CrossTask datasets show that our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction. Ablation studies show that introducing state supervision is a key driver of performance gains across all tasks. Additionally, our progressive pretraining strategy proves more effective than standard joint training, as it better enforces the intended hierarchical structure. Code — https://github.com/zhao-jinghan/TSS-unfolding Introduction Figure 1: (a) Our method extends the conventional task-step hierarchy by adding a state layer, forming the Task-Step-State (TSS) framework, where state can anchor abstract procedures to visual evidence. (b) To unfold the structure in TSS, we introduce a progressive pre-training strategy by cycling through the order: task→step→state→step→task. Understanding and performing goal-oriented procedural activities is a core capability for intelligent agents. Humans effortlessly acquire such procedural knowledge from instructional videos, learning not only the overall goals but also the sequence of steps required to achieve them. Enabling machines to learn similar procedural-aware video representations is therefore central to progress in areas such as embodied AI, intelligent assistants, and emerging paradigms like cobodied intelligence (human-AI co-embodied intelligence) ( 18 ) . To this end, existing methods typically seek to inject procedural knowledge into video representations by aligning visual content with textual descriptions from external knowledge bases ( 17 ; 43 ; 19 ) . This alignment is generally performed at two levels of abstraction: the overall task ( e.g. , “Make orange juice.”) and the constituent steps ( e.g. , “cut oranges”). By matching video features or ASR transcripts to these descriptions, models learn to recognize the high-level structure of an activity. However, we argue that this two-level approach suffers from a critical limitation. Due to their high level of abstraction, “task” and “step” descriptions often fail to form a robust alignment with the concrete, observable details in visual data. A significant semantic gap ( 34 ) exists between an abstract command like “cut oranges” and the raw pixels depicting the action. Thus, when training with visual-text alignment, this gap makes it difficult for models to ground abstract procedures in what they can actually see. To address this, we introduce “states”, i.e. , textual snapshots of object configurations, as a visually-grounded semantic layer that anchors abstract procedures to visual evidence. A state describes the observable configuration of key objects, such as “the oranges are no longer whole, with their flesh exposed.” From a logical standpoint, states form the skeleton of any procedural task. A high-level “Task” can be viewed as a macro-level transition from an initial state to a final state, while a series of “Steps” are the actions that drive the transitions between these intermediate states. Building on this insight, we formalize this structure in a novel Task-Step-State (TSS) framework. In this hierarchy, tasks are achieved via steps, and steps are semantically anchored by the observable state transitions they cause. To enforce this structure and ensure the model learns the intended hierarchy, we propose a novel progressive pre-training strategy that unfolds the TSS hierarchy. Instead of learning all semantic levels jointly, our strategy forces the model to ground its representations in the most concrete layer—the states. By this means, it better learns to associate states with steps, and finally, steps with high-level tasks. This progressive approach ensures that the learning of abstract concepts is firmly built upon a foundation of visually-grounded understanding. We validate our approach through extensive experiments on the COIN ( 32 ) and CrossTask ( 45 ) datasets. Our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction. Furthermore, detailed ablation studies confirm our core hypotheses: introducing state supervision is the key driver of these performance gains, and our progressive pre-training strategy is significantly more effective than standard joint training at enforcing the hierarchical structure. Our contributions are: • We propose a novel Task-Step-State (TSS) framework that introduces “states” as a visually-grounded semantic layer, effectively bridging the semantic gap between abstract procedural texts and concrete visual content. • We introduce a progressive pre-training strategy that unfolds the TSS hierarchy, ensuring that representations are grounded in concrete states before learning abstract steps and tasks, better enforcing the procedural logic. • Experiments demonstrate that our method achieves consistent performance gains over existing models and validate that both the TSS framework and the progressive training strategy are key to this success. Related works Procedural Learning from Instructional Video Instructional videos serve as a rich source for learning how agents perceive and act in the world, fueling applications such as procedure planning ( 35 ; 40 ; 14 ; 38 ) , step recognition ( 15 ; 23 ; 45 ) , localization ( 9 ) , and task recognition ( 11 ) . The introduction of large-scale, richly annotated datasets ( 44 ; 32 ; 2 ) has greatly advanced this field. However, acquiring dense temporal annotations for task-step structures remains a key challenge, limiting the scalability of supervised approaches. To address this bottleneck, recent efforts have embraced weakly-supervised paradigms that leverage external knowledge sources, such as wikiHow ( 16 ) , to generate supervisory signals for large-scale, unannotated video collections ( 17 ; 43 ; 19 ; 25 ; 42 ) . These approaches typically aim to learn procedural-aware video representations by aligning visual or transcribed-text features with textual descriptions of steps. For instance, Paprika ( 43 ) utilizes pre-trained visual and textual encoders to perform cross-modal matching, and builds a graph that captures the temporal progression of both visual content and procedural steps derived from wikiHow. Similarly, ( 19 ) integrates video frames, ASR-generated narrations, and step descriptions, proposing dual alignment paths (“step–frame” and “step–narration–frame”) to bridge the gap between vision and language modalities. While effective at capturing the high-level procedure, these approaches operate purely at the abstract task and step levels. This introduces a critical limitation, as noted by works like ( 34 ) : a semantic gap exists between abstract, action-centric text and concrete visual data. Our work directly confronts this gap. We argue that to truly ground procedural learning, the knowledge hierarchy itself must be augmented. We achieve this by introducing “states” as a new, visually-grounded semantic layer, bridging the