SPOC: Spatially-Progressing Object State Change Segmentation in Video
Priyanka Mandikal, Tushar Nagarajan, Alex Stoken, Zihui Xue, Kristen Grauman · 2025
Read SPOC: Spatially-Progressing Object State Change Segmentation in Video by Priyanka Mandikal, Tushar Nagarajan, Alex Stoken, Zihui Xue, Kristen Grauman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11953.txt.
Introduction
Priyanka Mandikal and co-authors present a 2025 paper on tracking how objects change state in video. A block of cheese, for instance, may be whole at first and grated at the end. Older tools mark when the change starts and ends, but not where on the object it unfolds.
Their task asks a model to mark, pixel by pixel, the parts that can still be worked on and the parts already changed. Top models did poorly at this task, which shows how hard it is. The team built a first method and a new test set from wild online videos. Tests show the method can track where change spreads and how fast.
Catalog record
| CREATOR | Priyanka Mandikal, Tushar Nagarajan, Alex Stoken, Zihui Xue, Kristen Grauman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11953
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SPOC: Spatially-Progressing Object State Change Segmentation in Video Priyanka Mandikal, Tushar Nagarajan, Alex Stoken, Zihui Xue, Kristen Grauman Abstract Object state changes in video reveal critical cues about human and agent activity. However, existing methods are limited to temporal localization of when the object is in its initial state (e.g., cheese block) versus when it has completed a state change (e.g., grated cheese), offering no insight into where the change is unfolding. We propose to deepen the problem by introducing the spatially-progressing object state change segmentation task. The goal is to segment at the pixel-level those regions of an object that are actionable and those that are transformed. We show that state-of-the-art VLMs and video segmentation methods struggle at this task, underscoring its difficulty and novelty. As an initial baseline, we design a VLM-based pseudo-labeling approach, state-change dynamics constraints, and a novel WhereToChange benchmark built on in-the-wild Internet videos. Experiments on two datasets validate both the challenge of the new task as well as the promise of our model for localizing exactly where and how fast objects are changing in video. We further demonstrate useful implications for tracking activity progress to benefit robotic agents. Overall, our work positions spatial OSC segmentation as a new frontier task for video understanding: one that challenges current SOTA methods and invites the community to build more robust, state-change-sensitive representations. † † Project webpage: https://vision.cs.utexas.edu/projects/spoc-spatially-progressing-osc 1 Introduction Understanding object state changes (OSC) is crucial for various applications in computer vision and robotics. OSC refers to the dynamic process where objects transition from one state to another, such as a whole apple being sliced, a potato being mashed, or a piece of cloth being ironed. Accurately capturing and analyzing these transitions is essential for tasks that require real-time decision-making and action planning. For instance, in human activity videos, precise OSC understanding could enable AR/MR assistants to monitor the progress of a user’s activities and prompt timely instructions and interventions. In robotics, it would enable robots to perform complex tasks, such as cooking or assembly, by determining when and where to act next. Figure 1 : a) An illustration of the spatially-progressing video OSC segmentation problem: With time, the regions within an object undergo a progressive state-change from actionable to transformed. b) A diverse set of spatially-progressing segmentations for different state-change activities. Red is actionable; green is transformed. Current approaches for OSC typically classify video frames as initial, transitioning, or end states [ 37 , 38 , 43 ] or segment entire objects undergoing OSCs [ 47 , 40 ] . However, existing work makes an important assumption that all parts of the interacted objects change simultaneously and uniformly over time. We observe that in reality, an OSC often also spatially progresses during an action : (1) state changes occur to specific localized object regions over time, and (2) parts of objects can undergo changes non-uniformly. For example, in Fig. 1 , the cheese is grated over dough, and parts of the potato mixture get mashed over time. Though not possible with existing methods, these pixel-level spatial distinctions are essential for planning and execution of any such state-changing process—such as a robot deciding exactly where to act next, or an AI assistant capturing a smart-glasses user’s progress on their task. To address this fundamental limitation, we introduce a novel task formulation: spatially-progressing OSC segmentation ( SPOC ). Given an OSC, we formally categorize the object regions into one of two states— actionable or transformed . For example, in Fig. 1 a, the grated pieces of the cheese are transformed, while the yet-to-be-grated chunk is actionable. Given a video, the goal is to actively segment and label these regions in each frame as the object undergoes the state change transition. These changes are typically irreversible, meaning the object cannot return to its original state—an inherent characteristic of common OSCs [ 37 , 43 ] . Note that not all OSCs are spatially-progressing. We define a spatially-progressing OSC as a state change that sequentially affects different regions of the object. This includes a wide variety of human activities like grating, peeling, shredding, painting and so on, and excludes those where the whole object undergoes state change uniformly, like grilling, frying, or blending 1 1 1 We consider cooking-related OSCs following prior work [ 37 , 43 ] due to their real-world value, complexity, wide diversity, and data availability; however our approach is generalizable to other domains. . Our approach copes with the intrinsic challenges of spatial OSC, where objects can undergo dramatic transformations in color, texture, shape, topology, and size, often bearing little resemblance to their original form. These challenges thwart direct application of today’s video object segmentation models [ 50 , 35 ] , as we will see in results, and call for new training resources and objectives to be developed. Equipped with this new task formulation, we propose two innovations to tackle spatially-progressing OSC. First, we design a pipeline to acquire large-scale training data by pseudo-labeling human activity videos with vision–language models (VLMs). By combining segmentation models with language–image embeddings, we generate weak labels sufficient to bootstrap training. Second, we introduce state-change dynamics constraints—causal ordering and ambiguity resolution—that enforce temporal consistency and correct ambiguous VLM predictions. Leveraging both, we train a video model that labels mask proposals with actionable or transformed states across frames. Complementing our task and model contributions, we present WhereToChange, the first benchmark for spatially-progressing video OSC, with fine-grained intra-object state-change segmentations (Table 1 ). Derived from HowToChange [ 43 ] and VOST [ 40 ] , our dataset includes human-annotated OSC segmentations for 1162 human-activity video clips, spanning 210 state changes and 102 unique objects. Experiments on WhereToChange demonstrate that our model outperforms adapted state-of-the-art methods on this new benchmark, and its spatial OSC signals provide useful progress curves for robotic reward shaping. Yet the benchmark remains far from solved: even state-of-the-art VLMs (e.g., GPT-4o) struggle to classify state-changing parts consistently. This positions spatial OSC segmentation as a novel and challenging benchmark that pushes beyond current VLM and VOS capabilities, offering a testbed for advancing state-change–aware video object representations. 2 Related Work Object state changes Current approaches to OSC understanding fall into three main areas: classification [ 37 , 38 , 43 ] , temporal segmentation [ 47 , 40 ] , and generation [ 39 ] . Early OSC methods focused on image-level classification [ 24 , 26 , 12 , 32 , 22 , 25 , 31 , 15 ] , later expanding to video-based approaches that leverage the dynamics of OSCs [ 1 , 19 , 37 ] . The HowToChange dataset [ 43 ] further extended classification to unseen object categories. Recently, VOST [ 40 ] and VSCOS [ 46 ] introduced the task of segmenting objects undergoing state changes in videos, but only address the segmentation of the entire object throughout the video. In contrast, we provide spatial annotations of state changes both within and across object instances, for a more detailed benchmark for video object segmentation. Additionally, our OSC dynamics operate at the individual object level rather than at a broader frame level. Video object segmentation Video object segmentation (VOS) aims to separate foreground objects from background