Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress

Priyanka Mandikal, Jiaheng Hu, Shivin Dass, Sagnik Majumder, Roberto Martín-Martín*, Kristen Grauman* · 2025

Read Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress by Priyanka Mandikal, Jiaheng Hu, Shivin Dass, Sagnik Majumder, Roberto Martín-Martín*, Kristen Grauman* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24129.txt.

Introduction

Mashing, spreading, and slicing change what a thing is, not just where it sits. Priyanka Mandikal and 5 co-authors present SPARTA, a framework for this family of robot tasks. Most robot work targets moves such as picking, placing, or opening, which leave the object itself intact.

SPARTA's key idea is that these state-change tasks share a pattern: regions of the object shift from actionable to transformed, step by step. The system maps those regions to build both its observations and a dense reward for progress. Two policy styles use those signals, one learned by reinforcement and one greedy for fast setup. On a real robot across ten objects and three tasks, SPARTA trained faster and scored higher than standard baselines.

Catalog record

CREATORPriyanka Mandikal, Jiaheng Hu, Shivin Dass, Sagnik Majumder, Roberto Martín-Martín*, Kristen Grauman*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24129

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress Priyanka Mandikal, Jiaheng Hu, Shivin Dass, Sagnik Majumder, Roberto Martín-Martín*, Kristen Grauman* Abstract Most robot manipulation focuses on changing the kinematic state of objects: picking, placing, opening, or rotating them. However, a wide range of real-world manipulation tasks involve a different class of object state change —such as mashing, spreading, or slicing—where the object’s physical and visual state evolve progressively without necessarily changing its position. We present SPARTA , the first unified framework for the family of object state change manipulation tasks. Our key insight is that these tasks share a common structural pattern: they involve spatially-progressing, object-centric changes that can be represented as regions transitioning from an actionable to a transformed state. Building on this insight, SPARTA integrates spatially progressing object change segmentation maps , a visual skill to perceive actionable vs. transformed regions for specific object state change tasks, to generate a) structured policy observations that strip away appearance variability, and b) dense rewards that capture incremental progress over time. These are leveraged in two SPARTA policy variants: reinforcement learning for fine-grained control without demonstrations or simulation; and greedy control for fast, lightweight deployment. We validate SPARTA on a real robot for three challenging tasks across 10 diverse real-world objects, achieving significant improvements in training time and accuracy over sparse rewards and visual goal-conditioned baselines. Our results highlight progress-aware visual representations as a versatile foundation for the broader family of object state manipulation tasks. More information at https://vision.cs.utexas.edu/projects/sparta-robot I Introduction Fig. 1 : Top: While most robotic manipulation focuses on rigid-body motion, many real-world tasks involve object state changes such as mashing, spreading, or slicing, where objects are progressively transformed. Bottom: SPARTA leverages spatially-progressing affordance maps of actionable vs. transformed regions, successfully demonstrating how to guide real robot manipulation for this family of tasks. The dominant paradigm in robotic manipulation centers on tasks involving rigid body motion—such as picking-and-placing [ 1 ] , opening and closing [ 2 , 3 ] , pushing [ 4 ] , or rotating [ 5 ] objects. While these tasks are foundational, they largely entail changing the kinematic state of objects where progress on the task is readily visible and easily monitored via changes in object pose. However, many real-world scenarios involve a fundamentally different class of manipulations: object state changes (OSC) 1 1 1 Here we adopt the term “object state change” (OSC) from the vision literature [ 6 , 7 , 8 ] : an OSC is a transformation of an object that entails a visually distinct post-condition (e.g., chopped apple) following an action imposed on it (e.g., chopping), often with irreversible changes to the object’s morphology, texture, and appearance. Not to be confused with Operational Space Control [ 9 ] . [ 6 , 7 , 10 ] —where an object’s physical state and visual appearance is progressively transformed, without necessarily altering its pose (see Fig. 1 , top). Everyday examples abound: mashing a banana into purée, spreading jam on bread, or slicing a cucumber. These tasks demand continuous interaction that alters the object’s shape, texture, and color, making them both mechanically challenging and visually complex. Such state changes are ubiquitous in everyday activities—from cooking (e.g., grating, peeling, shredding) to household chores (e.g., painting, wiping, ironing)—yet remain largely underexplored in robotics. What makes OSC manipulation challenging for robotics? Unlike motion-centric tasks, OSC requires continuous reasoning about where transformations have already occurred within a possibly non-rigid object, where they are still needed, and how to act accordingly. Two key obstacles arise. First, at the representation level , raw RGB observations entangle appearance with object state, obscuring the signals of progress and hindering generalization across objects. Second, at the learning level , obtaining a good reward function is challenging: sparse success rewards provide little guidance for exploration [ 11 ] , while goal-conditioned reward functions (e.g. LIV [ 12 ] ) often rely on global scene-level embeddings that miss the fine-grained, incremental progress essential for OSC. Together, these limitations make current approaches sample-inefficient and ill-suited to tasks where state changes unfold dynamically within the object. To tackle these challenges, we propose SPARTA (Spatial Progress-Aware Robotic object TransformAtion)—a robotic system that introduces structured, progress-aware visual affordances tailored to OSC manipulation (see Fig. 1 , bottom). SPARTA builds on recent computer vision advances in detecting spatially-progressing object state changes ( SPOC [ 8 ] ), integrating them into a fully autonomous robotic system. SPOC segments a transforming object into two regions: actionable and transformed . For instance, in mashing a potato, unmashed chunks are actionable, while mashed portions are transformed. SPARTA leverages SPOC affordance 2 2 2 Here “affordance” means regions requiring robot interaction, distinct from conventional grasp points. maps in two crucial ways: (1) as structured visual observations that strip away appearance detail while preserving progress cues, enabling generalization across objects; and (2) as dense, spatially grounded reward signals that quantify incremental progress at each step. By explicitly representing “what has changed” and “what remains,” SPARTA equips robots to reason about state progression rather than mere object kinematics. Our formulation enables two policy variants within the same framework. In SPARTA-L , we use SPOC-derived rewards to train real-world RL agents from scratch—without demonstrations or simulation—achieving highly sample-efficient learning. In parallel, SPARTA-G offers a non-parametric alternative, greedily steering toward nearby actionable regions in the SPOC map. Hence, this unified framework supports both (1) reinforcement learning, for robust, fine-grained control in settings where noise and uncertainty demand adaptive strategies; and (2) greedy control, for fast, lightweight deployment in simpler settings without training. Together, SPARTA ’s two policy variants demonstrate the versatility of its progress-aware affordances: a single representation can power both quick heuristic controllers and data-driven RL agents, depending on task complexity. In our experiments, we show that with just 1.5–3 hours of online RL training directly in the real world and no human demonstrations , SPARTA learns policies that reliably induce object state change. We evaluate across three representative OSC tasks—spreading, mashing, and slicing—on 10 diverse real-world objects, demonstrating both robustness and generality. By contrast, baseline methods fail to learn meaningful behavior, highlighting that dense, interpretable affordances for object state change are key to enabling sample-efficient, generalizable real-world robot learning—charting a path beyond rigid-body manipulation. II Related Work Non-rigid object manipulation. Recent advances tackle individual tasks requiring more complex manipulation than traditional pick-and-place-style tasks, such as cutting [ 13 , 14 , 15 , 16 ] , peeling [ 17 , 18 , 19 ] , and stir-frying [ 20 ] . However, these efforts tackle each task in isolation , often focus on the task’s mechanical aspects, lack general-purpose vision feedback, or rely heavily on simulation. In contrast, our work targets a broad class of spatially transformative tasks that require reasoning over visual state changes rather than contact dynamics alone, exploiting a unified visual representation that is shared across objects and state-change tasks. Visual representations for robot learning. To accelerate downstream policy learning, recent works pretrain visual representations on large-scale data [ 21 , 22 , 23 , 12 ] . More relevant to our novel visual rewards, VIP [ 23 ] learns an implicit value function over egocentric videos,

Read the complete text