GraphGarment: Learning Garment Dynamics for Bimanual Cloth Manipulation Tasks

Wei Chen, Kelin Li, Dongmyoung Lee, Xiaoshuai Chen, Rui Zong and Petar Kormushev · 2025

Read GraphGarment: Learning Garment Dynamics for Bimanual Cloth Manipulation Tasks by Wei Chen, Kelin Li, Dongmyoung Lee, Xiaoshuai Chen, Rui Zong and Petar Kormushev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05817.txt.

Introduction

Wei Chen and co-authors teach robots to handle laundry with two hands. Cloth defeats ordinary robot planning because it folds and stretches in hard-to-predict ways. Their system, GraphGarment, represents a garment as a graph of points and learns, in simulation, how the cloth moves when the robot acts. A small correction model then mops up the gap between simulation and the real world.

Tested on six garment types, the learned model predicted cloth positions about half a centimeter better than the best rival. In hanging trials it beat baselines by wide margins, and on a real robot its error grew only slightly over simulation. The authors present it as a step toward household robots that can do fabric chores.

Catalog record

CREATORWei Chen, Kelin Li, Dongmyoung Lee, Xiaoshuai Chen, Rui Zong and Petar Kormushev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05817

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GraphGarment: Learning Garment Dynamics for Bimanual Cloth Manipulation Tasks Wei Chen, Kelin Li, Dongmyoung Lee, Xiaoshuai Chen, Rui Zong and Petar Kormushev Abstract Physical manipulation of garments is often crucial when performing fabric-related tasks, such as hanging garments. However, due to the deformable nature of fabrics, these operations remain a significant challenge for robots in household, healthcare, and industrial environments. In this paper, we propose GraphGarment, a novel approach that models garment dynamics based on robot control inputs and applies the learned dynamics model to facilitate garment manipulation tasks such as hanging. Specifically, we use graphs to represent the interactions between the robot end-effector and the garment. GraphGarment uses a graph neural network (GNN) to learn a dynamics model that can predict the next garment state given the current state and input action in simulation. To address the substantial sim-to-real gap, we propose a residual model that compensates for garment state prediction errors, thereby improving real-world performance. The garment dynamics model is then applied to a model-based action sampling strategy, where it is utilized to manipulate the garment to a reference pre-hanging configuration for garment-hanging tasks. We conducted four experiments using six types of garments to validate our approach in both simulation and real-world settings. In simulation experiments, GraphGarment achieves better garment state prediction performance, with a prediction error 0.46 cm lower than the best baseline. Our approach also demonstrates improved performance in the garment-hanging simulation experiment—with enhancements of 12%, 24%, and 10%, respectively. Moreover, real-world robot experiments confirm the robustness of sim-to-real transfer, with an error increase of 0.17 cm compared to simulation results. Supplementary material is available at: https://sites.google.com/view/graphgarment . I Introduction Manipulating garments is a key part of human daily routines. There is an increasing demand for robots to assist with tasks such as folding laundry, dressing, and hanging garments [ 1 ] . These tasks require the ability to perceive, manipulate, and plan to handle deformable objects, such as garments. While significant progress has been made in manipulating rigid objects [ 2 , 3 ] , research in deformable object manipulation has largely focused on items with relatively simple topologies, such as ropes and square cloth. However, due to their complex dynamics, techniques for handling actual garments still remain in the early stages of development [ 4 ] . Research involving cloth manipulation has been generally categorized into robotic system hardware design [ 5 , 6 ] and control algorithm design [ 7 , 8 , 9 , 10 ] . Regarding the control aspect, perception and state estimation are critical factors in manipulating cloth-like objects. Due to their deformable nature, garments can adopt a wide range of configurations, making their manipulation more complex. Unlike regular square pieces of fabric, garments typically exhibit an enclosed structure, which adds further challenges when reasoning about their dynamics and action planning for manipulation. To address these challenges, a bimanual robot setup is often employed to constrain the garment deformation, thereby simplifying the perception during manipulation. However, while this bimanual setup helps manage deformation, it also introduces greater complexity in control and data acquisition. In this context, the representation of the garment state becomes crucial for understanding and controlling its dynamics. Figure 1: Our proposed approach, GraphGarment , allows a robot to learn a garment dynamics model that can predict the next state of the garment given the current garment state and robot action input. This is then used for the cloth manipulation of hanging the garment on a clothes hanger. Existing research has explored various cloth representations to facilitate learning and control. With significant advancements in computer vision, studies have utilized image-based perception techniques, such as corner detection for cloth folding and collar detection for hanging garment [ 11 , 8 ] . However, due to issues such as self-occlusion and the deformable nature of cloth, inferring states directly from raw image inputs is challenging. Additionally, 3D representations such as point clouds have also been employed for garment state estimation [ 12 , 13 ] . Furthermore, recent research has investigated GNN-based approaches to enhance learning and control in this area [ 14 ] . However, these methods, including GNN and point cloud-related approaches, impose high demands on data acquisition to train the dynamics model. In addition, the model learned in simulation is difficult to apply directly to the real-world due to the large sim-to-real gap. To learn the dynamics or perception of complex-shaped garment items, existing approaches typically require high-quality data in the real-world, which could be labour-intensive for data collection. In this paper, we propose GraphGarment : a novel approach that builds two models: a GNN model and a residual model to effectively learn garment dynamics in simulation and the real-world. Specifically, our approach first trains a GNN to predict the garment’s next state based on the current state and action in simulation. With a small amount of real-world data, a residual model is then learned to bridge the sim-to-real gap. Leveraging the predictive model, a model-based action sampling strategy is proposed for the robot to adjust the garment to the optimal pre-hanging configuration. Finally, we utilize a novel robotic bimanual garment-hanging experiment to demonstrate the effectiveness of our method both in simulation and the real-world, as illustrated in Fig. 1 . The key contributions of our proposed GraphGarment approach are: (1) a novel GNN-based model for learning the dynamics of garments; (2) a novel residual learning model to solve the sim-to-real challenges for garment manipulation; (3) a novel model-based action strategy for pre-hanging adjustment for the garment-hanging task. II Related Work II-A Data-driven Approaches for Cloth Manipulation Perception and action planning are fundamental components in the manipulation of objects, particularly when dealing with deformable objects like clothes. Due to its inherent deformation characteristics and high elasticity, cloth presents significant non-linearity, making accurate state estimation challenging. This complexity can result in unpredictable behaviour during manipulation, necessitating sophisticated approaches to ensure precise control. Recent progress in learning-based methods has led to the emergence of various data-driven approaches, which have proven effective in addressing these challenges. Different representations can be used for learning cloth manipulation. For instance, they can be modelled as 1-D objects (such as rope segments), 2-D representations (like images of garments), or 3-D structures (including point clouds for detailed shape and pose estimation). In the context of these representations, research efforts have explored several innovative solutions. Deep learning methods for 1-D rope perception have enabled more accurate tracking of linear and flexible objects [ 15 ] , while 2-D structural detection techniques have enhanced the identification of key regions in garments, such as collar [ 8 ] . Additionally, 3-D garment pose estimation using point clouds has provided an understanding of the spatial arrangement and shape of cloth [ 12 ] . Data-driven control methods such as using reinforcement learning (RL) and imitation learning (IL) for decision-making [ 7 ] , perception for manipulation and grasping [ 8 ] , learning-based model predictive control for human-robot interaction [ 16 ] have also made notable progress in data-driven control within the field of cloth manipulation. Figure 2: This figure demonstrates the overview of GraphGarment : A GNN predicts garment dynamics in simulation, refined by a residual model

Read the complete text