GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning
Mingleyang Li 1∗ , Yuran Wang 1∗ , Yue Chen 1 , Tianxing Chen 2, Jiaqi Liang 1 , Zishun Shen 1 , Haoran Lu 3 , Ruihai Wu 1† , Hao Dong 1† · 2026
Read GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning by Mingleyang Li 1∗ , Yuran Wang 1∗ , Yue Chen 1 , Tianxing Chen 2, Jiaqi Liang 1 , Zishun Shen 1 , Haoran Lu 3 , Ruihai Wu 1† , Hao Dong 1† (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04158.txt.
Introduction
Li and others propose GarmentPile++, affordance-driven garment retrieval. Garment manipulation matters for home-assistant robotics, yet most work assumes a single garment, while garment piles are far more common. Their pipeline follows language instructions for safe clean retrieval and guarantees exactly one garment per attempt, mixing vision-language thought with affordance. A segmentation model segments the pile for thought cues, with mask fine-tuning, and a dual-arm method handles large garments and sagging. Effectiveness is shown across tasks in real and simulated environments.
Catalog record
| CREATOR | Mingleyang Li 1∗ , Yuran Wang 1∗ , Yue Chen 1 , Tianxing Chen 2, Jiaqi Liang 1 , Zishun Shen 1 , Haoran Lu 3 , Ruihai Wu 1† , Hao Dong 1† |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33554 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04158
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning Mingleyang Li 1∗ , Yuran Wang 1∗ , Yue Chen 1 , Tianxing Chen 2, Jiaqi Liang 1 , Zishun Shen 1 , Haoran Lu 3 , Ruihai Wu 1† , Hao Dong 1† Abstract Garment manipulation has attracted increasing attention due to its critical role in home-assistant robotics. However, the majority of existing garment manipulation works assume an initial state consisting of only one garment, while piled garments are far more common in real-world settings. To bridge this gap, we propose a novel garment retrieval pipeline that can not only follow language instruction to execute safe and clean retrieval but also guarantee exactly one garment is retrieved per attempt, establishing a robust foundation for the execution of downstream tasks (e.g., folding, hanging, wearing). Our pipeline seamlessly integrates vision-language reasoning with visual affordance perception, fully leveraging the high-level reasoning and planning capabilities of VLMs alongside the generalization power of visual affordance for low-level actions. To enhance the VLM’s comprehensive awareness of each garment’s state within a garment pile, we employ visual segmentation model (SAM2) to execute object segmentation on the garment pile for aiding VLM-based reasoning with sufficient visual cues. A mask fine-tuning mechanism is further integrated to address scenarios where the initial segmentation results are suboptimal. In addition, a dual-arm cooperation framework is deployed to address cases involving large or long garments, as well as excessive garment sagging caused by incorrect grasping point determination, both of which are strenuous for a single arm to handle. The effectiveness of our pipeline are consistently demonstrated across diverse tasks and varying scenarios in both real-world and simulation environments. Project page: https://garmentpile2.github.io/ . I INTRODUCTION As a key aspect of general home-assistant robot capabilities, garment manipulation is highly challenging because of its near-infinite space of self-deformable states and complex kinematic and dynamic properties. These properties become even more pronounced in cluttered scenarios, where garments are prone to entanglement with more complicated states, making successful retrieval challenging. Although significant progress has been made in garment manipulation including folding [ 38 , 4 ] , flinging [ 13 , 29 ] , hanging [ 31 , 8 ] , or dressing humans [ 26 , 16 ] , most of the existing work assumes the manipulation of a single garment. However, garment piles are far more common in everyday environments, significantly limiting the real-world deployability of these approaches. Therefore, the ability to retrieve individual garment from a pile is of critical importance. A recent work, GarmentPile [ 35 ] , presents a promising approach for cluttered garment manipulation using a per-point affordance model to predict retrieval actionability. Furthermore, For scenarios where the overly complex cluttered state of garments leaves no suitable manipulation region, an additional adaptation model is trained to perform simple adjustments to the initial garment pile, boosting the performance of method. However, due to the specificity of their task scenarios (e.g., placing garments into washing machine), their approach doesn’t consider a dedicated mechanism for single garment retrieval, often resulting in the retrieval of multiple garments simultaneously, which in turn limits the applicability of some downstream tasks. Moreover, due to its reliance on visual affordance alone, the method lacks integration with language, significantly constraining its flexibility. Furthermore, The garment retrieval process is based solely on a single arm, which struggles to handle large/long garments, therefore compromises the robustness and overall performance of the method in various types of garments. To overcome these limitations, we propose GarmentPile++, a novel pipeline which not only leverages the high-level reasoning capabilities of VLMs, but also exploits the low-level manipulation accuracy and generalization of visual affordance models. As shown in Fig. , we structure the process into three stages: (1) Which to Retrieve: By integrating visual segmentation model (SAM2 [ 23 ] ) with VLM-based reasoning, we identify the garment which is suitable for retrieval under task constraints. Furthermore, to address the potential inaccurate segmentation caused by heavy occlusion and color similarity in garment pile, we introduce a mask fine-tuning procedure, which utilizes SAM2 video tracking and point-prompt segmentation to get better segmentation results. (2) Where to Retrieve: To maximize the feasibility of single-arm retrieval while ensuring garment safety and cleanliness, the Retrieval Affordance Model is trained to execute the garment grasping, in which the affordance representation is enhanced by incorporating segmentation mask feature into the garment pile’s point cloud, enabling seamless integration into a language-guided manipulation pipeline. (3) How to Retrieve: The dual-arm system is deployed for handling different garments and boosting the robustness of pipeline. After master-arm grasp and lift, the current scene situation will be reasoned by VLM for deciding whether to trigger dual-arm cooperation or not. If triggered, a tracking selection procedure will be used for selecting cooperation point for slave-arm grasp. What’s more, the grasp of more than one garments will be detected by VLM to terminate current retrieval and continue next attempt. We design two typical task scenarios encountered in daily life: open-boundary and closed-boundary. Within these scenarios, we define two retrieval tasks: 1) retrieve garments sequentially, 2) retrieve specific garment. These settings collectively cover nearly all practical cases of garment pile retrieval in everyday environments. Extensive experiments demonstrate the effectiveness of our pipeline. In summary, our main contributions are as follows: • we propose a novel pipeline that follows language instruction to execute safe garment retrieval while ensuring exactly one garment is retrieved per attempt, establishing a robust foundation for downstream tasks. • Our pipeline seamlessly combines vision-language reasoning with visual affordance, fully exploiting the high-level reasoning and planning capabilities of VLMs alongside the generalization power of affordance for low-level grasping. • Through extensive experiments in both real-world and simulated environments, we demonstrate the effectiveness of our method by successfully tackling two distinct tasks across the two designed scenarios. II RELATED WORK II-A Deformable Object and Garment Manipulation Deformable object manipulation poses significant challenges due to complex dynamics and high-dimensional state spaces [ 6 ] . Compared with manipulating 1D cables and ropes [ 25 , 36 , 40 ] or square-shaped fabrics [ 11 , 17 , 34 , 30 ] , garments with diverse categories, shapes, and deformations are significantly more challenging to handle. Current research primarily focuses on manipulating single garments, proposing methods for folding [ 1 , 38 ] , unfolding [ 3 , 13 , 18 ] , hanging [ 7 , 32 ] , and dressing [ 16 , 26 , 28 ] . Recent advancements in simulation platforms like GarmentLab [ 19 ] and DexGarmentLab [ 29 ] have enabled systematic evaluation of garment manipulation techniques. However, most studies focus on isolated garments in structured environments. In this paper, we address the more challenging problem of cluttered garment manipulation. II-B Cluttered Environment Manipulation Robotic manipulation in cluttered environments is a fundamental capability for tasks such as grasping [ 27 , 39 ] , retrieving [ 37 , 14 ] , and rearranging [ 12 , 9 ] , yet it remains highly challenging due to severe occlusions, tight object arrangements, and unstable configurations. Previous works have addressed these difficulties through methods such as Visual-Language-Action Model (VLA) [ 10 ] , multimodal reasoning [ 22 ] , and