Attention-Guided Integration of CLIP and SAM for Precise Object Masking in Robotic Manipulation
1 st Muhammad A. Muttaqien 1 2, 2 nd Tomohiro Motoda, 3 rd Ryo Hanai, 4 th Domae Yukiyasu · 2025
Read Attention-Guided Integration of CLIP and SAM for Precise Object Masking in Robotic Manipulation by 1 st Muhammad A. Muttaqien 1 2, 2 nd Tomohiro Motoda, 3 rd Ryo Hanai, 4 th Domae Yukiyasu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18842.txt.
Introduction
Muhammad Muttaqien and co-authors build a pipeline that outlines store products for robot hands in a 2025 paper. A robot in a convenience store must pick the exact item asked for, so it needs a clean mask of that item in its camera view. Two known AI models anchor the pipeline: one links images with text, the other cuts out regions.
What is new is the join. A gradient based attention method guides where the cutter should work, and a custom data set tunes the pair for store goods. The masks feed straight into robot control. The authors frame the work as a step toward precise, adaptive handling of small products.
Catalog record
| CREATOR | 1 st Muhammad A. Muttaqien 1 2, 2 nd Tomohiro Motoda, 3 rd Ryo Hanai, 4 th Domae Yukiyasu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-18842
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Attention-Guided Integration of CLIP and SAM for Precise Object Masking in Robotic Manipulation 1 st Muhammad A. Muttaqien 1 2, 2 nd Tomohiro Motoda, 3 rd Ryo Hanai, 4 th Domae Yukiyasu Abstract This paper introduces a novel pipeline to enhance the precision of object masking for robotic manipulation within the specific domain of masking products in convenience stores. The approach integrates two advanced AI models, CLIP and SAM, focusing on their synergistic combination and the effective use of multimodal data (image and text). Emphasis is placed on utilizing gradient-based attention mechanisms and customized datasets to fine-tune performance. While CLIP, SAM, and Grad-CAM are established components, their integration within this structured pipeline represents a significant contribution to the field. The resulting segmented masks, generated through this combined approach, can be effectively utilized as inputs for robotic systems, enabling more precise and adaptive object manipulation in the context of convenience store products. 1 1 footnotetext: Also affiliated with the Department of Computer Science, University of Tsukuba, Tsukuba, Japan. I Introduction In recent years, the ability to recognize and manipulate specific objects within well-defined domains, such as products in convenience stores, has become increasingly important in the field of robotic manipulation [ 1 ] [ 2 ] [ 3 ] . As robots are expected to perform more complex tasks in diverse environments, the need for precise object identification and interaction grows, particularly in domains where a high level of accuracy is crucial. For instance, in convenience stores (Figure 1 ), robots must reliably identify and handle a wide variety of products, each with unique visual characteristics, to automate tasks such as stocking, sorting, and customer assistance. The advent of large foundation models like CLIP (Contrastive Language-Image Pretraining) [ 4 ] and SAM (Segment Anything Model) [ 5 ] has opened new possibilities for improving object recognition and segmentation tasks. These foundation models, which are pre-trained on vast datasets, provide a robust framework for tackling complex AI challenges. CLIP’s ability to align images and text through a shared embedding space, combined with SAM’s flexibility in segmenting any part of an image, offers a strong foundation for developing advanced robotic systems. Fine-tuning CLIP for specific applications within a system can enhance the precision of the prompts that guide SAM in producing accurate object masks, which are critical for robots to perform complex manipulation tasks more accurately and reliably. Fig. 1 : A convenience store scene displaying various products used for segmentation tasks in our laboratory. However, integrating language and vision in multi-modal tasks presents significant challenges. While models like CLIP and SAM are powerful individually, their real-world use is limited by a key bottleneck: the lack of domain-specific data, such as object detection, grounding, and caption data, which are essential for fine-tuning. Existing pipelines like GroundedSAM [ 6 ] , FastSAM [ 7 ] , or even YOLO World [ 8 ] integrated with SAM perform well in generalized settings but struggle in highly specific environments, such as convenience stores, where the variety of objects and their contextual relevance differ significantly from the data these models were trained on. The difficulty lies in the fact that preparing such domain time-consuming and labor-intensive. In this paper, we propose a novel pipeline that integrates fine-tuned CLIP and SAM, enhanced by gradient-based attention mechanisms, to address the challenges of object masking in robotic manipulation. Focusing on convenience stores, our method optimizes performance with customized datasets and task-specific fine-tuning, overcoming the data limitations faced by existing pipelines. This allows for more precise and reliable robotic systems in real-world applications. The remainder of this paper discusses our approach, the challenges overcome, and the potential applications for enhancing robotic systems. Fig. 2 : Summary of our system integration. The diagram illustrates the flow from input processing in CLIP to the generation of accurate segmentation masks by SAM, guided by Grad-CAM attention maps. II Related Work The development of object recognition, segmentation, and multimodal AI has been significantly shaped by pioneering models like AlexNet [ 9 ] , VGGNet [ 10 ] and ResNet [ 11 ] , which introduced deep architectures capable of capturing intricate visual patterns. VGGNet, known for its deep and simple structure, uses small convolution filters to achieve high accuracy, while ResNet introduced residual learning, enabling deeper networks without the vanishing gradient problem. These advancements in convolutional neural networks (CNNs) have laid the groundwork for more sophisticated models that are crucial for tasks such as object manipulation in complex environments like convenience stores. Object detection has also evolved with the introduction of models such as Faster R-CNN [ 12 ] and YOLO [ 13 ] , both of which have significantly influenced the field. Faster R-CNN integrates region proposal networks with CNNs to enhance detection accuracy, making it a preferred choice for tasks requiring precision. In contrast, YOLO redefined object detection by framing it as a single regression problem, allowing for real-time performance. The progression of object detection models has been further supported by attention mechanisms, particularly through the Transformer architecture and its adaptation to visual tasks with Vision Transformers (ViT) [ 14 ] . ViT demonstrated that self-attention could effectively capture global dependencies in images, offering a new approach to image recognition that reduces reliance on convolutional operations. The integration of language and vision has opened new possibilities in multimodal AI, with models like CLIP, GLIP [ 15 ] , and DINO [ 16 ] playing a central role. CLIP, developed by OpenAI, showed the power of aligning text and image representations, enabling robust zero-shot learning across various vision tasks. Building on this, GLIP enhanced the grounding of language in visual contexts, making it more effective for object detection in multimodal scenarios. DINO and its extension, Grounding DINO [ 17 ] , explored self-supervised learning for visual representations, emphasizing the importance of grounding visual concepts in multimodal inputs. These advancements have significantly enhanced the capabilities of systems that require a deep understanding of both visual and textual information. Segmentation, a crucial aspect of object manipulation, has advanced significantly with models like the SAM by Meta AI. SAM represents a major leap in segmentation technology, delivering high-quality results across a wide range of objects and scenes. Trained using a vast and diverse dataset, SAM enhances its ability to generalize across various environments. These advanced segmentation models are particularly relevant in environments like convenience stores, where precise object detection and manipulation are critical. To ensure the reliability and interpretability of these complex systems, techniques like Grad-CAM [ 18 ] have been developed to provide visual explanations of model decisions. Grad-CAM highlights the regions of an image that influence a model’s output, offering insights into the model’s decision-making process. This explainability is essential in multimodal AI systems, where understanding the model’s focus areas can lead to more trustworthy and effective applications. III Foundation Model Foundation models have revolutionized artificial intelligence by providing versatile, pre-trained models that can be adapted across various tasks and domains. These models, trained on extensive datasets, capture a wide range of patterns and representations, making them invaluable for complex applications like robotic manipulation. Their scalability and ability to be fine-tuned for specific tasks allow for significant advancements in performance, particularly in environments requiring precise