TextureSAM: Towards a Texture Aware Foundation Model for Segmentation
Inbal Cohen †, Boaz Meivar †, Peihan Tu, Shai Avidan, Gal Oren · 2025
Read TextureSAM: Towards a Texture Aware Foundation Model for Segmentation by Inbal Cohen †, Boaz Meivar †, Peihan Tu, Shai Avidan, Gal Oren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16540.txt.
Introduction
Inbal Cohen, Boaz Meivar, Peihan Tu, Shai Avidan and Gal Oren build TextureSAM, a segmentation model that pays heed to texture, not just object shape. Segment Anything Models learned from semantic data sets lean toward shape cues. In fields such as medical imaging, materials and remote sensing, boundaries are often marked by texture change instead.
Their fine-tuning plan alters training images step by step to stress texture features, using a texture-swapped form of the ADE20K dataset. The tuned model guides itself to texture-defined regions and cuts the shape bias of the base model. It beat SAM-2 on both natural and synthetic texture test sets. Code and the altered dataset are to be shared in public.
Catalog record
| CREATOR | Inbal Cohen †, Boaz Meivar †, Peihan Tu, Shai Avidan, Gal Oren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16540
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TextureSAM: Towards a Texture Aware Foundation Model for Segmentation Inbal Cohen †, Boaz Meivar †, Peihan Tu, Shai Avidan, Gal Oren Abstract Segment Anything Models (SAM) have achieved remarkable success in object segmentation tasks across diverse datasets. However, these models are predominantly trained on large-scale semantic segmentation datasets, which introduce a bias toward object shape rather than texture cues in the image. This limitation is critical in domains such as medical imaging, material classification, and remote sensing, where texture changes define object boundaries. In this study, we investigate SAM’s bias toward semantics over textures and introduce a new texture-aware foundation model, TextureSAM, which performs superior segmentation in texture-dominant scenarios. To achieve this, we employ a novel fine-tuning approach that incorporates texture augmentation techniques, incrementally modifying training images to emphasize texture features. By leveraging a novel texture-alternation of the ADE20K dataset, we guide TextureSAM to prioritize texture-defined regions, thereby mitigating the inherent shape bias present in the original SAM model. Our extensive experiments demonstrate that TextureSAM significantly outperforms SAM-2 on both natural ( +0.2 mIoU ) and synthetic ( +0.18 mIoU ) texture-based segmentation datasets. The code and texture-augmented dataset will be publicly available. Image SAM-2 SAM-2* TextureSAM GT Figure 1 : Examples for segmentation of natural images from the Real-World Textured Dataset. Compared with the original SAM (2nd and 3rd rows, SAM-2* indicating modified inference parameters), TextureSAM (4th row) is more adept at recognizing areas defined by texture changes. SAM’s reliance on semantic shape results in fragmented segmentation, while TextureSAM provides segmentation of whole areas defined by texture changes. † † footnotetext: † Equal contribution. 1 Introduction Conventional segmentation models are designed to recognize objects based on semantic features, yet many real-world applications rely on texture, which is inherently difficult to define. Texture can manifest as structured, repeating patterns or irregular, stochastic variations, making it challenging to model in a unified framework. In biomedical imaging [ 4 ] [ 8 ] such as histopathology, radiology, and microscopy—diagnostic tasks depend on subtle texture differences to identify tumors, tissue abnormalities, and cell boundaries, where explicit object edges are often absent. Similarly, in materials science [ 2 , 29 , 30 , 7 ] , metallography and composite analysis require texture-based segmentation to detect grain structures, fractures, and surface irregularities, as these features lack clear semantic categories. In remote sensing, texture is essential for land cover classification and vegetation mapping, while in industrial inspection, defect detection relies on analyzing complex spatial statistics rather than discrete object boundaries. Despite the broad relevance of texture-based segmentation, prior methods have relied on ad-hoc models trained on often small, domain-specific datasets, limiting generalization. No foundation model has been established to provide a unified, transferable texture representation across domains. This work addresses this gap by introducing a protocol that adapts a pretrained foundation model for segmentation into a texture-aware model, enabling the use of learned texture representations across diverse segmentation tasks. Existing segmentation models, including SAM [ 21 ] and its successor SAM-2 [ 28 ] , struggle in texture-driven applications where boundaries are defined by local surface properties rather than distinct object contours. This limitation highlights a fundamental research gap: can a generic foundation model for segmentation be developed that explicitly incorporates texture information rather than relying solely on shape-based priors? We suspect that SAM-2 couples shape and texture cues. That is, it learns to expect that certain objects often have certain texture. Hence, our goal is to try and decouple the two. To do that, we rely on a recently introduced compositional neural texture approach (CNT) [ 32 ] , that can interpolate between a source and style (i.e., texture) images. This way, we can rely on ground truth masks of existing datasets (ADE20k in our case) to dress instances with arbitrary new texture, thus decoupling shape from texture. Equipped with this new and augmented dataset we finetune SAM-2. We call the resulting model, TextureSAM . We report the details of our method, and evaluate TextureSAM on a number of datasets. Contributions. The key contributions of this work are as follows: • A texture-augmented ADE20K dataset is constructed to fine-tune SAM-2 as well as to provide benchmark segmentation performance in texture-aware scenarios. • We present a novel fine-tuning approach to shift SAM to focus on texture cues. • A fine-tuned segmentation model, TextureSAM, is introduced to enhance performance in texture-driven segmentation tasks. • Extensive quantitative and qualitative evaluations demonstrate that TextureSAM significantly outperforms SAM-2 on both natural and synthetic texture-based segmentation datasets. Research Questions. This work investigates the following key research questions: • RQ1: Is Segment Anything shape-biased? Can it comprehend textures? • RQ2: Can a shape-biased foundation model be shifted towards texture-driven segmentation? The remainder of this paper, is organized as follows: Section 2 reviews related work Section 3 details the fine-tuning approach and texture augmentation strategy. Section 4 presents experimental results and comparisons. Section 5 discusses limitations, applications, and future research directions. 2 Prior Work 2.1 Texture Analysis Texture analysis is often associated with semantic segmentation and can be roughly divided into two parts: Semantic Segmentation and Instance Segmentation. In Semantic Segmentation, each pixel is assigned a label of the class (i.e., road, vehicle, sky, person, etc.) it belongs to. In Instance Segmentation, each pixel is assigned a label of the class and the instance it belongs to (i.e., two different people will be assigned different labels). Recently, the two have been combined into a single, holistic approach, termed Panoptic Segmentation, where the goal is to assign each pixel its class label, and in case there are multiple instances of the object, assign the different objects a different label. Panoptic Segmentation is used for various applications such as Autonomous Driving and general image understanding. Another closely related branch of texture analysis is that of Boundary Detection, where the goal is to detect meaningful boundaries/interfaces between textures, cells, crystallites, etc. This branch evolved from the problem of edge detection, where edge detection is concerned with detecting changes in intensity values, while boundary detection is concerned with detecting boundaries between regions (i.e., textures). Traditional methods mostly focus on low-level cues. For example, Martin et al. [ 24 ] proposed a probabilistic boundary (Pb) detection module that learns to detect boundaries in images using local brightness, color, and texture cues. Dollár and Zitnick [ 9 ] use Structured Forests to map local image patches to local edge maps. A sequence of papers [ 3 , 31 , 22 ] suggested using deep features to improve image boundary detection. Common to all is the use of deep features as the space in which to detect boundaries. Exploiting multi-scale representations and dilated convolutions helped He et al. [ 14 ] push the state-of-the-art in the field even further. Recently, Pu et al. adapted Transformers for boundary detection [ 27 ] . Their approach is based on a two-level Transformer architecture, where the first level captures global scene information, and the second refinement level calculates the boundaries. Their network achieves state-of-the-art results on several datasets [ 19 ] , including the BSD dataset [ 1 ] . The latest advancement using Transformers for this task is the Segment Anything Model (SAM) [ 21 ] , which presents a significant step