Patch-sampled contrastive learning for dense prediction pretraining in metallographic images Authors: Mingchun Li, Yang Liu, Dali Chen, Jining Bao, Yan Huo Published: 2025 ABSTRACT The shapes, sizes, and distributions of microstructures determine the mechanical properties of alloy products, necessitating identification of microstructures in metallographic images during the manufacturing process. Previous studies demonstrated the success of deep learning in analyzing such images. However, existing methods do not fully consider the high cost of annotations, which limits the application of deep learning-based methods. Considering the popularity of self-supervised learning for natural images, further research is required to design a microstructure-specific pretraining framework. To this end, we propose a novel patch-sampled contrastive learning (PSCL) method, considering the characteristics of metallographic image segmentation tasks. First, image- and patch-level contrastive learning frameworks are designed to help the model effectively capture visual features. The former captures global features, whereas the latter captures local features in greater detail. Second, a multiscale strategy is introduced to improve the adaptability of the subsequent microstructure segmentation. Finally, considering that patches may concurrently contain multiple microstructures, a sampling method based on feature similarity was designed to capture the discriminable features of different categories. In experiments, after model fine-tuning of only one annotated image, the proposed method achieved a Dice score of 0.6296 which is higher than those of existing self-supervised learning methods with identical model structure, proving the effectiveness of the method in microstructure segmentation. The source code and dataset of the proposed PSCL in this work are available at https://github.com/neulmc/PSCL. Introduction Metallographic images are microscopic images that reveal the microstructures of materials. They serve as an important tool for identifying and measuring these microstructures1. Considering the significance of microstructures in relation to mechanical properties of materials, analyzing metallographic images is an essential part of the manufacturing process2–4. In this study, the research object was an aluminum-magnesium-silicon alloy with good forming and processing properties5,6. The observed metallographic images after casting and homogenization heat treatments are shown in Fig. 1. Fig. 1Metallographic images and corresponding annotations for the three types of microstructures. Metallographic images and corresponding annotations for the three types of microstructures. In Fig. 1, three types of particles are observed: silicon particles, a magnesium-silicon phase, and an Fe-containing phase densely appearing in the aluminum matrix. In the grayscale images, the magnesium-silicon phase is the darkest, followed by the silicon particles, whereas the iron-containing phase is relatively light-colored. In addition, the proportion of silicon particles is the largest, and the distribution is relatively uniform, whereas the Mg–Si and Fe-containing phases are relatively small in proportion. These microstructures are of great significance for analyzing product quality. However, because the grayscale is relatively close, distinguishing between the three often requires material experts. Manually measuring and identifying these individually is a time-consuming and laborious task7. To reduce the amount of manual labor, scholars have attempted to convert this into a computer vision problem, designing corresponding algorithms8–10. Early studies mainly adopted rule-based and traditional machine-learning methods. Typical methods used for metallographic image segmentation include watershed11, level sets12, and support vector machine13. Such methods14–18 are easy to implement and do not require large datasets. However, most of them can only be used to deal with relatively simple problems, such as foreground and background segmentation with significant inherent inter-class differences. When dealing with more complex problems, promising performance is often difficult to obtain. Deep learning has become increasingly popular for natural images19. Its application to complex metallographic image analysis has also attracted considerable attention. Early researchers used deep models as feature extractors for traditional classifiers20. Gradually, several studies adopted an end-to-end approach for mainstream deep models to solve relevant problems. For example, in 2022, Luengo1 designed an image segmentation method that includes multiple convolutional networks, such as UNet21, DeepLabV322, and the feature pyramid network (FPN)23 to solve the problem of microstructure analysis of steel materials. Recently, vision transformer models have been used to solve specific problems24,25. Thus, methods26–30 based on deep learning have become the mainstream for solving metallographic image-related tasks. However, most existing metallographic image analysis methods are designed under supervised conditions31,32. In particular, for segmentation problems that require pixel-level annotation, the upfront cost of acquiring deep models is typically high. Deep-learning methods typically outperform traditional methods. However, their drawbacks still limit practical applications. Unlike natural images where patches often contain distinct semantics (e.g., “cat” vs. “tree”), metallographic microstructures exhibit dense distributions with minimal inter-class variation (Fig. 1). This motivates our supervised-assisted sampling strategy to ensure meaningful contrastive pairs. Some scholars have attempted to introduce strategies such as semi-supervised, unsupervised and few-shot learning33–39. Recent works employ diffusion models to implement anomaly and defect detection, taking into account the extensive high-precision annotations in real-world scenarios40,41. However, allowing the model to capture effective visual representations in a metallographic context remains an open problem. Fortunately, recent self-supervised learning has provided an effective pretraining paradigm42, capturing effective visual representations in advance without annotations through contrastive learning43–46 or masked image modeling47–49, following by fine-tuning the model using labeled data. For example, self-supervised learning brings the features of augmented views of the same image closer together. It also designs parameter updates to prevent degradation42. Such methods have attracted considerable attention in natural image-classification tasks. Subsequently, to improve the performance of self-supervised methods in dense prediction tasks, patch-based contrastive learning or generation methods were studied to strengthen local information50,51. When the cost of supervised information is high, this type of method is popular for solving natural imaging tasks. Existing self-supervised methods do not entirely work on metallographic images. Natural images52 or other types of images53,54 often contain patches with discriminative semantic information. In contrast, metallographic images have densely distributed microstructures. For example, even without supervision information, it would be reasonable to assume that two randomly selected natural images or small patches contain different categories in many datasets. However, in metallographic images, the differences between the samples are very small. Distinguishing these instances with the same semantics may lead to negative effects. Therefore, the instances used for contrastive learning should be carefully selected. Considering that the actual segmentation task following pretraining is clear, an intuitive idea is to introduce supervised information in advance. To solve this problem, self-supervised learning methods for metallographic image segmentation require specific designs. In this study, a novel contrastive learning method called patch-sampled contrastive learning (PSCL) is proposed for metallographic image segmentation tasks. Unlike the original self-supervised learning framework without manual annotations, a small number of labeled images are used to enhance the effect of pretraining. Unlabeled patches are sampled based on feature similarity with such supervised information, and instances with different semantics are carefully selected for contrastive learning. To adapt to microstructure segmentation tasks, our method applies contrastive learning to both image-level (global) and patch-level (local) features. We also introduce multiscale strategies to improve sensitivity to differently sized microstructures. In general, the contributions of this study are summarized as follows: A novel pretraining strategy for metallographic image segmentation is designed, thereby simultaneously extracting image- and patch-level visual features through instance discrimination contrastive learning, to capture effective global and local features, respectively.Considering the characteristics of microstructures in metallographic images, a supervised assisted patch-sampling strategy is proposed. Through feature similarity with labeled patches, unlabeled patch-level instances are carefully selected to ensure meaningful negative and positive pairs for distinguishing the microstructures.A concise encoder-decoder structure is designed for the microstructure segmentation task. By introducing an effective multiscale strategy in contrastive learning, the model becomes sensitive to objects of different sizes in the pretraining stage, which results in better performance in subsequent segmentation tasks. A novel pretraining strategy for metallographic image segmentation is designed, thereby simultaneously extracting image- and patch-level visual features through instance discrimination contrastive learning, to capture effective global and local features, respectively. A novel pretraining strategy for metallographic image segmentation is designed, thereby simultaneously extracting image- and patch-level visual features through instance discrimination contrastive learning, to capture effective global and local features, respectively. Considering the characteristics of microstructures in metallographic images, a supervised assisted patch-sampling strategy is proposed. Through feature similarity with labeled patches, unlabeled patch-level instances are carefully selected to ensure meaningful negative and positive pairs for distinguishing the microstructures. Considering the characteristics of microstructures in metallographic images, a supervised assisted patch-sampling strategy is proposed. Through feature similarity with labeled patches, unlabeled patch-level instances are carefully selected to ensure meaningful negative and positive pairs for distinguishing the microstructures. A concise encoder-decoder structure is designed for the microstructure segmentation task. By introducing an effective multiscale strategy in contrastive learning, the model becomes sensitive to objects of different sizes in the pretraining stage, which results in better performance in subsequent segmentation tasks. A concise encoder-decoder structure is designed for the microstructure segmentation task. By introducing an effective multiscale strategy in contrastive learning, the model becomes sensitive to objects of different sizes in the pretraining stage, which results in better performance in subsequent segmentation tasks. Methods In this study, a contrastive learning approach based on supervised assistant patch sampling is proposed. This is a pretraining method for metallographic image segmentation. The exact process, which employs a type of UNET for microstructure segmentation, is illustrated in Fig. 2. To reduce the demand for manual labeling, a pretraining method was designed based on contrastive learning to capture task-related visual representations. This is achieved by making the extracted features of the two views of the same image closer. To meet the characteristics of microstructural analysis, both global image-level and local patch-level features with multiscale strategy were considered. Furthermore, given that microstructures are densely distributed in metallographic images, supervised information is introduced to enhance the discrimination between microstructures. In the following section, the proposed method is described in detail. Fig. 2The overall schematic diagram of the proposed contrastive learning method and the adopted UNet network structure. The overall schematic diagram of the proposed contrastive learning method and the adopted UNet network structure. Image-level contrastive learning Contrastive learning is a popular framework for self-supervised learning. In computer vision tasks, contrastive learning has better linear separability than masked image modeling, such as image reconstruction. Representative implementations include MoCo44 and SimCLR55, which have achieved significant success on natural image datasets. Considering the effectiveness of these methods in image classification tasks, we introduced contrastive learning to capture the global features of metallographic images. Global contrastive learning enhances feature discriminability by minimizing distances between features from different augmented views of the same image. Simultaneously, it maximizes separation from features of other images in the embedding space. Specifically, image enhancement methods were used for a metallographic image, to obtain two instances. In the network shown in Fig. 2, these instances are fed into the encoder and projection head functions. The encoder is used to extract the global metallographic image features, which are retained for the subsequent fine-tuning stage. The head function is removed once the pretraining is completed. During the pretraining stage, two views are fed into the same encoder (SimCLR framework), or the respective networks with moment updates to avoid degradation (MoCo framework). The overall implementation process is illustrated in Fig. 2. Formally, the size of a metallographic image is assumed to be \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:h\times\:w$$\end{document}, and its two augmented instances with different views are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{i}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{j}$$\end{document}. Using the encoder and global head functions, they are encoded into vectors \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:q,k$$\end{document} with the same dimension. MoCo, SimCLR, and InfoNCE56 were used for instance discrimination. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathrm{s}\mathrm{i}\mathrm{m}\left(u,v\right)={u}^{T}v/ \vert \vert u \vert \vert \vert \vert v \vert \vert$$\end{document} denote the cosine similarity, implemented as follows:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{q}^{ID}=-\mathrm{log}\frac{\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left(q,{k}_{+}\right)/\tau\:\right)}{\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left(q,{k}_{+}\right)/\tau\:\right)+{\sum\:}_{{k}_{-}}\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left(q,{k}_{-}\right)/\tau\:\right)},$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\tau\:$$\end{document} denotes a temperature parameter; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{k}_{+}$$\end{document} is the instance corresponding to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:q$$\end{document}; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{k}_{-}$$\end{document} denotes the other augmented instance, and the term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:ID$$\end{document} is contrastive learning based on image-level instance discrimination. For natural image datasets, considering that images typically belong to different categories, the instance discrimination loss in (1) has been proven to be effective44. At this point, the positive sample pairs have the same semantic information, whereas negative sample pairs usually belong to different categories, regardless of whether they come from a historical dictionary (MoCo) or different instances in the current batch (SimCLR). However, the metallographic images in this study were from the same type of aluminum alloy. Some of the images observed were from metallographic samples obtained using the same manufacturing process parameters, making it inappropriate to distinguish between them solely based on whether they are from the same sample. Therefore, a specific contrastive learning strategy for metallographic images must be designed. To capture an effective global visual representation of metallographic images, an additional process-discriminative contrastive learning loss function was designed. The core idea was to strengthen the correlation of metallographic images obtained from the same manufacturing process. Considering that the sources of the images are known in advance, constructing such a loss function requires slight adjustments to existing functions. The function of the instance discrimination contrastive loss in (1) can be expanded as follows:2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{q}^{PD}=-{\sum\:}_{{k}_{p+}\in\:Pr\left(q\right)}\mathrm{log}\frac{\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left(q,{k}_{p+}\right)/\tau\:\right)}{\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left(q,{k}_{p+}\right)/\tau\:\right)+{\sum\:}_{{k}_{-}}\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left(q,{k}_{p-}\right)/\tau\:\right)},$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{k}_{p+}$$\end{document} denotes the augmented instance with the same manufacturing process of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:q$$\end{document}, whereas \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{k}_{p-}$$\end{document} refers to other instances. The term \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:PD$$\end{document} is contrastive learning based on manufacturing process discrimination, and the function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Pr\left(q\right)$$\end{document} returns a collection of the instances with the same manufacturing process of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:q$$\end{document}. Compared with (1), the process-discriminative loss function can be considered a multilabel classification task. In (2), instances observed from the same manufacturing process \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Pr\left(q\right)$$\end{document} are considered to be positive sample pairs \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left(q,{k}_{p+}\right)$$\end{document}, rather than just from the same image. Different manufacturing processes alter microstructure morphology. Our strategy leverages this to improve global visual representation learning. In practice, both instance and process discriminative functions are used to strengthen common instance visual representation and process consistency, respectively. The fused loss function is expressed as follows:3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{q}={L}_{q}^{ID}+\gamma\:{L}_{q}^{PD},$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\gamma\:$$\end{document} is a preset parameter for balancing the two loss functions. Patch-level contrastive learning To analysis the microstructure quantitatively, a deep network is used to achieve a dense prediction of the metallographic images. Considering that image-level contrastive learning focuses more on capturing global visual representations, corresponding training strategies should be introduced to strengthen locality. Recently, scholars designed various patch-contrastive strategies based on transformer structures57 and convolutional neural networks (CNNs)50. Inspired by these studies, we propose a multiscale patch-level contrastive learning method for decoders. Specifically, for the UNet network shown in Fig. 2, a local projection head function is introduced. It is placed in the decoder and composed of activation function, convolution layer, and batch normalization layer. Assuming that the size of the feature map in stage \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:s$$\end{document} is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{h}^{s}\times\:{w}^{s}\times\:{c}^{s}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{c}^{s}$$\end{document} represents the feature dimension, the dimensions of the head function output \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}^{s}$$\end{document} at this stage are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{h}^{s}\times\:{w}^{s}\times\:{c}^{d}$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{c}^{d}$$\end{document} is the transformed feature dimension. For the enhanced instance, the features output by the decoder are fed into the head function to obtain the transformed feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}^{s}$$\end{document}. The patch-level contrastive loss function is expressed as follows:4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{p}^{s}=-\frac{1}{{h}^{s}{w}^{s}}{\sum\:}_{i}^{{h}^{s}{w}^{s}}\mathrm{log}\frac{\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left({p}_{i}^{s},{d}_{i+}^{s}\right)/\tau\:\right)}{\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left({p}_{i}^{s},{d}_{i+}^{s}\right)/\tau\:\right)+{\sum\:}_{{d}_{i-}^{s}}\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left({p}_{i}^{s},{d}_{i-}^{s}\right)/\tau\:\right)},$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:i$$\end{document} represents the index of the internal patches in the image, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({p}_{i}^{s},{d}_{i+}^{s}\right)$$\end{document} is the positive pair for patch \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:i$$\end{document} in the feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}^{s}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}^{s}$$\end{document}, and the corresponding negative pair is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({p}_{i}^{s},{d}_{i-}^{s}\right)$$\end{document}. In (4), the object of contrastive loss is a small region originating from a certain position in the feature map. Unlike the global loss, it is used to enhance the local information in the decoder. Considering the inherent locality of convolutional networks, this is regarded as patch-level contrastive loss. Here, the positive pair refers to the patch features extracted at the same position in two augmented instances. A negative pair refers to the patch features extracted from other positions. The local head function removes the global pooling layer and retains the local information for subsequent dense predictions. Furthermore, to enhance sensitivity to microstructure of different sizes, a multiscale strategy was introduced for patch-level contrastive learning. Specifically, the UNet decoder structure shown in Fig. 2 contains three upsampling stages. After each upsampling, the feature map size is doubled until the original metallographic image size is restored. We implemented the patch-level contrastive loss by setting a patch head function at the end of each stage. The contrastive loss function is expressed as follows:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{p}=\frac{1}{S}{\sum\:}_{s=1}^{S}{L}_{p}^{s},$$\end{document} where the upper character \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:S=3$$\end{document} denotes the total number of upsampling stages in UNet. Supervised-assisted sampling strategy The proposed model pretraining method for aluminum alloy metallographic image segmentation comprises two components: image-level contrastive learning and patch-level contrastive learning. The former is designed to capture global features by constructing image-level pairs \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left(q,{k}_{+}\right)$$\end{document}, while the latter aims to capture local features to facilitate subsequent semantic segmentation tasks by forming patch-level pairs \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\left({p}_{i},{d}_{i+}\right)$$\end{document}, as illustrated in Fig. 3. In the upper section of Fig. 3, the objective of contrastive learning is to align the global features of the metallographic sample with its augmented instance, while distancing them from the features of other images through (1). This approach enables the model to acquire effective feature representations at the image level, such as differentiating metallographic samples subjected to varying process parameters. Fig. 3Image-level and patch-level contrastive learning methods of PSCL. Image-level and patch-level contrastive learning methods of PSCL. In the lower section of Fig. 3, we illustrate the proposed sampling method for patch-level contrastive learning. This method markedly differs from existing approaches like DenseCL. Specifically, the proposed method does not consider all patches from different positions or other images as negative samples; instead, it employs a supervised assisted sampling strategy. The supervised metallographic sample pair is denoted as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:({x}_{sup},{y}_{sup})$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{y}_{sup}\in\:{R}^{h\times\:w\times\:N}$$\end{document} represents pixel-level annotations of the same dimensions as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{sup}$$\end{document}, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:N=4$$\end{document} corresponds to the number of categories (three types of microstructures and the background). During the pretraining phase, the image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{x}_{sup}$$\end{document} is sequentially fed into the encoder and decoder, yielding the feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Gp}_{sup}^{s}$$\end{document} at stage \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:s$$\end{document}. Concurrently, the annotation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{y}_{sup}$$\end{document} is downscaled to the corresponding scale \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{y}_{sup}^{s}$$\end{document} via nearest-neighbor sampling. Utilizing the positional information from \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Gp}_{sup}^{s}$$\end{document}, four vector sets \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Gp}_{Si}^{s},G{p}_{Mg}^{s},{Gp}_{Fe}^{s},{Gp}_{Al}^{s}$$\end{document} are constructed from the feature map \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Gp}_{sup}^{s}$$\end{document} for each category. The subscripts \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Si$$\end{document} denote silicon particles; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Mg$$\end{document} represents the magnesium-silicon phase; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Fe$$\end{document} corresponds to the iron-containing phase; and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Al$$\end{document} signifies the aluminum matrix. The notation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Gp}_{n}^{s}=\{g{p}_{n,1}^{s},{gp}_{n,2}^{s},\cdots\:,{gp}_{n,T}^{s}\}$$\end{document} is used to represent the general case for class \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:n$$\end{document}, with each set having a length of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:T$$\end{document}. The sampling strategy for unsupervised patches is as follows:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{n,j}^{s}=p\left[\mathrm{arg}\underset{i}{\mathrm{max}}\:\mathrm{s}\mathrm{i}\mathrm{m}\left({p}_{i}^{s},{gp}_{n,j}^{s}\right)+\delta\:\right],$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{n,j}^{s}$$\end{document} means the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{j}{\rm th}$$\end{document} sample of class \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:n$$\end{document} at stage \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:s$$\end{document}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\delta\:\sim U \left({0,1}\right)$$\end{document} is a random variable that follows a uniform distribution to increase randomness. In (6), unlabeled patches are sampled based on their similarity to the supervised features. This sampling strategy enables the collection of unlabeled patches with potential semantic information. During contrastive learning, positive samples and their augmented instances share the same semantics, while negative samples are derived from patch sampling that is more akin to supervised patches of other categories. To maintain class balance, Eq. (6) is executed multiple times to randomly select patches via the variable \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\delta\:$$\end{document} until an equal number of patches per class is obtained. Notably, the supervised features utilized for segmentation constitute a set rather than a single representative feature. The diversity of the selected patches is enhanced by comparing them individually. Consequently, (4) and (5) can be reformulated as follows:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{p}=-\frac{1}{S}{\sum\:}_{s=1}^{S}\frac{1}{T\left(s\right)N}{\sum\:}_{n}^{N}{\sum\:}_{j}^{T\left(s\right)}\mathrm{log}\frac{\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left({p}_{n,j}^{s},{d}_{n,j+}^{s}\right)/\tau\:\right)}{\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left({p}_{n,j}^{s},{d}_{n,j+}^{s}\right)/\tau\:\right)+{\sum\:}_{{d}_{n,j-}^{s}}\mathrm{exp}\left(\mathrm{s}\mathrm{i}\mathrm{m}\left({p}_{n,j}^{s},{d}_{n,j-}^{s}\right)/\tau\:\right)},$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:T\left(s\right)$$\end{document} is used to control the number of samples at different stages and is set to an exponential form, returning \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:T,2T,4T$$\end{document} when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:s=\mathrm{1,2},3\:\:$$\end{document}and so on. Compared to (4), the contrastive learning objects \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{n,j}^{s}$$\end{document} in (7) are patches with potential semantic information pertaining to class \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:n$$\end{document}, rather than all patches \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{p}_{i}^{s}$$\end{document}. The negative sample set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{d}_{n,j-}^{s}$$\end{document} comprises patches more similar to other categories, as gauged by their similarity with supervised patches \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{Gp}_{sup}^{s}$$\end{document}. This approach offers two advantages. Firstly, it narrows the intra-class distance and widens the inter-class distance, capturing feature representations that aid in distinguishing different microstructures. Secondly, since only a subset of patches is sampled instead of all, contrastive learning can be applied to larger feature maps without increasing computational costs. This facilitates the model’s ability to capture effective features of some small-sized microstructures. Pretraining and fine-tuning implementation The segmentation of microstructures comprises two stages: pretraining based on contrastive learning and fine-tuning under a few supervised samples. Pretraining uses unlabeled data to learn visual representations. Fine-tuning then adapts these for segmentation tasks. In this section, the implementation details for pretraining and fine-tuning are introduced. For the pretraining stage, we introduced image- and patch-level contrastive loss under the sampling strategy. The image-level contrastive loss captures effective global features by bringing different instances of a sample or production process closer to the encoder. Patch-level contrastive loss focuses on restoring the local information in the decoder and enhances the effect through multiscale sampling strategies. The loss function of the pretraining stage is as follows:8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:L=(1-\alpha\:){L}_{q}+\alpha\:{L}_{p},$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\alpha\:$$\end{document} is a weighting parameter used for balancing the two loss functions. Overall, a contrastive learning pretraining method was designed based on the characteristics of the microstructure analysis field. The proposed patch sampling strategy is not limited to specific contrastive learning frameworks. This work considers implementing the proposed strategy under the MoCo and SimCLR frameworks, which are two landmark works in the field of self-supervised learning. When constructing a negative sample set, the former uses the momentum encoder to generate and store in a feature queue. These negative samples do not undergo gradient updates during backpropagation. And SimCLR utilizes negative samples from the current mini-batch, and all of them are subjected to gradient updates during backpropagation. These two methods have achieved significant results on natural images and have a wide range of applications. In order to comprehensively evaluate the effectiveness of the proposed strategies, implementation algorithms based on these two existing frameworks were designed. Among them, the implementation details of MoCo are provided in Algorithm 1. Algorithm 1Pseudocode of PSCL-MoCo. Pseudocode of PSCL-MoCo. Algorithm 1 presents the pseudocode that uses MoCo to implement the method. For brevity, the SimCLR version is not presented. The main difference between MoCo and SimCLR is the collection of negative samples. The MoCo algorithm relies on dynamically updated historical sequences. The negative SimCLR samples are obtained from the current training batch. In addition, MoCo uses the momentum of the twin network to update the model parameters (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\stackrel{\sim}{E}\left(\cdot \:\right),\stackrel{\sim}{D}\left(\cdot \:\right),\stackrel{\sim}{h}g\left(\cdot \:\right),\stackrel{\sim}{h}l\left(\cdot \:\right))$$\end{document} in Fig. 2, whereas SimCLR is more concise. Moreover, for the method implemented in SimCLR, process-discriminative loss was not used. This is because in experiments, the number of training mini-batches is generally small, and there are few cases where samples from the same production process are introduced in one training iteration. For our specific MoCo and SimCLR design, primarily a patch-level contrastive loss with a multiscale strategy was introduced. However, Algorithm 1 is not a completely self-supervised learning method. Supervised samples are introduced to improve the ability of the model to distinguish between different microstructures. Considering the dense distribution of microstructures in metallographic images, introducing supervised samples in advance can guide the model in capturing task-related visual representations. This design improves the adaptability of the method to microstructure image analysis. The fine-tuning stage is performed after model pretraining. Specifically, the projection head function used for contrastive learning is removed, and the weights of the encoder and decoder are transferred to the segmentation network. Compared with random initialization, the pretrained model can capture effective visual representations. To verify its effect on metallographic image segmentation, we used the cross-entropy and Dice loss58 based on supervised samples. For simplicity, the loss function is defined as a binary classification, which can be expressed as follows:9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{L}_{sup}=-\frac{1}{HW}{\sum\:}_{i=1}^{HW}{y}_{sup,i}\mathrm{log}{y}_{pred,i}^{{\prime\:}}+(1-\frac{2{\sum\:}_{i=1}^{HW}{y}_{sup,i}\cdot \:{y}_{pred,i}^{{\prime\:}}}{{\sum\:}_{i=1}^{HW}{y}_{sup,i}+{\sum\:}_{i=1}^{HW}{y}_{pred,i}^{{\prime\:}}}),$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{y}_{pred,i}^{{\prime\:}}$$\end{document} is the corresponding output from the network at position \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:i$$\end{document}. Results In this section, the effectiveness of the proposed method is verified. First, a metallographic image dataset and evaluation metrics are introduced. Subsequently, the experiments conducted using different methods are compared using only one annotated image. Finally, ablation studies, parameter sensitivity analyses and analysis of pretraining effectiveness are presented. Dataset In this study, the experimental object was a homogenized aluminum alloy from a production line. A series of preheating-treatment microstructural images with different holding times were obtained using a metallographic microscope. These industrial data are private, but it could be available for non-commercial purposes through our public repository. In total, 198 metallographic images of size 376 × 376 were collected in this study. Among them, there were 66 images with holding temperatures of 15/30 min, 60 with holding temperatures of 45/60 min, and 72 with holding temperatures of 90/120 min. In the experiments, to evaluate the effectiveness of the method, 80% of the images were selected as the training set for pretraining and the remaining 20% constituted the testing set. In the pretraining stage, to obtain different views of the same sample, augmentation methods such as rotation, flipping, cropping, and changing contrast were used to build positive pairs, as shown in Fig. 4. Fig. 4Instances of different views of same metallographic sample. Instances of different views of same metallographic sample. Furthermore, only one image was randomly selected from the training set for annotation. The annotation was manually implemented at the pixel level under the guidance of material experts. The annotated labels included four categories: silicon particles, Fe-containing phase, magnesium-silicon phase, and an aluminum matrix (background). This supervised sample was used for both unlabeled patch sampling in the pretraining stage, and training the model in the fine-tuning stage. This process was repeated five times to more accurately evaluate the effectiveness of the pretraining method with a small number of annotations. Evaluation metrics and experimental settings From the perspective of computer vision, the task of this study is microstructure segmentation. This is a dense pixel-level prediction method. In general, the accuracy and Dice scores, which are key metrics for verifying the effectiveness of segmentation methods59 are expressed as follows:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:ACC=\frac{1}{HW}{\sum\:}_{i=1}^{HW}\mathbb{l}\left({y}_{sup,i}=={y}_{pred,i}^{{\prime\:}}\right),$$\end{document}11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Dice=\frac{2{\sum\:}_{i=1}^{HW}{y}_{sup,i}\cdot \:{y}_{pred,i}^{{\prime\:}}}{{\sum\:}_{i=1}^{HW}{y}_{sup,i}+{\sum\:}_{i=1}^{HW}{y}_{pred,i}^{{\prime\:}}},$$\end{document} From the perspective of computer vision, the task of this study is microstructure segmentation. This is a dense pixel-level prediction method. In general, the accuracy and Dice scores, which are key metrics for verifying the effectiveness of segmentation methods59 are expressed as follows: where function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\mathbb{l}(\cdot \:)$$\end{document} represents the indicator function that returns 1 only when the internal condition is met. Specifically, the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Dice$$\end{document} score in Eq. (11) was calculated for a certain category. There are three categories of foregrounds in metallographic images, and the background is an aluminum matrix. To comprehensively reveal the effectiveness of the model in the segmentation task, the experiments calculated the average \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:Dice$$\end{document} scores of the three foreground microstructures. The pipeline of the proposed method consisted of two stages: pretraining and fine-tuning stages. In the pretraining stage, the batch size and learning rate were six and 1e-4, respectively. The coefficient \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\alpha\:$$\end{document} used to balance the image and patch contrastive loss in (8) was 0.7. The temperature coefficient plays a crucial role in controlling the sensitivity of the similarity measure. In contrastive learning, typically set to a positive number less than 1, and for the current dataset, it is set to 0.5 for both image-level and patch-level contrastive learning. The proposed method was implemented using two existing frameworks. For the MoCo framework implemented in Algorithm 1, the samples \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:T$$\end{document} was 30, and the consistency weight \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\gamma\:$$\end{document} in (3) was 0.5. For the SimCLR framework, the number of samples \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:T$$\end{document} was set to four. In the fine-tuning stage, the Adam60 optimizer was used for the encoder and decoder. The learning rate of the decoder was 1e-3, whereas that of the encoder was 1e-4. Using only one supervised image, the cross-entropy and Dice losses in (9) were used to verify the effectiveness of the proposed method. Microstructure segmentation results In this study, the UNet structure shown in Fig. 2 was used for metallographic image segmentation. As discussed above, a total of 39 (≈ 20%) metallographic images were used to evaluate the effectiveness of the method, and 159 (≈ 80%) images were used for pretraining without annotation. Only one annotated image was used to implement the sampling strategy and supervised learning during the fine-tuning stage. Various self-supervised learning methods were used for comparative experiments. To ensure a comprehensive and equitable evaluation of the methodologies, we performed experiments by randomly selecting annotated images and repeating the procedure five times. The outcomes of these predictions are depicted in Fig. 5. Furthermore, Table 1 details the mean and standard deviation of the model’s performance metrics over the five repeated experiments, employing various supervised samples to thoroughly assess the methods. Fig. 5Microstructure segmentation results of different methods. Microstructure segmentation results of different methods. Table 1The dice scores and accuracies of different methods for microstructure segmentation.Dice (mean)Dice (std)ACC (mean)ACC (std)Baseline0.51340.06090.91670.0373MoCo440.56960.06830.94480.0181SimCLR550.57120.03010.93290.0368BYOL460.54370.07430.93390.0232DenseCL500.56220.03770.93560.0227PSCL-MoCo (proposed)0.62840.03160.95460.0108PSCL-SimCLR (proposed) 0.6296 0.0522 0.9574 0.0086 Table 1The dice scores and accuracies of different methods for microstructure segmentation.Dice (mean)Dice (std)ACC (mean)ACC (std)Baseline0.51340.06090.91670.0373MoCo440.56960.06830.94480.0181SimCLR550.57120.03010.93290.0368BYOL460.54370.07430.93390.0232DenseCL500.56220.03770.93560.0227PSCL-MoCo (proposed)0.62840.03160.95460.0108PSCL-SimCLR (proposed) 0.6296 0.0522 0.9574 0.0086 The dice scores and accuracies of different methods for microstructure segmentation. Figure 5 shows the prediction results and detailed images of the three test images. “Baseline” represents the prediction results of the model under random initialization. “MoCo” refers to the pretraining method for microstructure segmentation and was implemented by the original MoCov2. PSCL-MoCo and PSCL-SimCLR represent the proposed methods implemented using MoCo and SimCLR, respectively. Compared with the pretraining models, the random initialization model was observed not to solve the problem very well. In the detailed images, the benchmark method predicted a large number of meaningless and redundant microstructures. Meanwhile, note that multiple microstructures are incorrectly predicted and mixed together. Considering that only one metallographic image was annotated, the model is likely to overfit. When the pretraining strategy is introduced, the model is committed to extracting similar features of the augmented instances, which effectively alleviates this situation. However, the original MoCo method still suffers from frequent class confusion as effective representation of local features is lacking. In the middle part of the microstructure, the predictions of the MoCo method are incorrect because of the subtle changes in image grayscale. By contrast, the proposed method largely avoids this phenomenon, regardless of whether it is implemented in the MoCo or SimCLR framework. This demonstrates the effectiveness of introducing patch-level contrastive learning into the classical MoCo and SimCLR frameworks for dense prediction. Table 1 provides the quantitative indicators for the different methods. Compared to Fig. 5, additional pretraining methods are featured here for comparison, including SimCLR, bootstrap your own latent (BYOL), and dense contrastive learning (DenseCL). Overall, compared with the benchmark, all self-supervised methods achieved higher scores. Among them, PSCL-SimCLR achieved the highest Dice and ACC scores of 0.6296 and 0.9574, respectively. It exhibited better performance in the metallographic image segmentation task. We observe that the selection of annotated samples and model initialization may have an impact on the performance. Because only one image was employed for the supervised learning, the fluctuation in the baseline method is significant. For the five repeated experiments, the standard deviations of the accuracy and Dice scores are both larger, whereas in our method, this impact is relatively small. This indicates that the proposed method is more robust because of the designed contrastive learning. Overall, our method exhibited better performance in the metallographic image segmentation task. Ablation study and parameter sensitivity analysis Compared with most existing self-supervised learning methods based on image-level contrastive loss, our method uses patch-level contrastive loss, a multiscale strategy, and patch sampling methods. The introduction of these strategies enables the method to achieve better results for the microstructural segmentation problem. An ablation study was conducted on these strategies. The quantitative results are presented in Table 2. Table 2Ablation study on the proposed method with respect to image-level contrastive learning, patch-level contrastive learning, and patch sampling strategies.MoCo/SimCLRGlobalLocalSampleDiceACCPSCL-MoCo√-0.57640.9451PSCL-MoCo√√0.61310.9475PSCL-MoCo√√0.50100.9197PSCL-SimCLR√-0.56470.9402PSCL-SimCLR√√0.59220.9450PSCL-SimCLR√√0.55840.9494 Table 2Ablation study on the proposed method with respect to image-level contrastive learning, patch-level contrastive learning, and patch sampling strategies.MoCo/SimCLRGlobalLocalSampleDiceACCPSCL-MoCo√-0.57640.9451PSCL-MoCo√√0.61310.9475PSCL-MoCo√√0.50100.9197PSCL-SimCLR√-0.56470.9402PSCL-SimCLR√√0.59220.9450PSCL-SimCLR√√0.55840.9494 Ablation study on the proposed method with respect to image-level contrastive learning, patch-level contrastive learning, and patch sampling strategies. Table 2 presents the results of the proposed method, implemented separately based on the two frameworks. The notations “Global,” “Local,” and “Sample” represent image-level contrastive loss, patch-level contrastive loss, and sampling strategy, respectively. The checkmarks indicate the options selected for each strategy. In Table 2, the overall trends of the methods implemented using the two frameworks are consistent. Specifically, the Dice scores of PSCL-MoCo and PSCL-SIM with global contrastive loss are 0.5764 and 0.5647, respectively. Compared with the full version of the method, the scores are lower by 0.0520 and 0.0649, respectively. The same phenomenon also holds for the case with only patch-level contrastive loss, where the scores are lower by approximately 0.0153 and 0.0374, respectively. Notably, the performance degradation of the method is significant in the absence of a sampling strategy. For PSCL-SimCLR, the Dice score without a sampling strategy is 0.5584. For PSCL-MoCo, this indicator drops to 0.5010, which is even lower than the 0.5134 for random initialization in Table 1. The introduction of sampling strategy provides discriminative patches for contrastive learning. This is also a significant difference between the proposed method and existing self-supervised pretraining methods. On the other hand, a sensitivity analysis was conducted on the balance coefficient \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\alpha\:$$\end{document} in (8), which is used to control the weighting between the two loss functions. The coefficient was sampled at intervals of 0.1, and the experiment was repeated five times to obtain its statistical characteristics. To provide a clearer visualization of the results, a box plot is presented, as shown in Fig. 6. Fig. 6Parameter sensitivity experiment results for balance coefficient. Parameter sensitivity experiment results for balance coefficient. In Fig. 6, the overall trend is relatively flat, indicating that performance will not undergo drastic changes with the variation of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\alpha\:$$\end{document}. But when its value is particularly small or very large, the performances of the method deteriorate slightly. Its peak appears around 0.7, regardless of whether it is PSCL-MoCo or PSCL-SimCLR. Meanwhile, its statistical variance is relatively small, which proves its robustness. It suggests that patch-level contrastive loss plays a more critical role in our segmentation task. In summary, both the image-level and patch-level loss functions are meaningful and complementary for the dense prediction of microstructures. The image-level loss excels at capturing global representations and processing consistent information, while the patch-level loss enhances the model’s ability to perceive fine details and spatial locations. Together, they contribute to a more robust and accurate segmentation model. Moreover, when implementing patch-level contrastive loss, a multiscale strategy was adopted. Considering the characteristics of the UNET structure, contrastive learning was performed after each up-sampling layer in the decoder. An ablation study was conducted to verify its effectiveness. The respective performance indicators for the proposed PSCL-MoCo and PSCL-SimCLR are listed in Tables 3 and 4. Table 3Experimental results based on contrastive learning at different scales for PSCL-MoCo.MoCo/SimCLRScale-1Scale-2Scale-3DiceACCPSCL-MoCo√0.58780.9552PSCL-MoCo√0.61350.9550PSCL-MoCo√0.60300.9512PSCL-MoCo√√0.6143 0.9556 PSCL-MoCo√√0.59350.9484PSCL-MoCo√√0.60370.9483PSCL-MoCo(Set)√√√ 0.6284 0.9546 Table 3Experimental results based on contrastive learning at different scales for PSCL-MoCo.MoCo/SimCLRScale-1Scale-2Scale-3DiceACCPSCL-MoCo√0.58780.9552PSCL-MoCo√0.61350.9550PSCL-MoCo√0.60300.9512PSCL-MoCo√√0.6143 0.9556 PSCL-MoCo√√0.59350.9484PSCL-MoCo√√0.60370.9483PSCL-MoCo(Set)√√√ 0.6284 0.9546 Experimental results based on contrastive learning at different scales for PSCL-MoCo. Table 4Experimental results based on contrastive learning at different scales for PSCL-SimCLR.MoCo/SimCLRScale-1Scale-2Scale-3DiceACCPSCL-SimCLR√0.60030.9547PSCL-SimCLR√0.58370.9507PSCL-SimCLR√0.58520.9430PSCL-SimCLR√√0.60570.9498PSCL-SimCLR√√0.61290.9540PSCL-SimCLR√√0.58930.9501PSCL-SimCLR(Set)√√√ 0.6296 0.9574 Table 4Experimental results based on contrastive learning at different scales for PSCL-SimCLR.MoCo/SimCLRScale-1Scale-2Scale-3DiceACCPSCL-SimCLR√0.60030.9547PSCL-SimCLR√0.58370.9507PSCL-SimCLR√0.58520.9430PSCL-SimCLR√√0.60570.9498PSCL-SimCLR√√0.61290.9540PSCL-SimCLR√√0.58930.9501PSCL-SimCLR(Set)√√√ 0.6296 0.9574 Experimental results based on contrastive learning at different scales for PSCL-SimCLR. In Tables 3 and 4, the number following “Scale” represents the upsampling stage. For example, the feature map size of “Scale-1” is the smallest, which refers to the first upsampling, and “Scale-3” represents the last upsampling, with the feature map restored to its original size. Intuitively, this effect diminishes when only one scale is effective. For PSCL-MOCO, the scores for the first three scales are 0.5878, 0.6135, and 0.6030. Among these, using only Scale-2 or Scale-3 provides slightly better results than using Scale-1. It demonstrates the importance of local details in microstructure segmentation methods. This trend is the same for the methods implemented based on SimCLR. The Dice score and accuracy of the method were significantly improved. Such a strategy is necessary for the dense prediction problem implemented by UNet. Through upsampling in the decoder, contrastive learning helps the model gradually capture effective visual representations. Through the design of a multiscale strategy that enhances the segmentation sensitivity to microstructures of different sizes, the method achieved better performance. Analysis of pretraining effectiveness In contrast to semi- and weakly supervised learning, which generate pseudo-annotations or introduce corresponding strategies during training, the proposed method captures effective visual features through pretraining. In the previous experiment, the effectiveness of the method was evaluated using only one supervised image. In practice, the number of annotated images may be larger. In this subsection, the performance of the model is verified for different numbers of annotated samples. Experiments were conducted under three settings: using only one annotated image (previous setting), using five annotated images, and fully supervised learning. Specifically, for the first two cases, one and five different annotated images were randomly selected, and the experiment was repeated five times to obtain the mean and standard deviation of their performance. For supervised learning, a more comprehensive evaluation was performed using five-fold cross-validation. The prediction results and performance metrics are presented in Fig. 7; Table 5, respectively. Fig. 7Microstructure prediction results under different numbers of supervised samples. Microstructure prediction results under different numbers of supervised samples. Table 5Quantitative indicators of the method under different numbers of supervised samples.Num1Num5Supervised learningBaseline0.5134 (0.0609)0.7542 (0.0174)0.8153 (0.0047)PSCL-MoCo0.6284 (0.0316)0.7707 (0.0159)0.8374 (0.0093)PSCL-SimCLR0.6296 (0.0522)0.7787 (0.0109)0.8391 (0.0060) Table 5Quantitative indicators of the method under different numbers of supervised samples.Num1Num5Supervised learningBaseline0.5134 (0.0609)0.7542 (0.0174)0.8153 (0.0047)PSCL-MoCo0.6284 (0.0316)0.7707 (0.0159)0.8374 (0.0093)PSCL-SimCLR0.6296 (0.0522)0.7787 (0.0109)0.8391 (0.0060) Quantitative indicators of the method under different numbers of supervised samples. In Table 5; Fig. 7, “Baseline” refers to the model with randomly initialized parameters. The values in parentheses in Table 5 represent the standard deviation under the three supervised scenarios. Overall, as the number of supervised samples increased, the prediction results and performance metrics improved. For PSCL-MoCo, the metrics in the three cases are 0.6284, 0.7707, and 0.8374, respectively. For PSCL-SimCLR, the metrics are 0.6296, 0.7787, and 0.8391, respectively. As for the standard deviation, it shows a significant downward trend with the increase of supervised quantity. It indicates that the representativeness of annotated samples is crucial. This trend aligns with the general principles of deep learning, where pretrained models benefit from more supervised samples. The prediction results shown in Fig. 7 further validate this point, as the results under fully supervised conditions are closer to the ground truth. This demonstrates that the pretraining method effectively captures visual representations relevant to the actual task and performs well across multiple scenarios. Discussion In this study, a novel contrastive learning method called patch-sampled contrastive learning (PSCL) is proposed for metallographic image segmentation tasks. Our method constructs two contrastive learning frameworks: image-level and patch-level. Both use a feature-similarity-based sampling strategy. Conventional self-supervised contrastive learning exhibits inherent limitations in metallographic image analysis due to the densely distributed and visually similar microstructures (e.g., Si particle vs. Fe-containing phase shown in Fig. 1). Randomly sampled patch pairs might belong to the same phase category, leading to ineffective learning of discriminative features. Our supervised-assisted sampling strategy addresses this fundamental challenge by leveraging labeled patches as anchors to ensure meaningful negative pairs that span distinct phase categories while filtering out semantically ambiguous comparisons through feature-space similarity measurement (Eq. 6). This approach specifically focuses the contrastive learning on the most discriminative local characteristics, which is particularly crucial for microstructure analysis. The improvement in Dice coefficient compared to random sampling (Table 1) demonstrates the essential role of this design in analyzing densely packed microstructures. The following is a more detailed analysis and discussion. First, for the microstructure segmentation results, compared with the existing models, the proposed method achieves higher scores. As shown in Table 1, PSCL-SimCLR achieved the highest Dice and ACC scores of 0.6296 and 0.9574, respectively. The prediction results shown in Fig. 5 more intuitively reveal this point. PSCL-MoCo and PSCL-SimCLR predictions align closely with expert annotations. This includes small or easily confused microstructures. Introducing multiscale patch-level contrastive learning enhances the capture of local information, leading to better prediction results. Secondly, we conduct an ablation study on the designed patch-level contrastive loss, the multiscale strategy, and patch sampling in our method, as shown in Tables 2 and 3, and 4. Among these components, the patch sampling strategy has the most significant impact. For PSCL-SimCLR, the Dice score without a sampling strategy is 0.5584. For PSCL-MoCo, this metric drops to 0.5010, which is even lower than the 0.5134 achieved by random initialization in Table 1. This is because, in metallographic images, microstructures are densely distributed across the alloy matrix. The metallographic images studied in this work differ significantly from natural images. The high similarity between images limits the performance of MoCo and SimCLR based solely on image-level contrastive learning. An intuitive solution is to construct contrastive learning using patches, like DenseCL. However, this approach does not entirely resolve the issue. Patches are smaller than full images. However, semantically similar negative samples still occur. Different patches may contain the same particles, making it challenging for traditional patch-based contrastive learning methods to capture the semantic information of microstructures effectively. By adopting a sampling strategy designed with supervised information, the model can learn meaningful visual representations and discriminative deep features during the pretraining stage. As a result, the segmentation performance of this method is significantly improved. Then, the performance of the model is verified for different numbers of annotated samples as shown in Fig. 7; Table 5. In general, regardless of the number of annotations, the methods with contrastive learning pretraining always performed better than the baseline. In the detailed images shown in the third row of Fig. 7, for fully supervised learning, the pretrained models predict more coherent regions for indistinct microstructures compared with baseline. For the quantitative analysis presented in Table 5, when the number of supervised images is five, the two proposed methods, PSCL-MoCo and PSCL-SimCLR, outperform the baseline method by 0.0156 and 0.0245, respectively. When the number of supervised images is 159, these two indices ​​become 0.0221 and 0.0238, respectively. This shows that even under fully supervised learning, the proposed pretraining method is still meaningful and effective. Finaly, to better illustrate this point, we used the K-means61 method to directly perform unsupervised clustering analysis of the pretrained model, and the results are shown in Fig. 8. The cluster analysis presented in Fig. 8 was conducted before the model was fine-tuned. The definition of the terms “Scale-1,” “Scale-2,” and “Scale-3” are the same as in Tables 3 and 4, which represents the visualization results of feature maps with different scales. The number of clusters was set to four to correspond to the actual number of categories. Overall, even without fine-tuning, the pretrained model still captures the task-related visual representation. Through image- and patch-level comparative learning, the foreground, background, and corresponding contours can be effectively distinguished. Meanwhile, with an increase in the feature map size, the clustering results of Scale-2 and Scale-3 show more details than those of Scale-1. Using this sampling strategy, clustering results close to the actual segmentation ground truth are obtained. This reveals the potential reason why the method can achieve better performance in the fine-tuning stage. Fig. 8Clustering visualization results of patch features of different scales after pretraining. Clustering visualization results of patch features of different scales after pretraining. The proposed pretraining method for microstructure segmentation tasks is based on contrastive learning, which significantly differs from semi-supervised learning and transfer learning approaches. First, unlike semi-supervised learning, the proposed method is a pretraining approach rather than directly training the model using labeled samples. Semi-supervised learning has two key limitations. It depends heavily on initialization parameters and requires high-quality supervised samples. This study considers the scenario where only one annotated image is available, making it more akin to a few-shot learning problem. If the supervised samples lack sufficient representativeness or the model initialization is poor, the generated pseudo-labels or constraints may fail, leading to the collapse of the semi-supervised strategy. Furthermore, there are significant differences between our pretraining methods and transfer learning. The latter is based on the fundamental assumption that there is a meaningful correlation between the source domain data and the target domain data. However, a significant discrepancy exists between the latent distribution of natural image datasets (such as Pascal or MOCO) as the source domain and the target data (metallographic images). In our view, neither the “pretraining on natural images followed by fine-tuning on metallographic images” paradigm nor the domain adaptation paradigm is fully suitable for the current problem. Additionally, a practical challenge is that, compared to natural image datasets, publicly available metallographic image datasets are extremely scarce, and their data volume is insufficient due to the limited availability of industrial data. Therefore, our solution is to perform pretraining directly on the metallographic image dataset. By leveraging a small amount of supervised annotations as guidance, the method aims to achieve close intra-class distances and distant inter-class distances. Unlike semi-supervised learning, it does not generate potentially unreliable pseudo-labels, nor is it a conventional unsupervised contrastive learning method. Instead, it focuses on pretraining to capture discriminative visual features. This is the most distinctive aspect of our method. Additionally, our pretraining approach remains valuable even in supervised learning scenarios. Even when all training images have corresponding annotations, our proposed pretraining can still enhance the method’s performance, further highlighting the fundamental differences between our approach and semi-supervised methods. Conclusions In this study, we proposed a pretraining method for microstructure segmentation task in metallographic image based on contrastive learning. To segment the microstructure, image- and patch-level contrastive learning methods were designed to help the model capture global and local features. Considering that the microstructure is densely distributed in the matrix, a sampling method based on feature similarity was also introduced to enhance the model’s ability to distinguish different types of microstructures. Through the specific design of metallographic images, the model effectively captured visual representations during the pretraining stage. In the experiments, compared with other existing methods, the proposed method displayed better computer vision performance indicators when only one image was annotated, thereby reducing manual annotation costs. Even in the fully supervised case, model pretraining is meaningful, which indicates the potential of the method in practical applications. However, some microstructures may have extremely small sizes, necessitating the use of smaller patches to capture their visual representations. This, in turn, increases computational power consumption. Additionally, processing noisy images should be considered to improve the method’s detection capability across diverse scenarios. For these reasons, future work in this area will focus on addressing extreme cases and noise-related challenges. Furthermore, developing effective model pretraining methods for other types of metal images and domains is also a promising direction for further research. To this end, broadening the applicability of pretraining methods and enhancing their practical deployment in extreme and complex production processes will constitute a pivotal direction of our future works.