A novel adjunctive diagnostic method for bone cancer: Osteosarcoma cell segmentation based on Twin Swin Transformer with multi-scale feature fusion Authors: Tingxi Wen, Binbin Tong, Yuqing Fu, Yunfeng Li, Mengde Ling, Xinwen Chen Published: 2024 ABSTRACT Highlights•Proposed a Twin Swin Transformer based on Swin Transformer, achieving efficient osteosarcoma cell segmentation with multi-scale feature fusion.•Replaced the original Swin block with the Twin Swin Transformer block to enhance feature interactions across stages.•Added channel attention mechanism, improving segmentation accuracy.•Extracted detailed morphological and spatial information, aiding in personalized treatment strategies. Introduction Osteosarcoma is one of the most common malignant primary bone tumours. It originates from the malignant proliferation of primitive mesenchymal cells and is predominantly found in adolescents, with a second peak incidence in older adults [1], [2]. The late-stage survival rate of patients with osteosarcoma is only 20 % [3]. Currently, the leading medical images used to diagnose osteosarcoma are electronic computed tomography (CT) images, X-ray images, and MRI [4]. Among them, MRI images have become the primary tool for physicians to diagnose osteosarcoma due to their good representativeness of soft tissue components such as tumours, muscles, and blood vessels and safety. In addition, the current diagnosis of osteosarcoma patients mainly relies on manual identification by physicians. This manual diagnostic method has the following urgent problems: The diagnosis of osteosarcoma disease requires extremely high experience and professional knowledge of doctors. At the same time, it is difficult for physicians to efficiently complete the diagnosis of patients due to the current lack of well-developed osteosarcoma-assisted segmentation technology in hospitals [5]. Therefore, how to assist physicians in efficiently diagnosing osteosarcoma disease is a challenge that needs to be urgently solved. Although approximately 600 to 700 MRI images can be acquired from each osteosarcoma patient, only 10 to 20 of these MRI images can be used to diagnose osteosarcoma disease [6], and the process of manual screening by physicians is very inefficient. Developing countries are relatively economically backward, the problem of few medical resources and lack of medical personnel. In countries, there are many deficiencies in the stages of diagnosis and treatment of osteosarcoma. Tight medical resources and insufficient funding [7], [8], coupled with the fact that medical equipment used to diagnose osteosarcoma is very expensive, means that most hospitals in these developing countries cannot afford the cost of the instruments [9]. It leads to increased pressure to diagnose osteosarcoma disease. Diagnostic MRI images present the following disadvantages: MRI scans usually take a long time to acquire high-quality images; the MRI equipment itself is costly and requires specialized technicians to operate and maintain it, which increases operating costs; certain patients may not be able to undergo MRI scans, such as those with pacemakers and metal implants; MRI images are susceptible to patient movement, and even slight movement may result in blurring or artifacts, which may affect the accuracy of the diagnosis [10], [11]. As cell segmentation techniques and deep learning algorithms continue to advance, precise instance segmentation of osteosarcoma cells can effectively assist physicians in making accurate and efficient diagnoses, thereby providing crucial support for treatment planning. Cell segmentation is a critical task in computer vision, aiming at precise boundary localization and segmentation of cells in microscope images. With the development of deep learning, the processing of cell microscopy images has been greatly enriched in recent years. By segmenting osteosarcoma cell images, we can better understand their morphological features, cellular structure, and potential growth and metastasis mechanisms, providing an important reference basis for clinical diagnosis and treatment. With the advancement and maturity of deep learning technology and convolutional neural networks, they play an increasingly important role in image segmentation, especially in medical images. Currently, medical image segmentation processing methods are roughly divided into two categories: the first category can be classified as traditional methods, including region-based methods, threshold-based methods, edge-based methods and clustering-based methods; the second category of methods can be summarized as methods using convolutional neural networks, which carry out feature extraction by convolution and then segmentation. The above methods only perform semantic segmentation of the image, while instance segmentation is a combination of target detection and semantic segmentation, which can both segment to get the edges of the objects and label the different individuals in the same kind of objects in the image. Related work The evolution of high-precision manufacturing systems and diagnostic imaging has significantly influenced quality improvement in engineering and medical fields. In manufacturing, Du et al. [12] introduced a Markov-based model to manage product sequence and bottlenecks, emphasizing the importance of handling sequential dependencies across stages to maintain quality. This concept mirrors the importance of upstream data preparation in osteosarcoma segmentation, where the accuracy of initial data processing can directly impact downstream segmentation results. In a complementary effort, Wang et al. [13] incorporated variable stiffness structures (VSS) into machining models using a state-space framework. Their work on modeling elastic deformations aligns with the segmentation challenges encountered in medical imaging, where accurately capturing complex biological structures—such as osteosarcoma tumors—requires nuanced modeling. Similarly, in our segmentation approach, we employ multi-scale feature fusion to capture intricate variations within osteosarcoma images, analogous to the adaptive strategies used in VSS machining systems. The insights from high-definition metrology (HDM) techniques further inform our work. Li et al. [14] used HDM to enhance surface texture evaluation, analogous to how we employ high-resolution fluorescence microscopy to achieve precise cell segmentation. Zhao et al. [15] leveraged dynamic modeling to optimize face milling parameters, enhancing process stability. Shao et al. [16], [17] extended the application of HDM by developing models for predicting leakage channels in static sealing interfaces, focusing on spatial irregularities. This challenge resonates with the difficulties of accurately segmenting irregular tumor boundaries in osteosarcoma detection. Our work builds on these insights by introducing advanced segmentation techniques to ensure that irregular cell structures are reliably delineated for diagnostic accuracy. In the context of osteosarcoma segmentation specifically, Anisuzzaman et al. [18] reviewed the state of deep learning for osteosarcoma detection, identifying key challenges and opportunities. Kayal et al. [19] evaluated nine algorithms, including traditional methods like Otsu thresholding and machine learning approaches such as logistic regression and support vector machines, to assess their effectiveness on osteosarcoma datasets. Their study underscores the need for more advanced models to overcome the limitations of traditional algorithms. Loraksa et al. [20] found that image format impacts segmentation accuracy, with PNG images yielding optimal results across multiple CNN models. Zhang et al. [21] proposed a multiple supervised residual network for CT segmentation, demonstrating that deeper hierarchical features improve segmentation performance, outperforming traditional architectures like U-Net. Building on these works, Shuai et al. [22] introduced the W-net++, a dual U-Net model that addresses spatial detail loss through multi-scale inputs, achieving superior segmentation of osteosarcoma lesions. Huang et al. [23] further advanced segmentation techniques with the MSFCN, an end-to-end network using edge supervision and multi-scale feature learning to enhance F1-scores in CT image analysis. Jia et al. [4] focused on MRI segmentation with DecoupleSegNet, a lightweight model leveraging flow field learning to enhance pixel-level consistency, achieving 90.51 % IoU with fewer parameters. Their work demonstrates that optimized networks can maintain high accuracy while minimizing computational costs, aligning with our objective to design an efficient and adaptive segmentation framework. Inspired by the multi-stage processing principles found in manufacturing [12], [14], [15], we develop a Twin Swin Transformer network to address the challenges of under-segmentation in osteosarcoma detection. This model integrates squeeze-and-excitation networks (SENet) [24], enabling adaptive channel learning and improving feature interactions. Overall network structure Mask R-CNN based on Swin Transformer backbone network The network structure of Mask R-CNN [25] based on Swin Transformer backbone network is shown in Fig. 1.First, the input zipper images are fed into the Swin Transformer backbone network for feature extraction, obtaining multi-scale feature maps at different levels. Then, these feature maps are fused with the feature maps obtained from upsampling on the feature pyramid to generate the output feature maps of the feature pyramid structure. Next, centering on each point of the feature map, candidate frames of different scales and sizes are generated by the Region Proposal Network (RPN), and these candidate frames are scored and filtered to select the regions of interest with scores above a threshold. Meanwhile, candidate frames with high overlap are removed using Non-Maximum Suppression (NMS). Subsequently, Region of Interest Align (RoI Align) to obtain feature maps of consistent size. Finally, these aligned feature maps are fed separately into the classification and regression branches and the mask branch, where they are processed through multiple convolutional networks to generate the final category predictions, bounding box regression results, and masks, achieving zipper identification and segmentation.Fig. 1Network structure diagram of Mask R-CNN model based on Swin Transformer backbone network. Network structure diagram of Mask R-CNN model based on Swin Transformer backbone network. The loss function of Mask R-CNN consists of three primary parts: classification loss, bounding box regression loss, and mask loss. The total loss is calculated as follows:(1)L=Lcls+Lbox+Lmask The classification loss is used to evaluate the accuracy of the model's category predictions for the candidate regions. The Cross-Entropy Loss function is utilized and can be expressed using the following formula:(2)Lcls=-1Ncls∑i=1Nlog[pi∗pi+(1-pi∗)(1-pi)]where pi∗ is the probability that the predicted candidate box is a foreground, and pi is the probability of the predicted candidate box. The bounding box regression loss is used to evaluate the accuracy of the model's predictions for the position and size of the candidate region's bounding box. The Smooth L1 loss function is employed, and the formula is as follows:(3)Lbox=∑i∈{x,y,w,h}smoothL1(ti-ti∗)(4)smoothL1(x)=0.5x2ifx<1x-0.5otherwisewhere ti and ti∗ represent the true and predicted bounding box parameters (including the center coordinates and dimensions), respectively, and smoothL1(x) is the Smooth L1 loss function. The mask loss is used to evaluate the accuracy of the model's predictions for the target masks. A per-pixel binary cross-entropy loss function is employed, and the formula is as follows:(5)Lmask=-1m2∑i,jyi,jlog(yi,j∗)+(1-yi,j)log(1-yi,j∗)where yi,j is the value of pixel (i,j) in the ground truth mask, yi,j∗ is the value of pixel (i,j) in the predicted mask, and m is the width or height of the mask, m2 indicates the total number of pixels in the mask. Swin Transformer The Swin Transformer efficiently models both local and global features by dividing the input image into fixed-size patches and applying a hierarchical structure and sliding window mechanism. Within each layer, window-based self-attention is performed, progressively reducing spatial dimensions while increasing the number of feature channels. This approach is highly effective for a wide range of computer vision tasks. As shown in Fig. 2. First, the input image is divided into non-overlapping chunks of fixed size, each of which is spread and projected as a feature vector of fixed length. This converts the 2D image into a series of 1D vectors for subsequent Transformer coding. Next, these feature vectors are mapped to a higher dimensional feature space through a linear embedding layer, a step similar to the convolutional layer operation in traditional convolutional neural networks for increasing the dimensionality of the feature representation.Then, feature extraction is performed through multiple Swin Transformer blocks, while downsampling is performed between each stage through the Patch Merging module, which combines small adjacent blocks into larger blocks, reduces the size of the feature map, and increases the number of feature channels through linear layers, which is similar to the pooling operation in convolutional neural networks. After multi-layer Swin Transformer block and multiple stages of processing, the final feature representation with rich hierarchical information is obtained, and these features can be used for various downstream tasks.Fig. 2Swin Transformer network structure diagram. Swin Transformer network structure diagram. The core module of Swin Transformer is the Swin Transformer Block, shown in Fig. 3, which consists of the Multi-head Self-Attention (MSA), the Multilayer Perceptron (MLP), and the Normalization Layer. Its key innovation is the sliding window mechanism.Fig. 3Swin Transformer Block structure diagram. Swin Transformer Block structure diagram. Its network structure can be roughly divided into two: Window Multi-head Self Attention (W-MSA) network and Shifted Window Multi-head Self-Attention (SW-MSA) network. The specific process of the W-MSA network is as follows: firstly, the input feature map is divided into multiple windows, and multi-head self-attention computation is carried out within each window, which is then processed by hopping connection, normalization, and multilayer perceptual machine. The difference between the SW-MS network and the W-MSA network is that the divided windows are slid to form new windows before performing the MSA computation, and then the MSA computation is performed within these slid windows. The window long self-attention is calculated as:(6)Attention(Q,K,V) = softmax(QKT^/d+B)V(7)MultiHead(Q,K,V) = Concat(head1,...,headh)WO In the above equation, the projection parameter matrix satisfies the condition WO is the linear transformation matrix of the output and headi denotes the output of the ith attention head.; B represents the relative positional offset; Q, K, and V are the features within the window mapped into the query (Q), key (K), and value (V) matrices by three linear transformations, respectively. The first layer of Swin Transformer is a linear embedding layer, which first divides the input image into non-overlapping chunks of fixed size and flattens each chunk into a one-dimensional vector. A linear transformation is applied to each of the flattened chunk vectors to map them to a fixed-length feature vector. The linear transformation is usually implemented through a fully connected layer. A series of feature vectors are generated after all the chunks have been linearly transformed. These feature vectors are rearranged into a feature map. In addition to the Swin Transformer Block, there is another important operation in Swin Transformer: patch merging. As shown in Fig. 4, it gradually reduces the spatial dimension between different levels and increases the number of feature channels to achieve hierarchical feature extraction. This process is similar to the pooling operation in convolutional neural networks, which can effectively reduce the computational complexity while retaining important feature information.Fig. 4Patch merging operation. Patch merging operation. Module design of Twin Swin Transformer with multiscale feature fusion The original Swin Transformer [26] performs self-attention computation within each window, reducing computational complexity but resulting in weak feature interactions between different windows and failing to capture global contextual information adequately. Furthermore, Swin Transformer is effective within local windows but has limitations in modeling complex non-linear relationships, especially when dealing with highly complex image features. Meanwhile, the Swin Transformer may be overfitted and undergeneralised on specific tasks or datasets. The Twin Swin Transformer Block is able better to capture the complex non-linear relationships between input features, enabling the model to more accurately understand and process complex visual information, and improve the overall recognition and classification capabilities. By enhancing feature interactivity and capturing complex non-linear relationships, the Twin Swin Transformer Block helps to improve the model's generalization ability, enabling it to perform well on different datasets and tasks. In addition, the Swin Transformer may lose some detail information during multi-level feature extraction, especially in deep networks where the detail information may be gradually weakened. To solve this problem, special Skip Connections are designed in this paper to enable the model to better retain and capture detail information, and thus perform better when dealing with high-resolution images and fine-grained features. Module for Twin Swin Transformer based on Multi-scale feature fusion (MF Twin Swin) is shown in Fig. 5. Firstly, the original Swin Transformer Block is replaced by the twin Swin Transformer Block designed in this paper, and then a special skip connection and feature fusion module is set between each stage.Fig. 5Twin Swin Transformer module based on multi-scale fusion. Twin Swin Transformer module based on multi-scale fusion. In this paper, the twin Swin Transformer Block is designed based on the original Swin Transformer Block, as shown in Fig. 6. It mainly consists of two parallel Swin Transformer Blocks, so it is called Twin Swin Transformer Block.The specific network structure is as follows:Fig. 6Twin Swin Transformer module. Twin Swin Transformer module. The input feature vector of size (H × W) × C is divided into two feature vectors of the same size (H × W) × C/2 by equally spaced sampling, and these two feature vectors are input into two parallel Swin Transformer Blocks respectively, which output two feature vectors of size (H × W) × C/2. The two feature vectors are overlapped and merged into the (H × W) × C feature vector, and then the feature vectors are rearranged to obtain the H × W × C feature map. Next, further feature processing is performed on the feature map using a convolution kernel size of 1 × 1 to finally obtain an H × W × C feature map. To facilitate processing in the next module, the feature map is rearranged into (H × W) × C feature vectors and output. The Twin Swin Transformer module is calculated as follows: Suppose the input feature tensor is X ∈ RH×W×C, This tensor is split into two sub-tensors in each Transformer Block:(8)X1,X2=Split(X),X1,X2∈RH×W×C/2 Each sub-tensor is computed by a separate Transformer:(9)Y1=SwinTransformer(X1),Y2=SwinTransformer(X2) Finally, the output features are passed through the feature fusion operation:(10)Y=Concat(Y1,Y2)andY=Conv1×1(Y) The overall network flow is: The network structure can be divided into four stages and a total of (2, 2, 6, 2) 12-layer network is used. In stage 1, an image dimension of H × W × 3 is input into the model, which is processed and computed by linear embedding layer to obtain a feature vector of (H/4 × H/4) × 96 in two dimensions, which is inputted into the two-layer twin Swin Transformer Block for feature extraction, and outputs a feature vector of the same size, which is then rearranged to form a feature map of H/4 × H/ 4 × 96 feature map as the output of the first stage of the feature pyramid. In the second stage, the feature map is converted into a feature map of H/8 × H/8 × 192 by the Block Merge Module, which is feature fused with the feature vector output from the first stage linear embedding layer. This is done by rearranging the feature vectors output from the linear embedding layer of the first stage into a feature map of H/4 × H/4 × 96, which is sequentially converted into a feature map of H/8 × H/8 × 192 by a convolutional layer with a convolutional kernel size of 3 × 3 and a SENet attention network, which is connected with the feature map output from the block merging module of the second stage in a top and bottom stacking, and is combined into a H/8 × H/8 × 384 feature map, which in turn is converted into a H/8 × H/8 × 192 feature map by a convolution kernel of size 1 × 1, which is rearranged into a (H/8 × H/8) × 192 feature vector input into the two-layer twinned Swin Transformer Block in the second stage to obtain a (H/8 × H/8) × 192 feature vectors, which are then rearranged into H/8 × H/8 × 192 feature maps as the feature map output of the second stage of the feature pyramid. The network structure of the third and fourth stages is similar to that of the second stage. Specifically, the feature map of the second stage is converted into a feature map of H/16 × H/16 × 384 by the block merging module of the third stage, which is feature fused with the feature vectors output from the linear embedding layer of the second stage. The feature vector after performing feature fusion is input to the six-layer twin Swin Transformer Block in the third stage ultimately obtains the feature map of H/16 × H/16 × 384 by rearranging it, which is output as the feature map of the third stage of the feature pyramid. The feature map of the third stage is converted into the H/32 × H/32 × 768 feature map through the block merging module of the fourth stage, which is feature fused with the feature vector output from the linear embedding layer of the third stage. The feature vectors after performing feature fusion are input to the two-layer twin Swin Transformer Block in the fourth stage to finally get the H/32 × H/32 × 768 feature map by rearranging them, which is used as the feature map output of the fourth stage of the feature pyramid. Results and discussion In order to validate the segmentation effectiveness of the MF Twin Swin on osteosarcoma images, we compare it with three excellent and commonly used feature extraction networks: the ResNet50 [27], the GCNet [28], and the Swin Transformer [27]. ResNet50 improves the training efficiency of deep networks through residual connections, excelling at extracting local features from images. GCNet introduces a global context module, leveraging attention mechanisms to capture long-range dependencies and enhance global information understanding. Swin Transformer combines local windows with shifted window strategies to efficiently balance local and global feature extraction. This section firstly describes the dataset of osteosarcoma images used for experiments and segmentation performance evaluation method. Subsequently, the findings of the experiments are described. Finally, qualitative and quantitative analyses of the osteosarcoma segmentation results obtained by different methods are done. Training settings The training hardware configuration for this paper is: CPU model is Intel(R) Xeon(R) W-2155 with 16G of RAM; operating system Ubuntu 22.04.3 LT; GPU model is NVIDIA GeForce RTX 2080 Ti. The models in this paper were trained in the MMDetection framework, Python 3.9.17, CUDA 11.7; CuDNN 8.4, with 80 rounds of training, batch_size of 2, and no pre-training weights loaded. MMDetection [29] is an open source, PyTorch-based target detection toolkit developed by the OpenMMLab team. It provides a large number of pre-trained models, a flexible configuration system, and rich data enhancement features, and is widely used in academic research and industrial applications. MMDetection supports a variety of target detection methods, including Faster R-CNN, Mask R-CNN, RetinaNet, etc., with high scalability and ease of use. With the MMDetection framework, model training, evaluation, and inference can be easily performed. Its modular design allows users to freely combine model components according to their needs, and supports distributed training and multiple optimisation algorithms, making efficient training on large-scale datasets possible. In the experiments in this paper, we conducted model training and evaluation using the flexible configuration system and efficient training process provided by MMDetection to verify the effectiveness of the proposed method. Therefore, MMDetection is adopted as the basic framework for training in this paper. Data sets and evaluation indicators The data set used for the experiments in this paper is BBBC039 [30]. This image set is a high-throughput chemical screen for the human osteosarcoma U2OS cell line. The image set consists of 200 osteosarcoma cell images, each of which is 520 × 696 pixels in size. In this experiment, 140 images of this image set are classified as the training set and the remaining 60 images are classified as the test set. In order to quantitatively evaluate the recognition performance of the twin Swin Transformer on the osteosarcoma cell dataset, the mean Average Precision (mAP) is used in this paper. The mAP is a standard metric for evaluating target detection and instance segmentation models, which is widely used in academic research and industry. mAP is the average of the APs under multiple IoU thresholds, and a high mAP value usually indicates that the model has better detection accuracy and stability, which is an important measure of the model's merit. mAP [31] is specifically calculated as follows. Firstly for each prediction frame and real frame, IoU is calculated. IoU is a measure of the similarity between the prediction results and real labels. The calculation formula is as follows:(11)IoU=IntersectionAreaUnionAreaWhere Intersection Area denotes the area of the overlapping part of the prediction frame and the real frame, and Union Area denotes the total area of the prediction frame and the real frame, minus the area of the intersection area. The value of IoU ranges from 0 to 1. Afterwards, True Positive (TP) and False Positive (FP) are determined based on the value of IoU. If the IoU of the prediction frame is greater than or equal to a given threshold (e.g., 0.5, 0.55, …, 0.95) and matches a true frame, it is considered a TP. otherwise, it is considered a FP. Then the precision and recall are calculated, precision is used to indicate the proportion of positive samples in the prediction results, and recall is used to indicate the proportion of samples that are actually positive classes that are correctly recognised as positive classes by the model. The calculation formula is as follows:(12)Pre=TPTP+FP(13)Recall=TPTP+FNWhere TP denotes true cases, i.e., the number of samples that are actually positive and correctly predicted to be positive. FP denotes false positive cases, i.e., the number of samples that are actually negative but incorrectly predicted to be positive. FN denotes false negative cases, i.e., the number of samples that are actually positive but incorrectly predicted to be negative. The precision-recall curve is first plotted through the sorted detection results. The area under the P-R curve is calculated to represent Average Precision (AP), which is calculated using interpolation in this paper:(14)AP=∑n(Rn-Rn-1)PnWhere Pn is the maximum precision corresponding to the nth recall and Rn is the nth recall. Finally mAP is calculated by AP. the formula is as follows:(15)mAP=1N∑i=1NAPiwhere N is the number of categories. The mAP in this paper is the mean value of the average precision (AP) calculated at multiple IoU thresholds (from 0.5 to 0.95 in steps of 0.05). mAP_50 is the average precision of the detection frame calculated at IoU = 0.50. Calculated in the same way as mAP, but with the IoU fixed at 0.50. mAP_75 is the mean accuracy of the detection frame calculated at IoU = 0.75. The calculation method is the same as that of mAP, but the IoU is fixed at 0.75. mAP_s and mAP_m represent the mAP values obtained when the area of the detected target is in the range of 0 to 322 and 322 to 622 pixel units, respectively. Quantitative analysis In order to quantitatively compare the effectiveness of the twin Swin Transformer for segmentation of osteosarcoma cells, we applied four deep learning methods to the BBBC039 osteosarcoma image set, and plotted the loss curves and the curves of the five evaluation metrics mentioned above based on the experimental results. From Fig. 7, it can be seen that the convergence speed of the loss curves of the four methods basically the same, and the loss curve corresponding to the twin Swin Transformer method proposed in this paper is smoother, and the final loss value is the smallest. It shows that the model corresponding to the method proposed in this paper is more stable.Fig. 7Loss curve graphs for the four methods. Loss curve graphs for the four methods. In order to further verify the effectiveness of the proposed twin Swin Transformer for osteosarcoma cell segmentation. When the loss function converges and the model tends to be stable, we selected the training results of the last 10 rounds and averaged the values of the four methods in each of the five evaluation metrics to obtain Table 1 as follows:Table 1Mean values of the four methods on the five evaluation indicators.backbonesegm_mAPsegm_mAP_50segm_mAP_75segm_mAP_ssegm_mAP_mResnet500.540.650.600.560.48Gcnet0.510.620.570.550.39Swin transformer0.620.740.700.610.65MF Twin Swin0.640.780.740.630.71 Table 1Mean values of the four methods on the five evaluation indicators.backbonesegm_mAPsegm_mAP_50segm_mAP_75segm_mAP_ssegm_mAP_mResnet500.540.650.600.560.48Gcnet0.510.620.570.550.39Swin transformer0.620.740.700.610.65MF Twin Swin0.640.780.740.630.71 Mean values of the four methods on the five evaluation indicators. Combining Table 1 and Fig. 8, it can be seen that compared to the other three methods, the twin Swin Transformer proposed in this paper achieves the best results in all the five evaluation metrics of the index. There is an improvement over the best results obtained by the other three methods for each metric, specifically: segm_mAP improved by 0.02, segm_mAP_50 improved by 0.04, segm_mAP_75 improved by 0.04, segm_mAP_s improved by 0.02,segm_mAP_m improved by 0.06.Fig. 8Specific performance of the four methods on the four evaluation metrics. Specific performance of the four methods on the four evaluation metrics. Qualitative analysis To further validate the effectiveness of the Twin Swin Transformer with multi-scale feature fusion proposed in this paper for osteosarcoma cell segmentation. Fig. 9 shows the three representative images selected and the distribution of the three osteosarcoma cell images from top to bottom in the first column of each image can be described as uniformly distributed; cells of varying sizes with adhesions, and sparsely distributed cells.Fig. 9Segmentation effect of each of the four methods on three images of osteosarcoma cells. Segmentation effect of each of the four methods on three images of osteosarcoma cells. As shown in Figs. 9 and 10, it can be seen that compared to Resnet, Gcnet, and Swin Transformer, the Twin Swin Transformer with multi-scale feature fusion method proposed in this paper achieves the best segmentation results on all three of the presented images Specifically, the method of ResNet50 has insufficient segmentation; there are many cases of segmenting multiple osteosarcoma cells into one cell, and the segmentation omission is the most serious, which will have the worst effect. The GcNet method is better than ResNet50, and more cases of segmentation omission exist. The segmentation effect of the Twin Swin Transformer with multi-scale feature fusion method is significantly improved compared to the ResNet50 and the GcNet methods; there is no segmentation of multiple osteosarcoma cells into a single cell, and segmentation omissions are substantially reduced. The Twin Swin Transformer with multi-scale feature fusion method has only a few cases of segmenting multiple osteosarcoma cells into a single cell, and the segmentation omission is the least among the four methods.Fig. 10Segmentation performance comparison. Segmentation performance comparison. The method proposed in this paper focuses on replacing the Swin Transformer Block in the original Swin Transformer Network with a Twin Swin Transformer Block, which can enhance the feature interaction of the input features at different stages. In addition, SENet is added to the original network, which allows the network to focus on the most important feature channels. Moreover, Skip Connections are introduced between different Twin Swin Transformer Block stages to improve the model's ability to process detailed information. With these improvements, our method effectively segments osteosarcoma cells, thus assisting physicians in efficiently and accurately diagnosing bone cancer. The paper centers on improving the original Swin Transformer Network, mainly emphasizing replacing the Swin Transformer Block with a Twin Swin Transformer Block to enhance the feature interaction of the input features. SENet is added to the original network, which allows the network to focus on the most important feature channels, thus improving the model's performance. Skip Connections are introduced between different Twin Swin Transformer Block stages to enhance the model's ability to process detailed information. The experimental part deals with the evaluation setup, including the dataset, experimental metrics, and comparative analysis with existing methods. Specifically, the experiments are performed on a dataset of bone tumour cell images captured under Hoechst-stained fluorescence microscopy to demonstrate the proposed method's effectiveness in each segmentation metric. Comparative analysis with existing methods ResNet50, GcNet, and Swin Transformer showed that the method performed well in terms of osteosarcoma cell segmentation effectiveness, especially for segm_mAP_75, segm_mAP_75, and cells with cell areas ranging from 322 to 622 pixel-unit scale sizes. The method performs excellently in the bone tumor cell image segmentation task In conclusion, the method proposed in this paper dramatically improves the accuracy of the network in the osteosarcoma cell segmentation task based on the original Swin Transformer Network by replacing the Swin Transformer Block in the original network with a Twin Swin Transformer, adding SENet, and introducing Skip Connections between different stages of the Twin Swin Transformer Block. The method performs excellently in the bone tumor cell image segmentation task and has a promising future in clinical bone cancer-assisted diagnosis. Based on the above research findings, our future research directions include: 1) introducing automated hyperparameter search and adaptive learning rate strategies to optimize model efficiency; 2) employing pruning, quantization, and knowledge distillation to achieve lightweight design and mobile deployment; 3) enhancing the model’s generalization ability across multi-center datasets through transfer learning and domain adaptation techniques; 4) exploring multimodal data integration to further improve the model’s performance in recognizing complex lesions. Conclusion In this paper, we design the twin Swin Transformer Block, through which the interaction of input features can be enhanced to capture the complex nonlinear relationship between input features, which helps improve the model's generalisation ability. In addition, based on the Swin Transformer network, we design a Twin Swin Transformer network with muti-scale feature fusion by replacing the Swin Transformer Block in the original network with the designed Twin Swin Transformer Block, which enhances the feature interactions of the input features, and also Skip Connections is introduced to increase the ability to capture detail information. Moreover, SENet is introduced in the twin Swin Transformer network to adaptively adjust the importance of each channel to enhance the feature representation capability, thus improving the network's performance. The experimental results show that the twin Swin Transformer method with multi-scale fusion proposed in this paper achieves ideal segmentation results in the osteosarcoma cell segmentation task. Improving the segmentation accuracy of osteosarcoma cells enhances the precision and timeliness of diagnosis by better capturing the morphological and structural features of the tumor cells. Future research directions focus on enhancing model accuracy and computational efficiency through deep feature extraction and efficient computing strategies. Efforts will also be directed towards optimizing parameter tuning and designing lightweight architectures to improve the model's applicability and scalability. Additionally, the research will explore the model's adaptability and generalization capabilities across multicenter datasets and various imaging devices, aiming to assess the potential feasibility and advantages of MF Twin Swin in practical clinical applications. Ethics approval This study utilized publicly available osteosarcoma datasets for research purposes. The dataset acquisition address is https://bbbc.broadinstitute.org/BBBC039. As such, ethics approval was not required for the use of these datasets, as they are openly accessible and do not involve direct human or animal subjects. The datasets were obtained from reputable sources and used in accordance with their respective terms of use and data sharing policies. CRediT authorship contribution statement Tingxi Wen: Supervision, Methodology. Binbin Tong: Writing – original draft, Software, Methodology, Conceptualization. Yuqing Fu: Software, Methodology. Yunfeng Li: Visualization, Software. Mengde Ling: Validation. Xinwen Chen: Visualization. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.