Rethinking pathology image analysis through shuffling Authors: Zeyu Liu, Tianyi Zhang, Brian K Chen, Youdan Feng, Shangqing Lyu, Yanli Lei, Nan Ying, Yunlu Feng, Yu Zhao, Peng Zhang, Fan Song, Chenbin Ma, Yufang He, Kenji Kawaguchi, Hwee Kuan Lee, Yueming Jin, Guanglei Zhang Published: 2026 ABSTRACT Pathological examination is the current gold standard in cancer diagnosis, yet artificial intelligence (AI) methods still struggle to capture the multi-scale heterogeneity of tumor morphology across patients, tissues, and magnifications. Here, we introduce the PAthoentity Shuffle Strategy (PASS), a principled framework that explicitly models pathoentities, the critical biological structures such as cells, glands, and tissues, and their hierarchical relationships. By controlled shuffling of pathoentities within and across samples, PASS enriches the relational structure available to neural networks, encouraging them to learn both local homogeneity and global heterogeneity. We provide theoretical analysis showing that PASS achieves error bounds comparable to state-of-the-art methods, supporting shuffling as a generalizable computational principle rather than a heuristic. Extensive evaluation on 10 datasets spanning 8 diseases, 9 organs, and 4 magnification levels demonstrates consistent performance gains, robust generalization, and scalability across diverse pathological contexts. Importantly, PASS further shows translational value in a rapid onsite evaluation (ROSE) scenario in gastroenterology, highlighting its potential for clinical deployment. Overall, this study establishes pathoentity shuffling as an effective principle for pathological image analysis, bridging biological insight and computational design to enhance diagnostic modeling and morphological hierarchy learning. Introduction In the current clinical approach, microscope-based pathological diagnosis with the region of interest (ROI) or whole slide image (WSI) still stands as the gold standard1,2. With the global shortage of experienced pathologists, artificial intelligence (AI) has emerged as a powerful tool for computational pathology, achieving promising results in disease classification3–5. Recent deep learning (DL) approaches have explored novel architectures, training strategies, and augmentation methods informed by pathological features6–8, successfully capturing homogeneous high-frequency features such as cellular or glandular morphology9,10. However, pathology is inherently complex: tumor heterogeneity manifests across individuals, tissues, and magnifications11,12. Addressing this complexity requires DL models that go beyond task-specific designs to explicitly capture morphological relationships both within and across samples13. In this study, we investigate such explicit modeling and demonstrate that a simple yet powerful strategy of “mixing and shuffling” can effectively capture these relationships and substantially improve pathological image analysis. Pathology images reflect phenotypic outcomes of underlying molecular and cellular processes, which manifest at multiple physical scales14,15. In practice, pathologists diagnose by dynamically adjusting magnification to capture features ranging from nucleoli to tissue organization16,17. For instance, in prostate cancer (Fig. 1a), abnormalities may appear at the cellular scale as nuclear enlargement or prominent nucleoli18,19, progress to altered cytoplasmic content or basal proliferations20, and at lower magnifications manifest as irregular glandular arrangements or inter-gland fusions21. At each scale, dominant diagnostic features reside in distinct histological entities, which we term pathoentities, including cells, glands, and tissues (Fig. 1b). These entities collectively provide local features (e.g., nuclear shape) and global patterns (e.g., cell density or distribution)22,23. Understanding pathology therefore requires modeling both individual pathoentities and their hierarchical relationships24.Fig. 1Illustration of the pathological image, pathoentity, and pathoentity relationship.a Introduction of pathological image scales with corresponding features. Prostate cancer biopsy is given as an example, showing images at different physical magnifications with certain pathological features. b Introduction of pathoentity. Three representative pathological pathoentities of cellular structure, cell, and tissue are displayed in the diagrams, each representing the local pathological features. Where multiple pathoentities present within the same image, their quantity or distribution can serve as the global features. The specific pathoentities within the image are highlighted with orange dashed lines. c Introduction of pathoentity relationship. According to the pathoentities’ origins and categories, their relationships can be divided into same-image, same-class, and cross-class types. The hierarchical structure of three pathoentity relationships are shown in a chart, and the difference of them can be represented by distance from a feature space view. Illustration of the pathological image, pathoentity, and pathoentity relationship. a Introduction of pathological image scales with corresponding features. Prostate cancer biopsy is given as an example, showing images at different physical magnifications with certain pathological features. b Introduction of pathoentity. Three representative pathological pathoentities of cellular structure, cell, and tissue are displayed in the diagrams, each representing the local pathological features. Where multiple pathoentities present within the same image, their quantity or distribution can serve as the global features. The specific pathoentities within the image are highlighted with orange dashed lines. c Introduction of pathoentity relationship. According to the pathoentities’ origins and categories, their relationships can be divided into same-image, same-class, and cross-class types. The hierarchical structure of three pathoentity relationships are shown in a chart, and the difference of them can be represented by distance from a feature space view. Critically, pathology images display local homogeneity, where the morphologies are statistically invariant under translation or rotation25, yet global heterogeneity emerges across magnifications, tissue regions, and patients. In pancreatic cancer, for example, benign and malignant cells may differ subtly in nuclear pleomorphism26,27, while tumor microenvironments host pathoentities with high intra-class similarity28,29. Biological variability further amplifies heterogeneity within the same class30, and cross-cohort differences risk model overfitting31. Capturing such complexity requires explicitly modeling pathoentity relationships. We classify these into same-image, same-class, and cross-class categories (Fig. 1c), which reflect hierarchical similarities and differences observable in both spatial orientation and morphological detail. Recent pathology foundation models have begun to capture multi-scale and hierarchical representations through large-scale pretraining. However, in many task-specific settings, downstream models are still trained on individual samples, focusing primarily on intra-image features without explicitly modeling inter-sample relationships. To overcome this limitation, we explore explicit modeling of pathoentity relationships. Our approach constructs mixed scenarios by shuffling pathoentities within and across samples, thereby exposing the network to relational structures that would otherwise remain hidden. By training on these enriched inputs, the model is compelled to capture both intra- and inter-sample dependencies, enabling a deeper understanding of pathological heterogeneity. Building on our prior work in mixing and shuffling methods for WSIs32–35, we generalize these approaches into the PAthoentity Shuffle Strategy (PASS). Earlier iterations involved naïve patch-level mixing or ROI-level shuffling, which already revealed benefits in capturing pathoentity patterns. Our latest formulation incorporates positional awareness and hierarchical relationships, yielding superior performance. Unlike task-specific architectures or ad hoc augmentations, PASS offers a simple, principled, and generalizable framework. In this study, we elucidate how PASS enables neural networks to learn pathoentity relationships by systematically constructing mixed scenarios across samples. Through extensive experiments on 10 datasets spanning 4 magnification scales, we demonstrate that PASS consistently outperforms state-of-the-art baselines, achieving robust generalization across cancer types and cohorts. Supported by theoretical analysis showing error bounds comparable to leading methods (Theorem 1), PASS establishes shuffling as a viable computational principle for pathological image analysis and a plug-and-play option for future computational pathology pipelines. Results Our findings highlight the critical importance of pathoentity, as it carries the dominant pathological features and serves as a fundamental basis for pathological diagnosis. Inspired by this, we study how PASS can enable the neural networks to effectively learn and capture this essential pathological information. PASS is implemented in two complementary forms: PASS-SI-ViT, leveraging random shuffling within a Transformer architecture34, and PASS-Cellmix, an in-place shuffle augmentation module35. To validate their effectiveness, verify our hypothesis of pathoentities, and assess how PASS enhances model performance, we conducted experiments from three perspectives: (1) model performance evaluation; (2) assessment of pathoentity importance; and (3) analytical exploration of shuffling mechanisms. Datasets and experimental settings Pathological diagnostic samples inherently exhibit substantial heterogeneity, originating from diverse hospitals and institutions with varying analytical protocols and disease-specific magnification requirements. To comprehensively validate our pathoentity hypothesis and comprehensively evaluate our methodology, we curated 10 benchmark datasets representing distinct pathological contexts and imaging modalities (Fig. 2a). Specifically, the dataset collection encompasses 4 representative scales of pathological magnification: 2 cell scale datasets, white blood cell (WBC) and SIPaKMeD36,37; 2 cell cluster scale datasets, ROSE and Colonoscopy38,39; 4 local tissue scale datasets, papillary renal cell carcinoma (pRCC), Warwick-QU (Warwick), MARS, and SEED-Gastric Cancer (GS)40–43; and 2 tissue scale datasets, CAMELYON16 (CAM16) and PAIP2019 (PAIP)44,45. Representing 4 pathoentity scales from high to low, we designated WBC/ROSE/pRCC/CAM16 as the pyramid-group datasets for more specific evaluations. Details of the datasets are provided in Supplementary Notes 1, 2, Supplementary Figs. 1–3, and Supplementary Table 1, with preprocessing pipeline described in Methods.Fig. 2Illustration of PASS.a Illustration of data. 10 datasets are employed in this study, spanning 9 diseases and 8 organs across 4 magnification scales. b The pipeline of PASS. Step 1: Based on the original pathological images’ scale and pathoentity, they are located to a specific layer in the pathoentity pyramid and accordingly preprocessed. Step 2: The preprocessed pieces are fed into the shuffler. By splitting, shuffling and regrouping, shuffled images are produced, essentially shuffling the pathoentities and introducing more pathoentity relationship. Step 3: These shuffled images, along with the original ones, are alternatively input to train the neural network. c Illustration of the shuffler. The shuffler splits the original image into pieces and then shuffles them with two strategies. For random shuffle, the split pieces are randomly shuffled from different images in a shuffle bag. For in-place shuffle, partial pieces (masked orange) are shuffled while the others remain fixed (masked blue). The shuffled pieces swap their positions in different images, while maintaining their relative positions. The details of the model, shuffler and shuffling strategy are explained in the “Methods”. d Classification performance comparison. Across all 10 datasets, PASS-SI-ViT outperforms all other models and PASS-Cellmix surpasses all other data augmentation methods. Illustration of PASS. a Illustration of data. 10 datasets are employed in this study, spanning 9 diseases and 8 organs across 4 magnification scales. b The pipeline of PASS. Step 1: Based on the original pathological images’ scale and pathoentity, they are located to a specific layer in the pathoentity pyramid and accordingly preprocessed. Step 2: The preprocessed pieces are fed into the shuffler. By splitting, shuffling and regrouping, shuffled images are produced, essentially shuffling the pathoentities and introducing more pathoentity relationship. Step 3: These shuffled images, along with the original ones, are alternatively input to train the neural network. c Illustration of the shuffler. The shuffler splits the original image into pieces and then shuffles them with two strategies. For random shuffle, the split pieces are randomly shuffled from different images in a shuffle bag. For in-place shuffle, partial pieces (masked orange) are shuffled while the others remain fixed (masked blue). The shuffled pieces swap their positions in different images, while maintaining their relative positions. The details of the model, shuffler and shuffling strategy are explained in the “Methods”. d Classification performance comparison. Across all 10 datasets, PASS-SI-ViT outperforms all other models and PASS-Cellmix surpasses all other data augmentation methods. For model evaluation, we first implemented PASS-SI-ViT, which incorporates random shuffling and a regression head to jointly process shuffled and original images (Fig. 2b, c). It was benchmarked against 12 representative models under identical conditions: 2 Transformers: ViT and Swin Transformer (Swin-B)46,47, 7 CNNs: VGG-16/19, ResNet50, Inception-V3, Xception, MobileNetV3 and EfficientNet-B348–53, 3 recent hybrid models: ResNet-ViT, Conformer and CrossFormer46,54,55, and 2 strong pathology foundation models: Virchow256 and UNI57. To assess extensibility, we further developed PASS-Cellmix, a model-agnostic augmentation module that applies in-place shuffling to enrich pathoentity relationships. Comparative tests included 6 state-of-the-art mixing-based augmentation methods58–63. Results are summarized in Fig. 2d, with full details in Supplementary Tables 2–3. Evaluation of the PASS-SI-ViT PASS-SI-ViT employs a dual-input training paradigm, integrating shuffled and original images to expose the model to expanded pathoentity relationships. This design yields state-of-the-art performance across all datasets (Fig. 2d), with accuracy (ACC) improvements of 0.74–7.55% and F1-score (F1) gains of 1.06–7.93% relative to 14 competing models. Comparative analysis with previous studies targeting similar classification tasks confirms the advantages of our approach (Supplementary Note 3 and Supplementary Fig. 4). On pyramid-group datasets, receiver operating characteristic (ROC)64 analyses further demonstrate AUC advantages of 0.85–5.52% (Supplementary Figs. 5–12). These findings collectively confirm the efficacy of PASS-SI-ViT and validate the central hypothesis that explicit pathoentity modeling enhances learning. To better interpret model behavior, we used gradient-weighted class activation mapping (Grad-CAM) across 6 representative architectures (2 CNNs, 2 Transformers, 1 hybrid model, and PASS-SI-ViT). Binary-class samples from each pyramid-group dataset were analyzed, with pathologist annotations provided as reference (Fig. 3). CNNs demonstrated strong local feature sensitivity but failed to capture broader tissue organization, leading to misclassification in datasets like ROSE and CAM16. Transformer and hybrid models improved global context modeling but often overemphasized background or staining noise, reducing stability. In contrast, PASS-SI-ViT consistently focused on diagnostically relevant regions, producing attention maps closely aligned with expert annotations.Fig. 3Attention map analysis of PASS models against other SOTA classification models on pyramid-group datasets.One image is selected in two classes from WBC (5 classes) and each class from ROSE, pRCC and CAM16 datasets. The first column shows the original image and the next five columns are Grad-CAM results from other SOAT models. PASS models with two strategies are marked with a special color and shown behind. The classification possibilities provided by the models are annotated below the corresponding images. The last column presents the key diagnostic regions labeled by the pathologists. Attention map analysis of PASS models against other SOTA classification models on pyramid-group datasets. One image is selected in two classes from WBC (5 classes) and each class from ROSE, pRCC and CAM16 datasets. The first column shows the original image and the next five columns are Grad-CAM results from other SOAT models. PASS models with two strategies are marked with a special color and shown behind. The classification possibilities provided by the models are annotated below the corresponding images. The last column presents the key diagnostic regions labeled by the pathologists. At different pathological scales, PASS-SI-ViT consistently captured features that were often overlooked by baseline models. At the cellular level (WBC), the model accurately localized nuclear morphology, such as the bi-lobed nuclei of basophils and the kidney-shaped nuclei with foamy cytoplasm of monocytes, while effectively suppressing staining artifacts that misled conventional CNNs. Moving to the cell-cluster level (ROSE), PASS-SI-ViT successfully detected atypical cell clusters characterized by enlarged nuclei, high nuclear-to-cytoplasmic ratios, and irregular nuclear membranes, with attention patterns closely aligned with expert annotations. At the local tissue level (pRCC), the model demonstrated the ability to distinguish subtle yet diagnostically critical differences between papillary subtypes, whereas other architectures often conflated overlapping morphological features. Finally, at the tissue architecture scale (CAM16), PASS-SI-ViT revealed a sophisticated capacity to link spatially separated metastatic clusters that shared cytological features but were embedded within normal lymphoid tissue, thereby capturing inter-sample relationships beyond the scope of prior models. Together, these results highlight the capacity of PASS-SI-ViT to integrate both local and global morphological cues, yielding robust, interpretable, and clinically meaningful predictions. Evaluation of the PASS-Cellmix The ascendance of the PASS-SI-ViT in classification evaluation confirms the effectiveness of pathoentity-based learning. To further extend this principle, we developed PASS-Cellmix, an in-place shuffle module designed as a general-purpose data augmentation strategy for pathological image analysis. Unlike PASS-SI-ViT, which directly integrates shuffling into a specific Transformer framework, PASS-Cellmix can be flexibly combined with diverse model architectures while maintaining computational efficiency. Comparative analyses against 6 state-of-the-art augmentation methods, all sharing the same ViT backbone, demonstrate that PASS-Cellmix consistently achieves superior performance, with gains of 0.64–1.56% in ACC and 0.86–1.73% in F1 (Fig. 2d). On the pyramid-group datasets, it further attains the highest average AUC of 95.1%, highlighting its robust capacity to enhance representation learning through explicit pathoentity modeling. When compared with PASS-SI-ViT, PASS-Cellmix also yields a modest advantage (0.33% in ACC and 0.31% in F1). This improvement can be attributed to its dynamic training process, in which the in-place shuffle retains partial spatial information by fixing a subset of image pieces in their original locations. Such a strategy preserves local structural relationships while introducing diversity, producing distinctive feature patterns that enrich learning. To further adapt to training progression, PASS-Cellmix employs a dynamic shuffling schedule in which parameters such as patch size, fixed-position ratio, and shuffle list evolve with the model’s learning state. This adaptive mechanism broadens its representational capacity, enabling a more comprehensive capture of pathoentity relationships across scales and samples. As a plug-and-play augmentation module, PASS-Cellmix also demonstrates strong generalizability across multiple architectures, yielding consistent improvements beyond the ViT backbone (Supplementary Note 4 and Supplementary Table 3). Despite these advantages, there remain distinctions between the two PASS implementations. PASS-SI-ViT, benefiting from explicit pathoentity region masks, excels at localizing diagnostically relevant regions and generating uniform attention guided by prior information, as confirmed by Grad-CAM visualizations. In contrast, PASS-Cellmix does not rely on mask labels and instead learns to allocate attention weights autonomously during training. While this may reduce precision in boundary delineation, it equips the model with the flexibility to discover critical regions in a self-supervised manner. Compared with conventional augmentation techniques, PASS-Cellmix produces more stable attention maps and higher classification confidence (Supplementary Fig. 13), underscoring the added value of incorporating explicit morphological priors into augmentation strategies. CAM-based shuffling analysis PASS is designed to enrich model learning by introducing novel pathoentity relationships through shuffled images. Understanding how the models process and interpret these shuffled representations is therefore crucial. To this end, we conducted a comparative visualization of shuffled images, alongside the Grad-CAMs and ground truth annotations under different shuffling strategies (Fig. 4). Shuffling rearranges the spatial distribution of pathoentities while preserving the intrinsic information within each piece, thereby altering inter-piece relationships and compelling the models to recognize class-specific entities independent of their original context. Grad-CAM visualizations show that both PASS implementations successfully identify diagnostically relevant pieces while minimizing attention to background regions, even at patch boundaries. This suggests that the models prioritize morphological content within each piece rather than superficial spatial discontinuities.Fig. 4Attention map of PASS models on shuffled images.Random shuffle strategy (left) and in-place shuffle strategy (right) are both tested. In each part, the first column is the shuffled sample which composes cropped pieces from four images. The middle two columns show the Grad-CAM results on two classes. Here the piece size is set to 128 for clearer display. Last column shows the masks labeled on original images by experienced pathologists. Attention map of PASS models on shuffled images. Random shuffle strategy (left) and in-place shuffle strategy (right) are both tested. In each part, the first column is the shuffled sample which composes cropped pieces from four images. The middle two columns show the Grad-CAM results on two classes. Here the piece size is set to 128 for clearer display. Last column shows the masks labeled on original images by experienced pathologists. Significantly, when shuffled samples contain patches dominated by background, a common occurrence in WBC or ROSE datasets, the models consistently suppress attention to these irrelevant regions and instead focus on the critical pathoentities. As illustrated in Fig. 4, PASS models not only capture key features within each piece but also discriminate subtle differences between pathoentities of distinct classes, establishing a solid foundation for accurate classification. This robustness highlights the effectiveness of shuffling in guiding networks to attend selectively to biologically meaningful features, while avoiding confounding signals such as staining variations, background noise, or artificial patch boundaries. By emulating the pathologist’s focus on diagnostic regions and disregarding irrelevant context, PASS demonstrates both resilience to challenging imaging scenarios and broad applicability in computational pathology. Assessment of pathoentity importance In the previous sections we defined the concept of pathoentities and established their role as the foundation of PASS. To empirically validate their importance, we conducted masking experiments on the pyramid-group datasets, selectively occluding either diagnostically critical regions (pathoentities) or non-essential areas, and then evaluating performance using PASS-Cellmix. This model was chosen as the baseline because of its autonomous learning capacity, which minimizes bias from external label guidance. Grad-CAM visualizations (Fig. 5a) and quantitative metrics (Fig. 5b) were used to assess both attention distribution and classification performance under these perturbations.Fig. 5Evaluation of the impact of pathoentity masking on model performance.a Grad-CAM of PASS model on original images and masked images. Two classes of samples are selected from pyramid-group datasets and discriminated by blue or orange color. Box outside the pathological image indicates its original label, and box outside the CAM denotes the class prediction from model. Two boxes with same color indicate the correct prediction. b Comparison of model classification performance on original images (histogram in pink) and masked images (purple). The Acc and F1 are presented in the left and right columns. Evaluation of the impact of pathoentity masking on model performance. a Grad-CAM of PASS model on original images and masked images. Two classes of samples are selected from pyramid-group datasets and discriminated by blue or orange color. Box outside the pathological image indicates its original label, and box outside the CAM denotes the class prediction from model. Two boxes with same color indicate the correct prediction. b Comparison of model classification performance on original images (histogram in pink) and masked images (purple). The Acc and F1 are presented in the left and right columns. Masking experiments revealed distinct patterns across datasets and scales. In WBC samples, removal of intracellular structures, which serves as pathoentities at the cellular level, led to a dramatic decline in performance, with accuracy falling to 20–30%, approaching random chance. Similarly, in ROSE samples, occlusion of atypical pancreatic cells caused a surge of false negatives, yielding a nearly 25% drop in ACC and a 60% decline in F1. In both cases, despite preservation of broader morphological context, the absence of pathoentities rendered the models nearly incapable of correct classification, highlighting their indispensability at fine-grained scales. By contrast, in pRCC images, where diagnostic cues are distributed across local tissue architecture, masking individual cellular features had only a moderate impact. The models maintained much of their performance by relying on preserved cluster-level structures, though attention shifts indicated that loss of key cellular details could still influence outcomes in some cases. In CAM16 datasets, masking metastatic regions created artificial contrasts that occasionally substituted for original pathological cues, sustaining relatively high attention and classification accuracy. However, when models redirected focus to normal tissue, occasional misclassifications occurred, explaining the modest overall performance drop. Together, these results demonstrate a strong correspondence between pathoentity masking, attention patterns, and classification outcomes. Severe performance deterioration in WBC and ROSE confirms that small-scale pathoentities carry dominant diagnostic value, while the resilience of pRCC and CAM16 reflects the contribution of higher-level tissue structures. These findings reinforce the centrality of pathoentities in pathological image analysis, as well as the necessity of modeling their hierarchical relationships. The consistency between model behavior and pathologists’ diagnostic reasoning highlights the validity of our framework and underscores the effectiveness of PASS in explicitly leveraging pathoentity information across scales. Analytical exploration of shuffling mechanisms To further optimize PASS and clarify the mechanisms underlying its effectiveness, we performed a series of ablation studies on the pyramid-group datasets. These analyses examined the contribution of key architectural components, the influence of piece- and image-level configurations, and the impact of different shuffle strategies on pathoentity learning. Effectiveness of key components in PASS PASS improves classification by explicitly modeling pathoentity relationships. In SI-ViT, this is achieved by extending the standard ViT architecture with an auxiliary regression head (REG) in addition to the classification head. The REG head provides patch-level supervision through soft label regression, enabling finer representation of pathoentities while maintaining standard classification through the MLP head. Compared to the baseline ViT, this dual-head design significantly boosts performance, with average gains of 1.26% in ACC and 1.70% in F1-score across pyramid-group datasets (Table 1). However, when random shuffle (RS) was applied only to the REG branch, performance fluctuated due to inconsistencies between shuffled and unshuffled supervisory signals. In the final PASS-SI-ViT design, we resolved this limitation by applying patch-level supervision to both shuffled and unshuffled inputs, while restricting classification to the original images. This unified training paradigm achieved the best results (96.30% ACC, 95.06% F1), confirming that explicit supervision at both the entity and relational level enhances the robustness of pathoentity representations.Table 1Ablation study on the model structure and shuffling strategyModelStrategyWBCROSEpRCCCAM16Acc (%)F1 (%)Acc (%)F1 (%)Acc (%)F1 (%)Acc (%)F1 (%)ViTNone98.34 (±0.55)97.04 (±0.87)91.24 (±0.75)86.29 (±0.92)92.58 (±0.54)89.95 (±0.66)93.06 (±0.68)92.96 (±0.68)REG-ViTREG Head98.32 (±0.47)97.04 (±0.61)92.13 (±0.55)88.51 (±0.76)93.99 (±0.77)91.71 (±0.79)95.83 (±0.81)95.77 (±0.82)SF-ViTShuffle (Random)98.32 (±0.49)97.05 (±0.63)93.41 (±0.48)90.41 (±0.71)92.93 (±0.47)90.99 (±0.5)94.91 (±0.64)94.84 (±0.65)PASS-SI-ViTREG Head + Shuffle (Random)99.12 (±0.24)98.38 (±0.42)94.00 (±0.19)91.32 (±0.24)95.76 (±0.35)94.29 (±0.38)96.30 (±0.62)96.23 (±0.62)PASS-CellmixShuffle (In-place)99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.40)96.30 (±0.69)96.23 (±0.70)The classification accuracy (Acc) and F1-score (F1) on different model structure and shuffling strategies are shown across pyramid-group datasets. Mean and standard deviation in parentheses are obtained through 5-fold cross validation.Bold textindicates the highest performance on the two PASS models. Table 1Ablation study on the model structure and shuffling strategyModelStrategyWBCROSEpRCCCAM16Acc (%)F1 (%)Acc (%)F1 (%)Acc (%)F1 (%)Acc (%)F1 (%)ViTNone98.34 (±0.55)97.04 (±0.87)91.24 (±0.75)86.29 (±0.92)92.58 (±0.54)89.95 (±0.66)93.06 (±0.68)92.96 (±0.68)REG-ViTREG Head98.32 (±0.47)97.04 (±0.61)92.13 (±0.55)88.51 (±0.76)93.99 (±0.77)91.71 (±0.79)95.83 (±0.81)95.77 (±0.82)SF-ViTShuffle (Random)98.32 (±0.49)97.05 (±0.63)93.41 (±0.48)90.41 (±0.71)92.93 (±0.47)90.99 (±0.5)94.91 (±0.64)94.84 (±0.65)PASS-SI-ViTREG Head + Shuffle (Random)99.12 (±0.24)98.38 (±0.42)94.00 (±0.19)91.32 (±0.24)95.76 (±0.35)94.29 (±0.38)96.30 (±0.62)96.23 (±0.62)PASS-CellmixShuffle (In-place)99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.40)96.30 (±0.69)96.23 (±0.70)The classification accuracy (Acc) and F1-score (F1) on different model structure and shuffling strategies are shown across pyramid-group datasets. Mean and standard deviation in parentheses are obtained through 5-fold cross validation.Bold textindicates the highest performance on the two PASS models. Ablation study on the model structure and shuffling strategy The classification accuracy (Acc) and F1-score (F1) on different model structure and shuffling strategies are shown across pyramid-group datasets. Mean and standard deviation in parentheses are obtained through 5-fold cross validation. Bold textindicates the highest performance on the two PASS models. Effectiveness of piece-level settings in pathoentity learning The PASS framework, comprising piece splitting, shuffling, and regrouping operations, enhances model performance by explicitly modeling both pathoentities and their relationships. Since pathoentities are inherently contained within individual pieces, piece size becomes a critical parameter that governs the balance between information retention and relational reconstruction. Larger pieces preserve more biological context but generate fewer inter-pathoentity relationships, whereas smaller pieces maximize relational diversity at the expense of intra-entity detail. Experiments across pyramid-group datasets revealed strong piece size–dependent effects, with optimal values of 64, 48, 48, and 64 pixels for WBC, ROSE, pRCC, and CAM16, respectively (Fig. 6a and Supplementary Fig. 14). These values closely approximate the characteristic dimensions of the dominant pathoentities in each dataset, suggesting that optimal shuffling occurs when individual entities are fully contained within appropriately sized pieces. Grad-CAM analysis further confirmed this phenomenon (Supplementary Fig. 15): excessively large pieces diluted fine-grained details, while excessively small pieces fragmented structural context. The optimal piece size thus reflects a trade-off, preserving essential structural information while maximizing relational learning through shuffling.Fig. 6Ablation study on piece-level and image-level settings.a, b Acc and F1 curves corresponding to different piece size on pyramid-group datasets and 3-scales Colonoscopy datasets. c Performance improvements resulting from the introduction of different pathoentity relationships on pyramid-group datasets. d Performance improvement curves corresponding to different sizes of shuffle bag. e The visualization of shuffling piece size on pyramid-group datasets. Ablation study on piece-level and image-level settings. a, b Acc and F1 curves corresponding to different piece size on pyramid-group datasets and 3-scales Colonoscopy datasets. c Performance improvements resulting from the introduction of different pathoentity relationships on pyramid-group datasets. d Performance improvement curves corresponding to different sizes of shuffle bag. e The visualization of shuffling piece size on pyramid-group datasets. To assess the generalizability of this observation, we examined datasets that contain multiple morphological scales within the same pathology. The Colonoscopy dataset, which includes images at cell, cell cluster, and local tissue magnifications, revealed consistent scale-dependent optimization: 48, 32, and 48 pixels, respectively (Fig. 6b), aligning with earlier findings that optimal piece size correlates with the magnification scale of the data (Supplementary Note 5 and Supplementary Fig. 16). Additionally, performance metrics revealed a systematic decline from low to high magnification scales. Grad-CAM analyses at fixed piece sizes (32 and 48 pixels) elucidate this phenomenon (Supplementary Fig. 17): the same piece size captures different information across scales, resulting in complete pathological structures at lower magnifications versus only partial colorimetric/textural cues at higher magnifications. Therefore, the functional utility of a piece depends on both its size and the imaging scale/content; accordingly, variability across piece sizes (intra-dataset) and magnifications (inter-scale) jointly shapes model performance. Effectiveness of image-level settings in pathoentity learning Since pathoentity relationships are determined by how pieces are shuffled, the construction of shuffled images directly shapes the type and complexity of information available for learning. Different shuffle strategies thus yield distinct relational patterns and diagnostic value. The same-image shuffle reorganizes patches from a single image, preserving intra-image features while altering spatial context. Although it introduces no external pathoentities, it enhances intra-sample relational learning and yields modest but consistent improvements over unshuffled baselines (Supplementary Fig. 18a). The same-class shuffle, which mixes patches across images of the same class, reflects the natural variability of pathological manifestations such as morphology, tissue architecture, and staining. By enforcing coherence within class labels while diversifying inputs, it improves robustness to intra-class heterogeneity (Supplementary Fig. 18b), a key challenge in pathology. Finally, cross-class shuffle generates the richest relational diversity by combining patches from multiple classes with soft labels (Fig. 4), compelling the model to perform fine-grained discrimination at the patch level. This strategy most closely parallels the diagnostic reasoning of pathologists, who interpret local structures within heterogeneous tissue contexts, and consistently achieves the strongest performance across datasets (Fig. 6c). While greater relational diversity generally enhances learning, our analysis identifies an optimal complexity threshold. Using the in-place shuffle strategy, we varied the shuffle bag size, the number of source images contributing patches to each shuffled sample, and observed a non-monotonic performance trend (Fig. 6d). Small bags (two images) provided limited diversity, whereas very large bags (eight images) introduced excessive complexity that biased the model toward individual patches and impaired global pattern recognition. The best results consistently emerged with four-image bags, which balanced relational richness with manageable complexity. A similar principle applies to the fix-position ratio (FPR), which controls how many pieces retain their original spatial positions during shuffling. Higher FPR values reduce relational novelty but simplify the learning task. Evaluations across pyramid-group datasets (FPRs 0.5–0.9) revealed dataset-specific optima, indicating that the effectiveness of spatial consistency depends on morphological characteristics (Supplementary Fig. 19). Moderate FPR values provided the best balance, preserving sufficient structural context while still introducing new relationships. These findings collectively support the view that carefully tuned relational complexity, through bag size and spatial consistency, is essential for maximizing the benefits of PASS while avoiding over-fragmentation or trivialization of pathoentity learning. Effectiveness of different shuffle strategies in pathoentity learning We have meticulously explored the optimal approach to convey pathoentity information to the model and figured out the most effective way to increase pathoentity relationships. Thus, we lastly explored the generator settings to facilitate the model better learning from the shuffled images. Building on previous ablation findings, we designed dynamic schedulers for piece size and FPR, enabling models to adaptively process shuffled images across a broad parameter space. Inspired by curriculum learning, we implemented a progressive strategy: training began with larger pieces and higher FPR values to stabilize feature learning, followed by gradual reduction to introduce increasingly complex pathoentity relationships. A loss-driven adaptive mechanism guided this progression, aligning shuffle granularity with the model’s convergence state. As shown in Table 2a, this loss-aware scheduler consistently outperformed alternative strategies (reverse, linear, random, and loop), demonstrating the value of dynamically matching shuffle complexity to learning pace.Table 2Ablation study on shuffle generator settingPartStrategyShuffle settingWBCROSEpRCCCAM16Acc (%)F1 (%)Acc (%)F1 (%)Acc (%)F1 (%)Acc (%)F1 (%)(a)In-place shuffleSchedularReverse98.50 (±0.46)97.59 (±0.52)93.21 (±0.53)90.10 (±0.61)96.47 (±0.43)95.24 (±0.56)95.37 (±0.80)94.17 (±0.81)Linear98.82 (±0.41)98.01 (±0.45)93.80 (±0.32)90.83 (±0.47)95.76 (±0.63)94.29 (±0.71)94.44 (±0.74)95.77 (±0.75)Random98.20 (±0.53)96.99 (±0.59)93.90 (±0.45)91.22 (±0.59)95.41 (±0.4)93.66 (±0.49)94.44 (±1.32)92.82 (±1.36)Loop98.78 (±0.35)97.93 (±0.37)92.32 (±0.52)88.50 (±0.77)95.41 (±0.37)93.72 (±0.47)95.83 (±0.64)93.90 (±0.65)Loss-drive99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.4)96.30 (±0.69)96.23 (±0.70)(b)In-place shufflePiece sizeSettled (Highest)99.12 (±0.34)98.38 (±0.43)94.00 (±0.48)91.32 (±0.55)96.11 (±0.29)94.84 (±0.39)95.83 (±0.87)95.77 (±0.88)Adaptive99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.4)96.30 (±0.69)96.23 (±0.70)(c)In-place shuffleFix- position ratioSettled (Highest)99.03 (±0.26)98.32 (±0.41)93.90 (±0.3)91.19 (±0.37)95.05 (±0.52)93.27 (±0.56)95.37 (±0.74)95.19 (±0.74)Adaptive99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.4)96.30 (±0.69)96.23 (±0.70)(d)In-place shufflePiece listEven98.62 (±0.38)97.62 (±0.4)93.70 (±0.54)90.75 (±0.63)96.47 (±0.49)95.28 (±0.51)93.98 (±0.96)93.95 (±0.97)Odd98.32 (±0.52)97.25 (±0.63)94.19 (±0.44)91.56 (±0.72)95.76 (±0.56)94.34 (±0.57)92.59 (±0.78)92.52 (±0.78)All99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.4)96.30 (±0.69)96.23 (±0.70)Ablation experiments on various generator settings are shown, including scheduler, piece size, fix-position ratio and piece size list in parts (a), (b), (c), (d), respectively. Model performances on pyramid-group datasets are listed, in which the highest are annotated in bold. Table 2Ablation study on shuffle generator settingPartStrategyShuffle settingWBCROSEpRCCCAM16Acc (%)F1 (%)Acc (%)F1 (%)Acc (%)F1 (%)Acc (%)F1 (%)(a)In-place shuffleSchedularReverse98.50 (±0.46)97.59 (±0.52)93.21 (±0.53)90.10 (±0.61)96.47 (±0.43)95.24 (±0.56)95.37 (±0.80)94.17 (±0.81)Linear98.82 (±0.41)98.01 (±0.45)93.80 (±0.32)90.83 (±0.47)95.76 (±0.63)94.29 (±0.71)94.44 (±0.74)95.77 (±0.75)Random98.20 (±0.53)96.99 (±0.59)93.90 (±0.45)91.22 (±0.59)95.41 (±0.4)93.66 (±0.49)94.44 (±1.32)92.82 (±1.36)Loop98.78 (±0.35)97.93 (±0.37)92.32 (±0.52)88.50 (±0.77)95.41 (±0.37)93.72 (±0.47)95.83 (±0.64)93.90 (±0.65)Loss-drive99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.4)96.30 (±0.69)96.23 (±0.70)(b)In-place shufflePiece sizeSettled (Highest)99.12 (±0.34)98.38 (±0.43)94.00 (±0.48)91.32 (±0.55)96.11 (±0.29)94.84 (±0.39)95.83 (±0.87)95.77 (±0.88)Adaptive99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.4)96.30 (±0.69)96.23 (±0.70)(c)In-place shuffleFix- position ratioSettled (Highest)99.03 (±0.26)98.32 (±0.41)93.90 (±0.3)91.19 (±0.37)95.05 (±0.52)93.27 (±0.56)95.37 (±0.74)95.19 (±0.74)Adaptive99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.4)96.30 (±0.69)96.23 (±0.70)(d)In-place shufflePiece listEven98.62 (±0.38)97.62 (±0.4)93.70 (±0.54)90.75 (±0.63)96.47 (±0.49)95.28 (±0.51)93.98 (±0.96)93.95 (±0.97)Odd98.32 (±0.52)97.25 (±0.63)94.19 (±0.44)91.56 (±0.72)95.76 (±0.56)94.34 (±0.57)92.59 (±0.78)92.52 (±0.78)All99.26 (±0.19)98.66 (±0.26)94.49 (±0.32)92.07 (±0.48)97.17 (±0.34)96.19 (±0.4)96.30 (±0.69)96.23 (±0.70)Ablation experiments on various generator settings are shown, including scheduler, piece size, fix-position ratio and piece size list in parts (a), (b), (c), (d), respectively. Model performances on pyramid-group datasets are listed, in which the highest are annotated in bold. Ablation study on shuffle generator setting Ablation experiments on various generator settings are shown, including scheduler, piece size, fix-position ratio and piece size list in parts (a), (b), (c), (d), respectively. Model performances on pyramid-group datasets are listed, in which the highest are annotated in bold. Comparisons with fixed-parameter baselines further confirmed the benefit of dynamic adaptation. Models trained with static piece sizes (16–192 pixels) or fixed FPR values (50–90%) performed consistently worse than the adaptive scheduler across all metrics (Table 2b, c). These results underscore the importance of exposing models to multi-scale features and variable spatial contexts for robust pathoentity representation. Moreover, experiments with restricted piece-size subsets (16/48/96/128 or 16/32/64/128) showed clear declines compared to the full range, indicating that each scale contributes unique information and that omitting any subset reduces the capacity to capture diverse structural patterns (Table 2d). These observations align with prior piece-level ablation studies (Fig. 6a, b) and highlight the advantage of comprehensive, multi-scale input compositions. Taken together, these ablation studies demonstrate that PASS achieves its strength by carefully balancing structural fidelity and relational diversity. Optimal performance arises when piece size aligns with biological scale, shuffling strategies introduce manageable complexity, and training schedules adaptively expose the model to multi-scale pathoentity relationships. These mechanisms collectively explain why PASS excels in capturing the heterogeneity and hierarchical structure of pathological images, aligning computational modeling with the diagnostic reasoning of expert pathologists. Discussion DL has achieved remarkable success in natural image analysis, largely driven by massive datasets and increasingly sophisticated architectures. However, replicating this progress in pathology remains difficult. Pathological images are inherently scarce, costly to annotate, and highly sensitive to slide preparation and staining protocols, which severely limit model generalizability. These challenges highlight the need for methods that move beyond brute-force scaling of data or networks and instead exploit the intrinsic characteristics of pathology images. In this study, we addressed this need by introducing the concept of pathoentities, biological units such as cells, glands, and tissues that encapsulate key diagnostic information, and by modeling their relationships explicitly. Our PASS, implemented through PASS-SI-ViT and PASS-Cellmix, generates new relational contexts among pathoentities, enabling networks to learn beyond intra-image features and toward inter-sample, hierarchical representations. By embedding prior pathological knowledge into the learning process, PASS bridges biological interpretability and computational modeling, establishing shuffling not merely as a data manipulation technique but as a principled strategy for representing the hierarchical and heterogeneous nature of pathology. Comprehensive evaluation across 10 diverse datasets confirmed both the robustness and scalability of PASS. PASS-SI-ViT achieved state-of-the-art performance with precise attention localization that closely aligned with expert annotations, while PASS-Cellmix extended the framework into a flexible augmentation module that improved multiple architectures (Figs. 2, 3). Pathoentity masking experiments provided further evidence: performance degradation occurred most severely when critical pathoentities were obscured, underscoring their indispensability for diagnosis (Fig. 4). Ablation studies clarified how piece size, shuffle complexity, and relational diversity jointly influence model behavior, revealing optimal scales that match the dimensions of underlying pathoentities and validating our hypothesis that effective modeling must respect biological structure (Figs. 4–6). Together, these results show that explicitly incorporating pathoentity relationships enables DL models to capture both intra- and inter-sample morphological structures with improved accuracy, robustness, and interpretability. Importantly, PASS also demonstrated strong clinical relevance. In gastroenterology workflows, PASS-powered models proved effective for rapid onsite evaluation (ROSE), reducing diagnostic burden without sacrificing interpretability (Supplementary Note 6, Supplementary Figs. 20–22, Supplementary Table 4 and Supplementary Video 1). Beyond classification, PASS showed potential in predicting molecular biomarkers such as HER2 status directly from H&E slides, revealing its ability to capture morphological correlates of molecular alterations (Supplementary Note 7 and Supplementary Figs. 23, 24). Furthermore, under data-scarce conditions, PASS retained competitive performance with only a fraction of training data, suggesting substantial gains in data efficiency (Supplementary Note 8 and Supplementary Fig. 25). These advantages point to the translational potential of PASS in accelerating clinical adoption and broadening applicability to previously underexplored diagnostic domains. While our current methods show substantial promise, several directions remain open for exploration. First, the identification and formalization of pathoentity relationships offer a new perspective for understanding patterns in pathological images and may benefit from integration with weakly supervised or unsupervised techniques. Then, although our study evaluated two shuffling strategies (RS and in-place shuffle), future work could explore more principled and biologically motivated shuffling methods. In addition, the shuffler module itself is lightweight, plug-and-play, and architecture-agnostic, making it well-suited for transfer learning and application to other pathology tasks such as segmentation, object detection, and survival prediction. Finally, since routine pathological diagnosis often relies on integrating information from multiple sections or slides of the same patient and real-world deployment further involves cross-center and out-of-distribution variability, extending PASS toward patient-level multi-slide relational learning and broader external validation would be important future directions for making the framework more clinically realistic and robust. Methods In this section, we present theoretical support for shuffle-based data augmentation methods, the backbone of the PASS methodology. First, we describe how each data is preprocessed before passed through the PASS algorithms. In the following subsections, we express the shuffle data-augmentation methodology (RS and in-place shuffle) in the same notation as other state-of-the-art approaches. Last but not least, we introduce error bounds for in-place shuffle to be compared with existing SOTA methods such as Mixup59 and Cutmix60. Preprocess Given the diverse range of datasets in terms of quantity, scale, and resolution, a multitude of processing methods are employed in our study. We divide the dataset into 4 groups to evaluate model performance on images and pathoentities of different scales. Due to the mismatch between the magnification of the original dataset and the 4 groups, we adjusted the magnification by cropping the image to modify the field of view. Furthermore, RS requires foreground and background segmentation, which is absent in some datasets. To resolve this, we apply a semi-automatic segmentation algorithm for rough segmentation on data without masks. Specifically, CAM16 and PAIP datasets are annotated WSI datasets. We extract non-overlapping square image patches containing tissue regions from each WSI, with positive patches containing no less than 40% positive regions. We determine the pixel side length for patch division based on the micron per pixel information, ensuring the physical side length of each patch is 1280 μm. After pre-processing, all images are center-cropped and resized to 384 × 384, then divided into training (70%), validation (10%), and test (20%) sets. Standard notation: random shuffle RS breaks each datapoint into multiple pieces and combines the existing pieces to form new images. When conducting RS, each datapoint consists of an image, a mask and its associated label. Let (X,M,Y)(X,M,Y) denote the training image, mask and label respectively. The training set contains mm datapoints: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{({X}^{\left(j\right)},{M}^{\left(j\right)},{Y}^{\left(j\right)}){\}}_{j\in \left[m\right]}$$\end{document}{(Xj,Mj,Yj)}j∈m. In RS, we attempt to split the images and associated masks into “puzzle” pieces of fixed sizes, before these pieces are shuffled to generate new images with soft labels. The generated images are then used to train the model with a custom loss function. Puzzle generator To generate the individual puzzle pieces, we begin by deciding a fixed cutout \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\{{C}_{1},{C}_{2},\ldots ,{C}_{p}\right\}$$\end{document}C1,C2,…,Cp to separate the images and masks into pp pieces of the same size. Cutouts are defined such that for all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i,j\in [p],{C}_{i}\odot {C}_{j}=\underline{O},|{C}_{i}|=|{C}_{j}|$$\end{document}i,j∈[p],Ci⊙Cj=O_,|Ci|=|Cj| if i≠ ji≠j and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${C}_{{ij}}=1$$\end{document}Cij=1 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${or}$$\end{document}or 00. We define the ⊙⊙ operator as the inner product involving element-wise multiplication. Furthermore, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\sum }_{i=1}^{p}{C}_{i}=J$$\end{document}∑i=1pCi=J where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${J}_{{ij}}=1\forall i,j\in [d]$$\end{document}Jij=1∀i,j∈[d]. This cutout is used to split both the image and its corresponding mask in equal fashion to obtain the same pieces. Given datapoint X,M,YX,M,Y, we cutout the image and the mask \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\{{X}_{1},{X}_{2},\ldots ,{X}_{p}\right\}=\left\{{C}_{1},{C}_{2},\ldots ,{C}_{p}\right\}\odot X$$\end{document}X1,X2,…,Xp=C1,C2,…,Cp⊙X and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\{{M}_{1},{M}_{2},\ldots ,{M}_{p}\right\}=\left\{{C}_{1},{C}_{2},\ldots ,{C}_{p}\right\}\odot M$$\end{document}M1,M2,…,Mp=C1,C2,…,Cp⊙M. The resulting pieces are \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\left\{{(X}_{i},{M}_{i})\right\}}_{i=1}^{p}$$\end{document}(Xi,Mi)i=1p. For RS, each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${C}_{i}$$\end{document}Ci is assumed to a rectangular cutout of the same size. Notably, in this work, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}_{i},{M}_{i})$$\end{document}(Xi,Mi) is still the size of the full image, with a rectangular cutout and the remaining values as zero. Since you want small pieces, remove all the unmasked values to obtain \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}_{i}^{{\prime} },{M}_{i}^{{\prime} })$$\end{document}(Xi′,Mi′) which resemble uniformly sized rectangular pieces of the original images. The resulting image size is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left|{X}_{i}^{{\prime} }\right|=\frac{{{|X}}_{i}|}{p}$$\end{document}Xi′=|Xi|p. Soft labels Now that we have cutout pieces from the image and mask, it is necessary to assign soft labels to each piece. We assume that there are KK classes. To represent this in the form of soft labels, we first conduct 1-hot encoding for each label. Given original label YY, the new label takes form: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}^{{\prime} }=\left[{{\mathbb{1}}}_{Y=1},\ldots ,{{\mathbb{1}}}_{Y=K}\right]$$\end{document}Y′=1Y=1,…,1Y=K. It has a single value taking 1 and the rest taking 0. Given piece \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${(X{\prime} }_{i},{M{\prime} }_{i})$$\end{document}(X′i,M′i) the new soft label takes form \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}_{i}^{{\prime} }={[Y}_{i1}^{{\prime} },\ldots ,{Y}_{{iK}}^{{\prime} }]=\frac{\parallel {M}_{i}{\parallel }_{\mathrm{1,1}}}{{\sum }_{l=1}^{p}\parallel {M}_{l}{\parallel }_{\mathrm{1,1}}}\times \left({Y}^{{\prime} }\right)$$\end{document}Yi′=[Yi1′,…,YiK′]=∥Mi∥1,1∑l=1p∥Ml∥1,1×Y′ such that \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\sum }_{i=1}^{p}{Y}_{i}^{{\prime} }={Y}^{{\prime} }$$\end{document}∑i=1pYi′=Y′. Intuitively, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}_{i}^{{\prime} }$$\end{document}Yi′ scales the 1-hot encoding for the piece \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${(X{\prime} }_{i},{M{\prime} }_{i})$$\end{document}(X′i,M′i) based on the ratio of masked pixels in the piece compared with the entire image. Each datapoint(X,M,Y)(X,M,Y) is transformed to become \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\left\{\left({X{\prime} }_{i},{M{\prime} }_{i},{Y}_{i}^{{\prime} }\right)\right\}}_{i=1}^{p}$$\end{document}X′i,M′i,Yi′i=1p. Shuffle step Given the original training set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{({X}^{\left(j\right)},{M}^{\left(j\right)},{Y}^{\left(j\right)}){\}}_{j\in \left[m\right]}$$\end{document}{(Xj,Mj,Yj)}j∈m, after splitting each datapoint into pieces, you obtain the following “bag” of puzzle pieces \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{({X{\prime} }_{i}^{\left(j\right)},{M{\prime} }_{i}^{\left(j\right)},{Y{\prime} }_{i}^{\left(j\right)}){\}}_{j\in \left[m\right],i\in [p]}$$\end{document}{(X′ij,M′ij,Y′ij)}j∈m,i∈[p]. These pieces are then shuffled randomly to obtain new datapoints. A new datapoint \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}^{* },{M}^{* },{Y}^{* })$$\end{document}(X*,M*,Y*) is generated based on the following algorithm:Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }=0.$$\end{document}Yˆ*=0.For each i∈ [p]i∈[p], sample without replacement \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\sim {unif}(\left[m\right])$$\end{document}j∼unif(m) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l\sim {unif}(\left[p\right])$$\end{document}l∼unif(p).Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]={{X}_{l}^{{\prime} }}^{(j)}$$\end{document}X*Ci=Xl′(j), where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]$$\end{document}X*Ci is the standard submatrix replacement notation.Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${M}^{* }\left[{C}_{i}\right]={{M}_{l}^{{\prime} }}^{(j)}$$\end{document}M*Ci=Ml′(j).Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }={\hat{Y}}^{* }+{{Y}_{l}^{{\prime} }}^{\left(j\right)}$$\end{document}Yˆ*=Yˆ*+Yl′j.Set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}^{* }=[\parallel {\hat{Y}}^{* }{\parallel }_{1};{\hat{Y}}^{* }]$$\end{document}Y*=[∥Yˆ*∥1;Yˆ*].Return \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}^{* },{M}^{* },{Y}^{* })$$\end{document}(X*,M*,Y*) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left|{X}^{* }\right|=\left|{M}^{* }\right|={|X|}$$\end{document}X*=M*=|X|. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }=0.$$\end{document}Yˆ*=0. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }=0.$$\end{document}Yˆ*=0. For each i∈ [p]i∈[p], sample without replacement \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\sim {unif}(\left[m\right])$$\end{document}j∼unif(m) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l\sim {unif}(\left[p\right])$$\end{document}l∼unif(p).Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]={{X}_{l}^{{\prime} }}^{(j)}$$\end{document}X*Ci=Xl′(j), where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]$$\end{document}X*Ci is the standard submatrix replacement notation.Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${M}^{* }\left[{C}_{i}\right]={{M}_{l}^{{\prime} }}^{(j)}$$\end{document}M*Ci=Ml′(j).Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }={\hat{Y}}^{* }+{{Y}_{l}^{{\prime} }}^{\left(j\right)}$$\end{document}Yˆ*=Yˆ*+Yl′j. For each i∈ [p]i∈[p], sample without replacement \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\sim {unif}(\left[m\right])$$\end{document}j∼unif(m) and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l\sim {unif}(\left[p\right])$$\end{document}l∼unif(p).Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]={{X}_{l}^{{\prime} }}^{(j)}$$\end{document}X*Ci=Xl′(j), where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]$$\end{document}X*Ci is the standard submatrix replacement notation.Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${M}^{* }\left[{C}_{i}\right]={{M}_{l}^{{\prime} }}^{(j)}$$\end{document}M*Ci=Ml′(j).Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }={\hat{Y}}^{* }+{{Y}_{l}^{{\prime} }}^{\left(j\right)}$$\end{document}Yˆ*=Yˆ*+Yl′j. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]={{X}_{l}^{{\prime} }}^{(j)}$$\end{document}X*Ci=Xl′(j), where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]$$\end{document}X*Ci is the standard submatrix replacement notation. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]={{X}_{l}^{{\prime} }}^{(j)}$$\end{document}X*Ci=Xl′(j), where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }\left[{C}_{i}\right]$$\end{document}X*Ci is the standard submatrix replacement notation. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${M}^{* }\left[{C}_{i}\right]={{M}_{l}^{{\prime} }}^{(j)}$$\end{document}M*Ci=Ml′(j). Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${M}^{* }\left[{C}_{i}\right]={{M}_{l}^{{\prime} }}^{(j)}$$\end{document}M*Ci=Ml′(j). Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }={\hat{Y}}^{* }+{{Y}_{l}^{{\prime} }}^{\left(j\right)}$$\end{document}Yˆ*=Yˆ*+Yl′j. Let \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }={\hat{Y}}^{* }+{{Y}_{l}^{{\prime} }}^{\left(j\right)}$$\end{document}Yˆ*=Yˆ*+Yl′j. Set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}^{* }=[\parallel {\hat{Y}}^{* }{\parallel }_{1};{\hat{Y}}^{* }]$$\end{document}Y*=[∥Yˆ*∥1;Yˆ*]. Set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}^{* }=[\parallel {\hat{Y}}^{* }{\parallel }_{1};{\hat{Y}}^{* }]$$\end{document}Y*=[∥Yˆ*∥1;Yˆ*]. Return \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}^{* },{M}^{* },{Y}^{* })$$\end{document}(X*,M*,Y*) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left|{X}^{* }\right|=\left|{M}^{* }\right|={|X|}$$\end{document}X*=M*=|X|. Return \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}^{* },{M}^{* },{Y}^{* })$$\end{document}(X*,M*,Y*) with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left|{X}^{* }\right|=\left|{M}^{* }\right|={|X|}$$\end{document}X*=M*=|X|. Note in the penultimate step, we include the norm of the label \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{Y}}^{* }$$\end{document}Yˆ* to the new label to inform the neural network how substantial the new image is. Standard notation: in-place shuffle In-place shuffle (IS) represents a significant simplification of RS. IS uses partial shuffling to exchange pieces while preserving their positional relationships. Furthermore, not all pieces are exchanged at random, a fixed proportion of pieces are always kept together to form a framework. For IS, we start with training samples of the form (X,Y)(X,Y) which are the image and label respectively. Note that, unlike RS, there is no mask needed for IS. The training set contains mm datapoints: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{({X}^{\left(j\right)},{Y}^{\left(j\right)}){\}}_{j\in \left[m\right]}$$\end{document}{(Xj,Yj)}j∈m. Once again, cutouts are generated following the same process as RS to obtain \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{{C}_{1},\ldots ,{C}_{p}\}$$\end{document}{C1,…,Cp}. Pieces are then generated by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left\{{X}_{1},{X}_{2},\ldots ,{X}_{p}\right\}=\left\{{C}_{1},{C}_{2},\ldots ,{C}_{p}\right\}\odot X$$\end{document}X1,X2,…,Xp=C1,C2,…,Cp⊙X to yield \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\left\{{X}_{i}^{\left(j\right)}\right\}}_{i\in \left[p\right],j\in [m]}$$\end{document}Xiji∈p,j∈[m]. When applying IS, since we want the pieces to stay in their original position, there is no need to remove the masked segments of the pieces. Furthermore, for each piece, the label remains the same as the original datapoint. The final “bag” of pieces takes form: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\left\{{X}_{i}^{\left(j\right)},{Y}^{(j)}\right\}}_{i\in \left[p\right],j\in [m]}$$\end{document}Xij,Y(j)i∈p,j∈[m]. Before shuffling, the user first specifies the number of pieces that are kept from the original image. This is specified by the shuffle count \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu \in [p]$$\end{document}μ∈[p]. As a result, a total ratio of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda =\frac{\mu }{p}$$\end{document}λ=μp of the original image remains unshuffled and fixed in place. Shuffle step Given a value μμ, we sample without replacement μμ terms out of the set [p][p]. We refer to these as the background index set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b=\{{c}_{1},\ldots ,{c}_{\mu }\}$$\end{document}b={c1,…,cμ}. The background cutout is the cutout comprising of these indices: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C={\sum }_{i=1}^{\mu }{C}_{{c}_{i}}$$\end{document}C=∑i=1μCci. New images are then generated as follows:First sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\sim {unif}\left(\left[m\right]\right)$$\end{document}j∼unifm to obtain a background sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}^{\left(j\right)},{Y}^{\left(j\right)})$$\end{document}(Xj,Yj).Fix the background image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C\odot {X}^{(j)}$$\end{document}C⊙X(j).For each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}$$\end{document}i∈p∖{c1,…,cμ}, sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${j}_{i}\sim {unif}(\left[m\right])$$\end{document}ji∼unif(m).Final generated image is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }=C\odot {X}^{\left(j\right)}+{\sum }_{i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}}{C}_{i}\odot {X}^{\left({j}_{i}\right)}$$\end{document}X*=C⊙Xj+∑i∈p∖{c1,…,cμ}Ci⊙Xji.Final generated label is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}^{* }=\lambda * {Y}^{\left(j\right)}+\frac{1}{p-\mu }{\sum }_{i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}}{Y}^{{(j}_{i})}$$\end{document}Y*=λ*Yj+1p−μ∑i∈p∖{c1,…,cμ}Y(ji). First sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\sim {unif}\left(\left[m\right]\right)$$\end{document}j∼unifm to obtain a background sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}^{\left(j\right)},{Y}^{\left(j\right)})$$\end{document}(Xj,Yj). First sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j\sim {unif}\left(\left[m\right]\right)$$\end{document}j∼unifm to obtain a background sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$({X}^{\left(j\right)},{Y}^{\left(j\right)})$$\end{document}(Xj,Yj). Fix the background image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C\odot {X}^{(j)}$$\end{document}C⊙X(j). Fix the background image \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C\odot {X}^{(j)}$$\end{document}C⊙X(j). For each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}$$\end{document}i∈p∖{c1,…,cμ}, sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${j}_{i}\sim {unif}(\left[m\right])$$\end{document}ji∼unif(m). For each \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}$$\end{document}i∈p∖{c1,…,cμ}, sample \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${j}_{i}\sim {unif}(\left[m\right])$$\end{document}ji∼unif(m). Final generated image is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }=C\odot {X}^{\left(j\right)}+{\sum }_{i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}}{C}_{i}\odot {X}^{\left({j}_{i}\right)}$$\end{document}X*=C⊙Xj+∑i∈p∖{c1,…,cμ}Ci⊙Xji. Final generated image is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{* }=C\odot {X}^{\left(j\right)}+{\sum }_{i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}}{C}_{i}\odot {X}^{\left({j}_{i}\right)}$$\end{document}X*=C⊙Xj+∑i∈p∖{c1,…,cμ}Ci⊙Xji. Final generated label is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}^{* }=\lambda * {Y}^{\left(j\right)}+\frac{1}{p-\mu }{\sum }_{i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}}{Y}^{{(j}_{i})}$$\end{document}Y*=λ*Yj+1p−μ∑i∈p∖{c1,…,cμ}Y(ji). Final generated label is \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}^{* }=\lambda * {Y}^{\left(j\right)}+\frac{1}{p-\mu }{\sum }_{i\in \left[p\right]\setminus \{{c}_{1},\ldots ,{c}_{\mu }\}}{Y}^{{(j}_{i})}$$\end{document}Y*=λ*Yj+1p−μ∑i∈p∖{c1,…,cμ}Y(ji). Theoretical discussion In this section, we compare the theoretical support of the IS algorithm with other state of the art mixed sample data augmentation methods such as Mixup59 and Cutmix60. We focus on IS since it is the more generalized algorithm. Simplification For the sake of clarity, we consider a special case where we only shuffle two images with each other at a time. Since there are mm samples in the training sample, we would end up with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${m}^{2}$$\end{document}m2 pieces of augmented data. Given a sample pair \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{{X}^{\left(A\right)},{Y}^{\left(A\right)}\}$$\end{document}{XA,YA}, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{{X}^{\left(B\right)},{Y}^{\left(B\right)}\}$$\end{document}{XB,YB} and fixed background cutout CC, we define the augmented data as:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${X}^{(A,B)}=C\odot {X}^{\left(A\right)}+(J-C)\odot {X}^{\left(B\right)},$$\end{document}X(A,B)=C⊙XA+(J−C)⊙XB,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${Y}^{(A,B)}=\lambda {Y}^{\left(A\right)}+(1-\lambda ){Y}^{\left(B\right)},$$\end{document}Y(A,B)=λYA+(1−λ)YB, Loss functions Given the original training set \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\{{(X}^{\left(j\right)},{Y}^{\left(j\right)})\}}_{j=1}^{m}$$\end{document}{(Xj,Yj)}j=1m, we define \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${D}_{X}$$\end{document}DX as the empirical distribution. In accordance with current literature, we define the following loss functions. Given model parameters \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta \in \Theta$$\end{document}θ∈Θ:True Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L\left(\theta \right)={E}_{X,Y\sim {P}_{X,Y}}[l\left(\theta {;X},Y\right)],$$\end{document}Lθ=EX,Y∼PX,Y[lθ;X,Y],Empirical Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${L}_{m}\left(\theta \right)=\frac{1}{m}{\sum }_{j=1}^{m}l\left(\theta ;{X}^{\left(j\right)},{Y}^{\left(j\right)}\right),$$\end{document}Lmθ=1m∑j=1mlθ;Xj,Yj,IS- Empirical Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${L}_{m}^{{IS}}\left(\theta \right)=\frac{1}{{m}^{2}}{\sum }_{i,j=1}^{m}l(\theta ;{X}^{\left(i,j\right)},{Y}^{(i,j)}),$$\end{document}LmISθ=1m2∑i,j=1ml(θ;Xi,j,Y(i,j)), True Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L\left(\theta \right)={E}_{X,Y\sim {P}_{X,Y}}[l\left(\theta {;X},Y\right)],$$\end{document}Lθ=EX,Y∼PX,Y[lθ;X,Y], True Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$L\left(\theta \right)={E}_{X,Y\sim {P}_{X,Y}}[l\left(\theta {;X},Y\right)],$$\end{document}Lθ=EX,Y∼PX,Y[lθ;X,Y], Empirical Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${L}_{m}\left(\theta \right)=\frac{1}{m}{\sum }_{j=1}^{m}l\left(\theta ;{X}^{\left(j\right)},{Y}^{\left(j\right)}\right),$$\end{document}Lmθ=1m∑j=1mlθ;Xj,Yj, Empirical Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${L}_{m}\left(\theta \right)=\frac{1}{m}{\sum }_{j=1}^{m}l\left(\theta ;{X}^{\left(j\right)},{Y}^{\left(j\right)}\right),$$\end{document}Lmθ=1m∑j=1mlθ;Xj,Yj, IS- Empirical Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${L}_{m}^{{IS}}\left(\theta \right)=\frac{1}{{m}^{2}}{\sum }_{i,j=1}^{m}l(\theta ;{X}^{\left(i,j\right)},{Y}^{(i,j)}),$$\end{document}LmISθ=1m2∑i,j=1ml(θ;Xi,j,Y(i,j)), IS- Empirical Loss: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${L}_{m}^{{IS}}\left(\theta \right)=\frac{1}{{m}^{2}}{\sum }_{i,j=1}^{m}l(\theta ;{X}^{\left(i,j\right)},{Y}^{(i,j)}),$$\end{document}LmISθ=1m2∑i,j=1ml(θ;Xi,j,Y(i,j)), where ll is the loss function. Given the pathological problems we are faced with, we consider the following class of loss functions:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal L} =\{l|l({\rm{\theta }},x,y)=h({f}_{\theta }(x))-y{f}_{\theta }(x);for\,twice\,differentiable\,h\},$$\end{document}ℒ={l|l(θ,x,y)=h(fθ(x))−yfθ(x);fortwicedifferentiableh},where θθ represents the parameters of the model. This includes the loss functions generated by generalized linear models and cross-entropy. Typically, pathological problems are limited to these cases. Theorem 1. Given the loss function \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l{\mathscr{\in }}{\mathscr{L}}{\mathscr{.}}$$\end{document}l∈L. Assume \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E}_{{r}_{x \sim {D}_{X}}}\left[{r}_{x}\right]=0$$\end{document}Erx~DXrx=0, we can rewrite the IS loss as: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${L}_{m}^{{IS}}\left(\theta \right)={L}_{m}\left(\theta \right)+\mathop{\sum }\limits_{i=1}^{3}\left[\lambda {R}_{i}+\left(1-\lambda \right){\hat{R}}_{i}\right]+{E}_{C}[\lambda {\left(J-C\right)}^{T}\varphi \left(J-C\right)\left(J-C\right)+\left(1-\lambda \right){C}^{T}{\rm{\varphi }}\left({\rm{C}}\right){\rm{C}}],$$\end{document}LmISθ=Lmθ+∑i=13λRi+1−λRˆi+EC[λJ−CTφJ−CJ−C+1−λCTφCC],\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\rm{where}}\begin{array}{c}\mathrm{lim}\\ {\rm{a}}\to 0\end{array}{\rm{\varphi }}({\rm{a}})=0$$\end{document}wherelima→0φ(a)=0. We have:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${R}_{1}\left(\theta \right)=\frac{1}{m}\mathop{\sum }\limits_{i=1}^{m}\left({h}^{{\prime} }\left({f}_{\theta }\left({x}^{\left(i\right)}\right)\right)-{y}^{\left(i\right)}\right)\odot {E}_{{r}_{x} \sim {D}_{X}}\left[{r}_{x}-{x}^{\left(i\right)}\right]{E}_{C}\left[J-C\right],$$\end{document}R1θ=1m∑i=1mh′fθxi−yi⊙Erx~DXrx−xiECJ−C,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${R}_{2}\left(\theta \right)=\frac{1}{2m}\mathop{\sum }\limits_{i=1}^{m}{{h}^{\prime\prime}} \left({f}_{\theta }\left({x}^{\left(i\right)}\right)\right){E}_{C}{\left[J-C\right]}^{T}{E}_{{r}_{x} \sim {D}_{X}}\left[\left.\nabla f\left({x}^{\left(i\right)}\right)\odot \left({r}_{x}-{x}^{\left(i\right)}\right)\right)^{T}\right](J-C),$$\end{document}R2(θ)=12m∑i=1mh′′(fθ(x(i)))EC[J−C]TErx∼DX[∇f(x(i))⊙(rx−x(i)))T](J−C),\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${R}_{3}\left(\theta \right)=\frac{1}{2m}\mathop{\sum }\limits_{i=1}^{m}{E}_{C}{\left(J-C\right)}^{T}{E}_{{r}_{x} \sim {D}_{X}}\left[{\nabla }^{2}{f}_{\theta }\left({x}^{\left(i\right)}\right)\odot \left(\left({r}_{x}-{x}^{\left(i\right)}\right){\left({r}_{x}-{x}^{\left(i\right)}\right)}^{T}\right)\right](J-C),$$\end{document}R3(θ)=12m∑i=1mEC(J−C)TErx∼DX[∇2fθ(x(i))⊙((rx−x(i))(rx−x(i))T)](J−C),\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{R}}_{1}(\theta )=\frac{1}{m}\mathop{\sum }\limits_{i=1}^{m}\left({h}^{{\prime} }\left({f}_{\theta }\left({x}^{\left(i\right)}\right)\right)-{y}^{\left(i\right)}\right)\odot {E}_{{r}_{x} \sim {D}_{X}}\left[{r}_{x}-{x}^{\left(i\right)}\right]{E}_{C}\left[C\right],$$\end{document}Rˆ1(θ)=1m∑i=1mh′fθxi−yi⊙Erx~DXrx−xiECC,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{R}}_{2}\left(\theta \right)=\frac{1}{2m}\mathop{\sum }\limits_{i=1}^{m}{{h}^{\prime\prime}}\left({f}_{\theta }\left({x}^{\left(i\right)}\right)\right){E}_{C}{C}^{T}{E}_{{r}_{x} \sim {D}_{X}}\left[\left.\nabla f\left({x}^{\left(i\right)}\right)\odot \left({r}_{x}-{x}^{\left(i\right)}\right)\right)^{T}\right]C,$$\end{document}R^2(θ)=12m∑i=1mh′′(fθ(x(i)))ECCTErx∼DX[∇f(x(i))⊙(rx−x(i)))T]C,\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\hat{R}}_{3}\left(\theta \right)=\frac{1}{2m}\mathop{\sum }\limits_{i=1}^{m}{E}_{C}{C}^{T}{E}_{{r}_{x} \sim {D}_{X}}\left[{\nabla }^{2}{f}_{\theta }\left({x}^{\left(i\right)}\right)\odot \left(\left({r}_{x}-{x}^{\left(i\right)}\right){\left({r}_{x}-{x}^{\left(i\right)}\right)}^{T}\right)\right]C,$$\end{document}R^3(θ)=12m∑i=1mECCTErx∼DX[∇2fθ(x(i))⊙((rx−x(i))(rx−x(i))T)]C, Proof. For the proof refer to the Supplementary Note 9. Theorem 1 provides a set of theoretical bounds on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${||}{L}_{m}^{{CM}}\left(\theta \right)-{L}_{m}\left(\theta \right){||}$$\end{document}||LmCMθ−Lmθ|| for in-place shuffling. From observation, we find that these bounds match current literature for mixed sample data augmentation experiments. While there is a difference in application, the regularization effects brought about from the use of in-place shuffle largely matches the regularization effects of popular methods such as Mixup and Cutmix. This instils us with further confidence on the effectiveness of our methodology. In general, in-place shuffle does not necessary exceed these methods. However, as the pathoentity shuffling enables better introduction of different pathological features, the regularization effect can be further enhanced in the case of computational pathology specifically, as demonstrated in our experiments. IS can be considered a well-suited alternative to regular mixed sample data augmentation methods specifically for computational pathology problems. Experimental setting The same online data augmentation strategy was designed as preprocessing steps in each experiment. To deal with the data scarcity, the data and mask were processed following the steps: (1) randomly rotation; (2) center-crop (only the square center area with an edge size of 700 pixels was reserved); (3) resize, the cropped area was resized to 384 × 384 pixels; (4) the random vertical and horizontal flip; and (5), for each image, the PyTorch ColorJitter (with the setting of brightness=0.15, contrast = 0.3, saturation = 0.3, and hue = 0.06) was used to increase the sample-wise interference feature varieties during the sampling process. In the validation and test processes, only the (2) and (3) steps were applied, obtaining the center area and resizing it to 384 × 384 pixels. The model and its counterparts were trained for 50 epochs, and the model at the epoch with the highest validation accuracy was saved as the trained model. We report all results on 5-fold cross-validation. The Adam optimizer was utilized57, with a learning rate of 1e-5 and a momentum of 0.05. The cosine learning rate decay was employed to gradually decrease the learning rate twenty times sequentially over the course of the training process. The experiments were fairly done with a same Ubuntu20.04 server (Intel(R) Xeon(R) Platinum 8255 C CPU, 512 GB RAM and 8 Nvidia RTX3090 GPU). The PyTorch version is 1.10.0, the CUDA version is 11.3 and the python version is 3.8. In each experiment, only one GPU was used. The model backbones were built with TIMM58, and for the backbones not included, they were built following the official codes, and further details regarding the parameter settings can be found in the online release. Ethics statement This study was conducted in accordance with the Declaration of Helsinki. The use of the ROSE pathological image dataset for pancreatic cancer AI-assisted diagnosis was reviewed and approved by the Biological and Medical Ethics Committee of Beihang University (approval number: BM20220128; project title: “AI-based precision diagnosis of pancreatic cancer using ROSE pathological images”). All other datasets used in this study were publicly available and de-identified, and were used in accordance with their respective data access policies. Supplementary information Supplementary information Supplementary Video 1 Supplementary information Supplementary Video 1