Two-Step Ensemble Convolutional Neural Networks for Colonoscopic Biopsy Classification Resembling Pathologists’ Process Authors: Hongjun Yoon, Mohammad Rizwan Alam, Nishant Thakur, Sohyun Jeong, Gisu Hwang, Eun Jung Lee, Yosep Chong Published: 2026 ABSTRACT BackgroundColorectal cancer (CRC) is a leading cause of mortality worldwide, and early examination via colonoscopy is increasingly used to prevent CRC mortality. Recently, studies have attempted utilizing artificial intelligence for the classification of CRC. However, datasets were limited in these studies, and the limited number of findings resulting from these studies are not specific to cancerous or noncancerous findings. We present a full pipeline of ensemble deep learning approach to classify five diagnostic categories. Through this pipeline, whole slide images (WSIs) exhibiting low quality can be filtered out before being processed by the computer-aided diagnosis system.MethodsA dataset of 18,922 CRC WSIs collected and labeled as non-tumor, hyperplastic polyp, adenoma, adenocarcinoma, and neuroendocrine tumor (NET, carcinoid). We developed two models: clustering-constrained attention multiple instance learning model was used to classify adenocarcinomas, adenomas, NET, and non-dysplastic lesion classes. EfficientNet then distinguished between non-tumor and hyperplastic polyp for WSIs classified as non-dysplastic lesion class in the first step.ResultsThe proposed ensemble pipeline, which resembles the process of analysis from the pathologists, showed better performance even with multiple classes CRC. The micro-, macro-, and weighted-F1-scores were 86.57%, 83.83%, and 86.86%, respectively. Notably, the NET classification tasks scored an F1-score of 87.18%.ConclusionThe proposed ensemble model addresses common challenges in automated CRC diagnosis, offering reliable solutions with safeguards for exceptional cases. It enhances clinical practice by accurately classifying multiple CRC types. In conclusion, this method is effective where scanned colorectal WSIs can be of low quality for real-world diagnosis. INTRODUCTION Deep learning models have been actively applied in pathological diagnosis and have shown promising results, especially in the tumor classification of common cancers such as prostate, breast, lung, and colorectal cancers (CRCs).12345678 CRC is one of the leading causes of mortality worldwide, and cancer screening using colonoscopy is increasing.91011 Computed tomography colonography and other radiological tests can detect CRC.12 However, colonoscopy is considered the gold standard for CRC detection.1314 During colonoscopy, procedures such as polypectomy or endoscopic mucosal resection can be performed to remove suspicious lesions.1516 Further histopathological examination of the tissue is required for further confirmation; however, this is laborious and time-consuming. Therefore, there is an urgent need for methods that can reduce the cost, time and manual effort of pathologists. Recently, numerous attempts have been made to use artificial intelligence (AI) for the CRC classification.678 As a result, few studies have used deep learning-based algorithms to classify CRC. Hassan et al.17 2017, used a smaller subset of multispectral images to develop a convolutional neural network (CNN) model that can classify three types of tissue, benign hyperplasia, intraepithelial neoplasia, and neoplasia, and observed a high accuracy of 99.17% with segmentation and 79.23% without segmentation. Iizuka et al.6 utilized a large dataset of 4,036 whole slide images (WSIs) of the colon to train a CNN model to distinguish between adenocarcinoma (ADC), adenoma (ADM), and non-neoplastic classes. They were able to achieve an area under the receiver operating characteristic curve of 0.96 for ADC. However, applying deep learning to CRC is very difficult because of many challenges, such as small tissue artifacts, regenerative atypia, and inter-observer variation in ADM grading and subtype classification. Although the results of each study are promising, there are still many limitations, such as the small amount of training and validation data and the lack of validation. In addition, most previous studies have only used a single magnitude or perspective of WSI, which could be a disadvantage as opposed to pathologists. In our previous systematic review, we discussed the limitations of all the existing studies. The performance of all models was acceptable, and some models achieved 99.2% accuracy and 0.99 area under the curve. However, in most studies, tile images were used to train the model, which could result in the loss of diagnostic information of the tissue. The dataset was limited in scope because it utilized tiles instead of WSIs. The unstandardized classification system was another limitation of all the studies, with the datasets categorized into two to six categories: non-tumor (NT), hyperplastic polyps (HYPs), ADM, and ADC.18 WSIs can often be scanned with low quality, which may include artifacts or out-of-focus images, making accurate diagnosis challenging. In this study, we designed a novel deep learning-based pipeline consisting of several models to classify multiple diagnostic categories of CRC precisely. We also propose a filtering methodology to distinguish poor-quality scanned images prior to classification using a trained model for more efficient use in practice. To evaluate the proposed pipeline thoroughly, we developed a model using 18,922 WSI datasets for training and validation. Notably, in this method, we added “neuroendocrine tumor (NET [carcinoid]),” which, to our knowledge, had not been extensively studied in previous research. METHODS This study utilized colorectal biopsy, polypectomy specimens (measuring up to 10 mm in size), and endoscopic mucosal resection samples obtained from patients for research purposes and archived at Yeouido St. Mary’s Hospital between January 2016 and February 2018. Written informed consent was obtained from patients before surgery, and tissue samples used for research purposes. Colonoscopic biopsy tissues were collected from 8,987 patients, resulting in 18,922 WSIs. Each WSI was labeled into one of five diagnostic categories: NT, HYP, ADM, ADC, or NET (carcinoid). Each WSIs was scanned using a Leica Aperio AT2 scanner at a magnification of 20× with hematoxylin and eosin (H&E) staining. Each scan contained one to three tissues, with care taken to avoid tissue damage that could render WSIs undiagnosable, as shown in Supplementary Fig. 1. Two pathologists (E.J.L. and Y.C.) at Yeouido St. Mary’s Hospital labeled each WSI. In cases where there was a discrepancy between the diagnoses of the two pathologists, such samples were excluded. In the presence of multiple classes within the WSIs, the labeling priority was assigned as malignant, benign, or non-dysplastic lesions, following the specific order shown in Supplementary Fig. 2. There were no cases with both ADC and NET (carcinoid) classes. When HYP was found in the NT, it was labeled HYP. This study proposes an ensemble of two CNN-based algorithms for classifying colorectal histology findings in WSIs. Although the clustering-constrained attention multiple instance learning (CLAM) model showed outstanding performance in previous study by Lu et al.19 when examining the potential of the weakly supervised classification model, taking a closer look at each tile with different magnitudes and perspectives is crucial. Hence, we experimented with classifying all classes with tiles using EfficientNet.20 A flow diagram of the classification process for the five diagnostic categories is shown in Fig. 1. Overview of the two separate pipelines to classify five diagnostic categories of colorectal cancer. WSI = whole-slide image, CLAM = clustering-constrained attention multiple instance learning, CNN = convolutional neural network. Tissue segmentation, extraction, and filtering are commonly performed using CLAM and EfficientNet. We improved the original pipeline (Lu et al.19) by adding the blurry and artifact tile filtering model which the performance was shown as Supplementary Table 1. For WSI-level classification, the CLAM model was applied to encode each tile with the pre-trained CNN model as feature vectors, followed by an attention backbone to rank each tile from the WSI for each class. The final prediction was inferred using the clustering layers and a classifier. For tile-level classification, we added two additional steps to reduce the complexity of each tile by highlighting the tissue areas, that is nullifying all background areas. The EfficientNet model was then used to classify each tile and the decision was logically inferred as the order of CRC progression and prioritization from non-dysplastic lesions to benign and malignant neoplasms. Details of the methods, such as common preprocessing steps, tissue segmentation, tile extraction, and AI development, are provided in the Supplementary Methods (Section 2.2-2.6, Supplementary Data 1)1920212223 and also sample results of the preprocessing steps are shown in Supplementary Figs. 3, 4, 5, 6, 7. Experimental setup Our implementation of all the processes, including the preprocessing of images, training, validation, and testing of the CNN model, was written in Python2425 using open-source libraries. In the preprocessing steps, Openslide26 (one of the most popular pathology image handling packages), matplotlib,27 scikit-learn,28 OpenCV29 and Pillow30 (popular image handling packages in Python), and some essential libraries such as Numpy31 and Pandas32 were used. For stain normalization, we used SPArse Modeling Software,33 an optimization toolbox for sparse estimation tasks, to apply the Vahadane stain normalization method.23 For the training, validation, and testing of the CNN model, we used the two most popular deep-learning frameworks, Tensorflow34 and Pytorch.35 The hardware equipment used for the experiment was a server consisting of 48 Intel AND Xeon AND 4214 CPUs @ 2.20GHz, 252GB of RAM, and 8 GeForce RTX 2080 Ti. For the training process of both the CLAM and EfficientNet models, each of the five diagnostic categories findings in the colorectal histology images was divided into training, validation, and test datasets in a ratio of 7:2:1. Each WSI was mutually exclusive from the training, validation, and test datasets, as well as from each representative class. For instance, if a WSI is assigned as a NET (carcinoid) validation dataset, it cannot represent either a training or test dataset or be included in any other histological finding class. The numbers of colorectal histological WSIs and tiles with respect to classes after splitting into training, validation, and test datasets for CLAM and EfficientNet model training are shown in Table 1. Table 1Number and distribution of colorectal histological finding WSIs and tiles based on the classification label and training, validation, and test distribution for the CLAM and EfficientNet modelClassesAllTrainValidationTestWSITileWSITileWSITileWSITileNT4,362 (23.05)17,025 (18.31)3,009 (22.66)11,923 (18.32)901 (23.87)3,481 (18.65)452 (24.21)1,621 (17.54)HYP4,097 (21.65)17,078 (18.36)2,839 (21.38)11,820 (18.16)859 (22.76)3,441 (18.43)399 (21.37)1,817 (19.66)ADM9,705 (51.29)52,840 (56.82)6,878 (51.79)36,997 (56.84)1,876 (49.71)10,633 (56.96)951 (50.94)5,210 (56.39)ADC557 (2.94)4,728 (4.78)403 (3.03)3,394 (5.21)105 (2.78)859 (4.60)49 (2.62)475 (5.14)NET201 (1.06)1,325 (1.42)152 (1.14)956 (1.47)33 (0.87)252 (1.35)16 (0.86)117 (1.27)Total18,92292,99613,28165,0903,77418,6661,8679,240WSI = whole-slide image, CLAM = clustering-constrained attention multiple instance learning, NT = non-tumor, HYP = hyperplastic polyp, ADM = adenoma, ADC = adenocarcinoma, NET = neuroendocrine tumor. Number and distribution of colorectal histological finding WSIs and tiles based on the classification label and training, validation, and test distribution for the CLAM and EfficientNet model WSI = whole-slide image, CLAM = clustering-constrained attention multiple instance learning, NT = non-tumor, HYP = hyperplastic polyp, ADM = adenoma, ADC = adenocarcinoma, NET = neuroendocrine tumor. Performance evaluation and metrics The micro F1-score is a widely used metric in classification tasks owing to its easy calculation and comprehensive performance evaluation across all parameters. However, this metric may be misguided when evaluating an imbalanced dataset is evaluated.36 Our study showed a severely imbalanced dataset in terms of the number of original WSIs, as shown in Table 1. If our model were to predict all NET (carcinoid) classes, a total of 201 WSIs, which is approximately 1.06% of the entire set, to other classes still had 98.94% accuracy. Even if mis-predicting all ADC, 557 WSIs and 2.94% of the entire dataset and NET (carcinoid) classes could still be evaluated with an accuracy of 95.99%. Therefore, it is not only important to evaluate the predicted model or final algorithm to provide an overall performance as if the testing dataset were balanced but also to view each individual class using a set of multiple metrics for objectivity. For individual class performances, the recall, precision, and F1-scores were calculated. The recall measures the ability to predict a positive result from actual positive samples. On the other hand, precision indicates the ability of the model to predict negative classes as positive. The F1-score is the harmonic mean of recall and precision, which calculates the weighted average of precision and recall providing an objective view of the performance of a classifier for an uneven dataset. All the metrics used to evaluate the model performances are summarized in Supplementary Table 2. To provide a metric that could represent the classifier model, we evaluated the model using the conventional micro (or accuracy), macro, and weighted F1-scores. Macro F1-score for the classifier was calculated by taking the average of the F1-score of each class. The weighted F1-score considers the distribution of the dataset by multiplying the number of class samples by each class F1-score divided by the entire number of samples in the dataset. External validation To evaluate the generalizability and robustness of our proposed classification models, we conducted external validation using a colorectal histopathology dataset curated as part of the OPEN AI Dataset Project in Korea. This dataset was collected from six participating institutions: National Cancer Center, Seoul St. Mary’s Hospital, Seoul National University Hospital, Severance Hospital (Sinchon branch), Yonsei University Wonju Severance Christian Hospital, and GC Biopharma (Green Cross Medical Foundation). While the overall project dataset comprises 12,006 WSIs across various organ systems, we selectively used only WSIs labeled as intestinal in origin.37 Furthermore, from this intestinal subset, we included only those WSIs with diagnostic labels that clearly matched our five predefined classes: NT, HYP, ADN, ADC, and NET. This filtering resulted in a total of 1,944 WSIs, distributed as follows: NT (n = 672), HYP (n = 202), ADN (n = 450), ADC (n = 320), and NET (n = 300). All slides were scanned using high-resolution whole slide scanners with H&E staining and were curated to ensure diagnostic quality and consistent representation across institutions. Each WSI in the external dataset underwent the same preprocessing pipeline as the training and internal validation, including tissue segmentation, tile extraction, artifact and blur filtering, CLAHE, background reduction, and stain normalization. The processed tiles were then classified using both the CLAM-based WSI-level model and the tile-level EfficientNet-based CNN ensemble. The final diagnosis per WSI was inferred by integrating the outputs of the two models using the ensemble decision logic, which prioritizes malignant over benign and non-dysplastic lesions. The external validation results were used to assess model performance across unseen institutions and to evaluate class-specific metrics such as recall, precision, and F1-score in a real-world multi-institutional setting. Ethics statement The dataset was modified, split, and augmented for the training and testing. This study was approved by the Institutional Review Board of The Catholic University of Korea, College of Medicine (SC18RNSI0116). Written informed consent was obtained from patients before the surgical procedure, and tissue samples were used for research purposes. RESULTS Experimental dataset After preprocessing all the WSI datasets, 92,996 tiles were extracted from 18,992 WSIs, which was, on average, 4.91 tiles per WSI for the total dataset. For all datasets, the minimum number of tiles extracted from a single WSI was a single tile and the maximum was 52 tiles. The total WSI and tile distribution resulting from the preprocessing are summarized in Table 1. As mentioned in the previous subheading, ‘EfficientNet Training Setup,’ the distribution of tiles concerning the classes is extremely unbalanced. The majority class, ADM, had the highest number of datasets for both WSIs (9,705 WSIs, 51.29%) and tiles (52,840 tiles, 56.82%), representing more than half of the entire dataset. In contrast, the NET (carcinoid) had the lowest number of datasets for both WSIs (201 WSIs, 1.06%) and tiles (1,325 WSIs, 1.42%). CLAM model results For CLAM model training, a variant of MIL models, we experimented with a model based on the same backbone model for single and multiple attention branches with non-weighted and weighted losses. Each variant was trained using the same WSI dataset, as listed in Supplementary Table 3 (the results for all four different configurations for the CLAM configurations are included). Supplementary Table 3 also shows the micro-, macro-, and weighted-F1-scores resulting from the four different configurations of the CLAM model, which clearly shows that the single attention branch with weighted loss applied for the training process performs better than the other variant models in terms of micro- and weighted-F1-scores. The results table shows that a single attention branch with weighted loss exhibited the best performance, with a weighted F1-score of 85.00% (Table 2). Table 2Comparison of the evaluation result using WSI-level classification using the CLAM model and tile-level classification using the EfficientNet-B0 modelModel typeClassesRecall, %Precision, %F1-score, %Micro-F1-score, %Macro-F1-score, %Weighted-F1-score, %WSI-level classificationNT 86.19 73.71 79.47 84.83 82.22 85.00 HYP72.5779.9576.08ADM 89.08 94.43 91.68 ADC 89.47 67.11 76.69 NET 80.95 94.44 87.18 Tile-level classificationNT82.6366.4973.6877.2064.1078.62HYP 80.80 81.41 81.10 ADM73.7494.4682.82ADC70.1830.3042.33NET71.4328.3040.54Bold text represents better performance when comparing the results.WSI = whole-slide image, CLAM = clustering-constrained attention multiple instance learning, NT = non-tumor, HYP = hyperplastic polyp, ADM = adenoma, ADC = adenocarcinoma, NET = neuroendocrine tumor. Comparison of the evaluation result using WSI-level classification using the CLAM model and tile-level classification using the EfficientNet-B0 model Bold text represents better performance when comparing the results. WSI = whole-slide image, CLAM = clustering-constrained attention multiple instance learning, NT = non-tumor, HYP = hyperplastic polyp, ADM = adenoma, ADC = adenocarcinoma, NET = neuroendocrine tumor. EfficientNet model results For EfficientNet model training, we used the EfficientNet-B0 model, which was the base model, owing to the memory restrictions of our hardware resources. More complex models can be used; however, there is a trade-off in reducing the batch size in the training process, which may result in overfitting and lower performance. We tested three types of datasets. First, we used the original tile images to which preprocessing methods were applied but with a weighted loss. The loss values were calculated using the reciprocal of the number for each class. Second, we applied combined undersampling and oversampling methods to the original tile dataset to balance each class. Finally, only the oversampling method was applied to the level-imbalanced datasets. The results for the EfficientNet-B0 training dataset are listed in Supplementary Table 4. We compared the performance metrics in terms of the WSI level because the diagnostic result of the classification model should return the predicted class of the WSI, and it is only possible to compare the performance with the CLAM model. The preprocessing methods using the tile-level dataset with only the over-sampling method (Supplementary Table 4) showed the best performance, with scores of 77.20%, 64.10%, and 78.62% for the micro-, macro-, and weighted-F1-scores, respectively. The evaluation results for the best-performing model, which was trained with the oversampled dataset, are presented in Table 2 with a confusion matrix and corresponding metrics. The classes assigned as malignant (ADC and NET [carcinoid]) were relatively low compared to non-dysplastic (NT and HYP) and benign lesions (ADM). Ensemble algorithm prediction results When comparing the results of the CLAM and EfficientNet models for the test dataset, the CLAM showed significantly better performance in classifying benign (ADM) and malignant (ADC and NET (carcinoid)) individual classes and non-dysplastic lesion findings as a group. However, EfficientNet performed better in classifying the HYP class than CLAM. Moreover, when we narrowed down the classification model to predict only NT and HYP, the EfficientNet model (Table 3) performed better than the CLAM model (Table 3). Therefore, we applied an ensemble of two CNN model pipeline approaches to classify the ADC, ADM, and NET (carcinoid) classes, and if NT or HYP classes were predicted using CLAM, we then utilized the tile-level inference approach by obtaining inference results from EfficientNet. Evaluating the WSI-level model, CLAM, with a confusion matrix by four classes, is shown in Table 3, and the binary classification for NT and HYP of the selected EfficientNet model is shown in Table 3. Table 3Evaluation result of CLAM model distinguishing four different classes (non-dysplastic lesion [NT, HYP], ADM, ADC, NET [carcinoid]) and EfficientNet model differentiating NT and HYP, which the classes are shown to be well classifying each target classes in Table 2 VariablesModel typeCLAMEfficientNetWSI-level classificationWSI-level classificationTile-level classificationClassesNT + HYPADMADCNETNTHYPNTHYPRecall, %93.3489.6690.2777.5086.2573.5794.4087.10Precision, %89.6294.1172.3493.9473.9686.0188.5493.64F1-score, %91.4491.8380.3184.9379.6379.3091.3890.25Micro-F1-score, %91.2079.4790.85Macro-F1-score, %87.1379.4790.85Weighted-F1-score, %91.2479.4590.83CLAM = clustering-constrained attention multiple instance learning, NT = non-tumor, HYP = hyperplastic polyp, ADM = adenoma, ADC = adenocarcinoma, NET = neuroendocrine tumor, WSI = whole-slide image. Evaluation result of CLAM model distinguishing four different classes (non-dysplastic lesion [NT, HYP], ADM, ADC, NET [carcinoid]) and EfficientNet model differentiating NT and HYP, which the classes are shown to be well classifying each target classes in Table 2 CLAM = clustering-constrained attention multiple instance learning, NT = non-tumor, HYP = hyperplastic polyp, ADM = adenoma, ADC = adenocarcinoma, NET = neuroendocrine tumor, WSI = whole-slide image. Using the trained CLAM model, the test dataset was re-evaluated by relabeling the ground truth of the NT and HYP classes as non-dysplastic lesions. Combining the NT and HYP classes for evaluation significantly increased the micro-, macro-, and weighted-F1-scores significantly to 91.72% (+7.94%), 86.03% (+5.23%), and 90.76% (+6.95%), respectively. Because the tile-level classification model was only used to distinguish between the two classes, we retrained the EfficientNet-B0 model to output only NT and HYP. The dataset used for this process is the same as that listed in Table 1, except for the unnecessary classes, which are ADM, ADC, and NET (carcinoid). Using the trained EfficientNet-B0 model, the test dataset was evaluated for only the NT and HYP binary classes. Because other benign and malignant classes were filtered out using the CLAM model, the performances of the two classes increased significantly to 91.38% (+15.98%) and 90.25% (+15.74%) for the F1-scores of NT and HYP, respectively, shown in Table 3. By applying the ensemble classification algorithm, which first classifies ADM, ADC, NET (carcinoid), and non-dysplastic lesions using the CLAM model and then classifies NT and HYP using the EfficientNet-B0 model, we evaluated the five diagnostic categories in Supplementary Table 5. The results showed that the micro-, macro-, and weighted F1-score were 86.57%, 83.83%, and 86.86%, respectively. Compared to the CLAM-only results shown in Table 1, the micro-, macro-, and weighted F1-scores increased by 1.74%, 1.61%, and 1.86%, respectively. Using the same model that demonstrated the performance shown in Supplementary Table 4, we also evaluated the model according to its groups, which were non-dysplastic, benign, and malignant lesions, as shown in Supplementary Table 4. Classifying each WSI into groups showed much better performance, with 90.10%, 87.78%, and 90.12% for micro-, macro-, and weighted-F1-scores, respectively. Using the same model that demonstrated the performance shown in Supplementary Table 4, we evaluated the model using the negative and positive findings in Supplementary Table 4. The negative classes consisted of the NT and HYP classes, essentially the same as those of the non-dysplastic lesion class. The positive classes included the ADM, ADC, and NET (carcinoid) classes. Classifying each WSI into negative and positive classes showed a slightly better performance for the macro F1-score (90.80%). The micro- and weighted-F1-scores were shown as 90.94% and 90.92%, respectively. From the trained EfficientNet-B0, which is used for the tile-level inference model, we obtained GradCAM, which signifies important regions for the model to classify each image as one class. Each image in Fig. 2 shows the gradient class activation map; hence, the term GradCAM was used for each inferred class or finding. Sample GradCAM results shown by the trained classification models. WSI = whole-slide image, HYP = hyperplastic polyp, ADM = adenoma, NET = neuroendocrine tumor, ADC = adenocarcinoma. From the trained classification models, both CLAM and EfficientNet, we can obtain the gradient class activation map, that is, the GradCAM visualization results, which signify the important regions for the models to classify each image. Each image in Fig. 2 shows GradCAM overlaid onto the original image for each inferred finding. HYP results showed a high probability of marking lumens with saw-like shapes. Fig. 2B shows that the ADM tissue has a high probability of ADM even when the area seems to have undergone cauterization. Fig. 2A and B also showed a good distinction between HYP and ADM, where the upper part of the tissue could be confusing. The ADC results showed well-distinguished areas by displaying a high probability of in-situ ADC areas as Fig. 2D. External validation results The ensemble model demonstrated strong performance across all diagnostic categories, achieving a micro-F1-score of 95.32%, macro-F1-score of 95.28%, and weighted-F1-score of 95.31%. Class-specific performance metrics showed high concordance between predicted and ground truth labels, particularly for NET (F1-score: 99.83%), ADC (F1-score: 97.55%), and NT (F1-score: 94.84%). The confusion matrix (Table 4) shows minimal misclassification between diagnostic categories, further supporting the robustness of the proposed classification framework. Notably, F1-score for all classes exceeded 90%, with ADC and NET reaching 97.55% and 99.83%, respectively. Table 4External validation results of two-step ensemble pipeline model using tile and whole-slide image level classificationModel typeClassesRecall, %Precision, %F1-score, %Micro-F1-score, %Macro-F1-score, %Weighted-F1-score, %Ensemble modelNT91.6798.2594.8495.3295.2895.31HYP87.1394.6290.72ADM98.6788.8093.47ADC99.3895.7897.55NET99.67100.0099.83NT = non-tumor, HYP = hyperplastic polyp, ADM = adenoma, ADC = adenocarcinoma, NET = neuroendocrine tumor. External validation results of two-step ensemble pipeline model using tile and whole-slide image level classification NT = non-tumor, HYP = hyperplastic polyp, ADM = adenoma, ADC = adenocarcinoma, NET = neuroendocrine tumor. These results validate the applicability of the trained ensemble model across institutions and suggest that it is well-suited for multi-center deployment in colorectal histopathology analysis. Detailed per-class performance and confusion matrix results are summarized in Table 4. DISCUSSION In this study, we proposed an ensemble pipeline to classify five diagnostic categories of CRC based on a CNN. The full pipeline comprises two methods. The first method classifies ADM, ADC, NET (carcinoid), and non-dysplastic lesions into subgroups using the CLAM model, which predicts the class at the WSI level. The second method classifies non-dysplastic lesion classes into more specific findings, such as NT and HYP, using the EfficientNet model, which predicts the class at the tile level. To train and evaluate each model, 18,922 colorectal WSIs from 15,187 patients were used. To the best of our knowledge, this is the first study on the automated classification of CRC, including one of the five diagnostic categories of CRC, NET (carcinoid), which comprises only approximately 0.49% of all CRC malignancies. We found that the classification of benign and malignant classes (ADM, ADC, and NET [carcinoid]) performed better using the WSI-level model, whereas non-dysplastic lesion classes (NT and HYP) performed better using the tile-level model. Although multiple-instance learning considers the local features of WSI, the model’s prediction is based on the aggregation of inference results from individual tiles. Therefore, we can infer that the non-dysplastic lesion classes require more attention to the detailed features of the tissues, specifically, the information related to structural abnormalities, for the NT and HYP classes. Conversely, significant features distinguishing the ADM, ADC, and NET (carcinoid) classes can be found as tumor changes in the mucosa and submucosa, which can be observed at the WSI level. However, a problem with the tile-level model is that some of the tiles extracted from the WSI are unsuitable for cancer-type classification. Many tiles mispredicted that the tile-level classification model would include images not part of the tissue or small artifacts. To overcome this issue, we utilized the Laplacian filter thresholding method to exclude blurry tiles and included another EfficientNet model trained to distinguish between normal tissue and artifact-included tiles. In addition, to maximize the performance of the tile-level classification model, we applied a background reduction technique by first identifying the area identified as a tissue area and blackening all other unnecessary areas. Through these additional image-processing techniques and models, the proposed method can be applied in clinical practice to address low-quality WSIs or areas that could be considered exception cases when extracted as tiles. Moreover, this technique can be used as a feature to guide pathologists or decision-making agents in considering tissue rescanning by alerting them with a warning message. A post hoc pathologist-level error analysis was conducted to understand misclassification patterns, particularly for NET. We found that most NET misclassifications occurred as ADC or ADN, especially in small biopsy specimens with limited morphological context. In some instances, NETs exhibited pseudo-glandular architecture or eosinophilic cytoplasm, which overlapped visually with well-differentiated ADC. Furthermore, the submucosal location of NETs in fragmented specimens often complicated classification when lamina propria or architectural features were absent. These findings suggest that certain misclassifications stemmed from sampling artifacts or ambiguous morphologic presentation. This study’s results show that incorporating both WSI and tile-level inference models as an ensemble model can increase performance in classifying the five diagnostic categories of different types of classification models. As shown in Table 2 and Supplementary Table 5, the CLAM-based model performed well in classifying the five diagnostic categories different types of findings in the WSI. Combining the CLAM and EfficientNet models as an ensemble pipeline (Supplementary Table 5), the F1-scores increased by 1.74%, 1.61%, and 1.86% for micro, macro, and weighted scores, respectively. The relatively low precision of the ADC and CAR classes in the tile-level EfficientNet model raises valid concerns regarding potential false positives. However, the final ensemble approach mitigates this issue by prioritizing WSI-level classification from the CLAM model, which achieved markedly higher precision and recall for malignant categories. Additionally, we applied weighted loss functions and class balancing techniques to counteract training bias due to class imbalance. Threshold tuning was also explored but showed limited effect compared to the ensemble’s complementary strengths. Clinically, false positives in high-risk classes such as ADC or NET warrant caution, and our pipeline is designed to trigger secondary review in such cases. Importantly, misclassifications involving malignant findings tended to err toward other neoplastic categories rather than non-neoplastic tissue, thus limiting the risk of missed diagnoses. In addition to the performance evaluated when the models were used to predict each finding individually, we evaluated the ensemble model when predicting the classes by risk group (non-dysplastic lesion, benign, and malignant) and binary classification (non-dysplastic lesion and abnormal). Our model showed 91.05%, 87.43%, and 91.10% for micro-, macro-, and weighted-F1-scores, respectively. For binary classification, our model yielded 92.26%, 92.20%, and 92.27%, respectively. Prior research has yielded several AI models for CRC focused on detecting polyps, neoplasia, and ADM through endoscopic diagnosis, and eight studies regarding tumor classification have been summarized in a survey by Thakur et al.18 As the study’s author noted, comparing each result is nearly impossible and meaningless because of the differences in focus and characteristics. However, some major differences are notable in comparison to previous studies. The first difference can be found in the dataset; we used 18,922 WSIs, which is far larger than any other study for the tumor classification of CRC. In addition, we added another subtype, NET (carcinoid), which has not been introduced in prior research. NET (carcinoid), which comprise only approximately 0.49% of all CRC malignancies, can be considered unique to all the other subtypes in prior research, as they can be found in the submucosal region instead of the mucosal region. Furthermore, all studies showed good performance with respect to tile prediction, except for Iizuka et al.,6 who showed a discrepancy when applied to real-world applications. Moreover, differences in the model architecture are also present in our study. All previous studies used CNN models with respect to tiles. This is a limitation in the domain of histopathological images or any other computer vision task when using CNN models for very high-dimensional images. With the introduction of CLAM by Lu et al.,19 high-dimensional image inferencing and diagnosis have shown promising results with more efficiency. Although CLAM WSI-level prediction appears to have outperformed the traditional tile-level CNN inference mechanism, our results showed that tile- and WSI-level predictions must be thoroughly compared. As pathologists go through the diagnosis process, the WSI-level prediction model first locates areas with deformities to distinguish ADM and ADC in glandular areas and NET (carcinoid) in submucosal areas. If none of these were spotted, a general tile-level prediction model scanned each tile to detect deformities in the lumen of HYP. AI is being increasingly used to develop models for cancer diagnosis, histopathological classification, and other applications.18383940 Several AI models have been developed for CRC, focusing on gene identification, biomarker screening, image grading, microsatellite instability (MSI) and survival prediction.18414243 MSI serves as a prognostic marker, and its detection aids in selecting immunotherapy for patients.4445 Numerous studies have demonstrated that AI models can detect various gene mutations in CRC histological images with promising results.4647 Regarding CRC, multiple AI models for detecting gene mutations, including BRAF, KRAS, APC, CTNNB1, ARID1A, and P53, have been developed and shown promising results.4748 Identifying gene mutations in patients with cancer helps in choosing treatments and is also used as a prognostic marker.4950 Although AI models have shown promising results in gene mutations and MSI for CRC, their routine use in clinical practice requires further validation.4347 In summary, compared with previous studies, this study is the first to study the potential for classifying NET (carcinoid) using a large number of WSI datasets extensively. In addition, to the best of our knowledge, this is the first time the CLAM model has been examined for application in CRC multi-class classification tasks. Study of Kather et al.,8 who performed external validation using public datasets, in this study we also conducted external validation. In addition to the internal evaluation, we conducted external validation using the publicly available colorectal histopathology dataset from the OPEN AI Dataset Project in Korea to assess the generalizability of our ensemble model. Interestingly, the external validation yielded even higher performance than the internal test set, with a macro-F1-score of 95.28%, micro-F1-score of 95.32%, and weighted-F1-score of 95.31%. Per-class analysis revealed strong performance across all categories, including rare classes such as NET (carcinoid), which achieved an F1-score of 99.83%. This improvement in external performance can be attributed to the nature of our internal training and testing dataset, which deliberately included a substantial number of diagnostically ambiguous or borderline cases. During data curation and annotation, we incorporated difficult-to-classify WSIs near inter-class boundaries—such as ADMs with focal high-grade dysplasia mimicking ADC, or HYPs with atypical architecture overlapping with non-specific inflammation—to improve the discriminative capacity of the model. While this strategy may have introduced higher uncertainty in internal evaluations, it enhanced the model’s capacity to generalize across clear-cut cases encountered in external datasets. Therefore, the improved results on the external dataset may reflect a model trained under more challenging conditions, ultimately becoming more robust when applied to routine clinical samples. Moreover, the consistent performance across classes, including the rare NET subtype, demonstrates that the model is not only accurate but also reliable in real-world settings. These findings underscore the clinical potential of the proposed ensemble pipeline as a generalizable tool for multi-class colorectal histopathology classification across institutions. In conclusion, the proposed ensemble model addresses common challenges in automated CRC diagnosis, offering reliable solutions with safeguards for exceptional cases. It enhances clinical practice by accurately classifying multiple diagnostic categories of CRC, enabling better patient follow-up. Although the current model focuses on specific diagnostic categories of CRC, future improvements with larger datasets could expand its classification capabilities to include more intermediate cancer types.