Transformer-Based Wireless Capsule Endoscopy Bleeding Tissue Detection and Classification
An Auto-WCEBleedGen Version 1 Challenge Report By KU Researchers · 2024
Read Transformer-Based Wireless Capsule Endoscopy Bleeding Tissue Detection and Classification by An Auto-WCEBleedGen Version 1 Challenge Report By KU Researchers (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19218.txt.
Introduction
Basit Alawode and co-authors built a computer model that finds bleeding spots in video from a tiny camera pill that patients swallow. The camera travels through the gut and sends back many frames. Doctors need help spotting the few frames that show blood. The report dates to 2024 and covers a public contest entry.
One model does both jobs at once: it draws boxes around suspect areas and labels each frame. It pairs a standard image feature tool with a transformer that reads the whole scene. On contest test frames, its scores were high for both tasks. The entry placed third, and the code was shared in public for others to use.
Catalog record
| CREATOR | An Auto-WCEBleedGen Version 1 Challenge Report By KU Researchers |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 11 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19218
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transformer-Based Wireless Capsule Endoscopy Bleeding Tissue Detection and Classification An Auto-WCEBleedGen Version 1 Challenge Report By KU Researchers Abstract Informed by the success of the transformer model in various computer vision tasks, we design an end-to-end trainable model for the automatic detection and classification of bleeding and non-bleeding frames extracted from Wireless Capsule Endoscopy (WCE) videos. Based on the DETR model, our model uses the Resnet50 for feature extraction, the transformer encoder-decoder for bleeding and non-bleeding region detection, and a feedforward neural network for classification. Trained in an end-to-end approach on the Auto-WCEBleedGen Version 1 challenge training set, our model performs both detection and classification tasks as a single unit. Our model achieves an accuracy, recall, and F1-score classification percentage score of 98.28, 96.79, and 98.37 respectively on the Auto-WCEBleedGen version 1 validation set. Further, we record an average precision (AP @ 0.5), mean-average precision (mAP) of 0.7447 and 0.7328 detection results. This earned us a 3rd place position in the challenge . Our code is publicly available via https://github.com/BasitAlawode/WCEBleedGen. 1 Introduction Wireless Capsule Endoscopy (WCE) is a cutting-edge, non-invasive technique for diagnosing small bowel diseases. Its ability to provide comprehensive visualization of the gastrointestinal tract has revolutionized clinical diagnostics. However, the manual frame-by-frame analysis of the vast amount of data collected during WCE procedures poses significant challenges in terms of time and effort, creating a bottleneck for timely and accurate diagnosis. Artificial Intelligence (AI), particularly its subfield of Deep Learning (DL), offers promising solutions to overcome these limitations. By enabling descriptive, predictive, and prescriptive analyses, AI facilitates the extraction of valuable insights from complex data, which are otherwise unattainable through manual methods. The integration of AI into WCE analysis has the potential to enhance diagnostic accuracy while significantly reducing the workload of medical professionals. In this work, we present a novel approach based on the Detection Transformer (DETR) architecture to detect and classify frames in WCE data into bleeding and non-bleeding categories. Our model utilizes ResNet50 for feature extraction, a transformer encoder-decoder for detecting bleeding and non-bleeding regions, and a feedforward neural network for classification. Trained end-to-end on the Auto-WCEBleedGen Version 1 challenge training set, the proposed model integrates detection and classification into a unified framework, performing both tasks simultaneously with high efficiency. This approach marks a significant step forward in automating WCE data analysis, minimizing manual effort, and improving diagnostic precision. 2 Method Informed by the success of the transformer model in various computer vision tasks, we design an end-to-end trainable model for the automatic detection and classification of bleeding and non-bleeding frames extracted from Wireless Capsule Endoscopy (WCE) videos, as shown in Figure . Our model follows the DETR architecture [ 1 ] with a few structural adjustments to the task at hand. Given a tissue frame or image, we utilized ResNet50 [ 2 ] to learn the 2D representation of the tissue (feature extraction). The model then flattens it and adds positional encoding before passing it into a transformer encoder. A transformer decoder then takes as input a small fixed number of learned positional embeddings, called object queries, and additionally attends to the encoder output. The output embedding of the decoder is then passed to a shared feedforward network (FFN) that predicts either a detection (bleed/non-bleed and bounding box) or a “background” class. Figure 1: Pipeline of our end-to-end trainable single-model for WCE bleeding tissue classification and detection. The DETR architecture outputs 80 classes. However, to adapt this for our purpose of classifying bleeding and non-bleeding tissues, we append to, to the output of DETR, a single layer FFN with 3 neurons to predict the probability of the tissue image belonging to the 3 classes of bleed, non-bleed, or background. It should be noted that more than one region of a frame can be classified. Additionally, our model outputs the bounding boxes of the regions detected as bleeding or non-bleeding alongside the classification probability. A frame is finally classified as bleeding if at least one region in such frame has been classified as bleeding with a probability greater than 0.5. During inference, the bounding boxes of all the regions classified as bleeding are then superimposed on the frame for visualization purposes. 2.1 The Training and Test Data The training dataset consists of 2618 frames split equally and classified into bleeding (1309 frames) and non-bleeding (1309 frames) WCE frames collected from multiple internet resources, datasets with a wide variety and types of gastrointestinal (GI) bleeding throughout the GI tract, along with medically validated binary masks and bounding boxes in three formats (txt, XML, and YOLO txt). To facilitate our training, we further split the overall training set into train and validation subsets in the ratio 80:20. Both classes are equally represented in the two subsets, resulting in a balanced split. The test dataset is an independently collected WCE data containing bleeding and non-bleeding frames of more than 30 patients suffering from acute, chronic, and occult GI bleeding referred at the Department of Gastroenterology and HNU, All India Institute of Medical Sciences, New Delhi, India. Both datasets remain the property of the Auto-WCEBleedGen Version V1 Challenge. 2.2 Training Details 2.2.1 Data Preprocessing To achieve end-to-end training, we extracted the bounding boxes from the provided dataset XML files and one-hot encoded the classes. These form the output data of our model with the images as the input. To improve the model’s robustness, augmentations such as motion blurring, graying, random brightness contrasting, color jittering, random Gammaing, and normalization were applied. 2.2.2 Loss Function Following the DETR training flow, the bipartite matching loss [ 1 ] , which is a weighted combination of the Hungarian matching algorithm [ 3 ] loss, the cross-entropy loss [ 4 ] , and the IOU loss [ 5 ] , is adopted. The Hungarian matching loss ensures a one-to-one matching between predicted outputs, with the bounding boxes and ground truth annotations enabling end-to-end training without the need for hand-designed components like anchor boxes or non-maximum suppression (NMS). The cross-entropy loss is the classification loss measuring the difference between the predicted probability distribution and the true label distribution, penalizing incorrect predictions more severely as they become more confident. The IOU loss captures the overlap between predicted and ground-truth bounding boxes. For our model, the weights of the Hungarian matching loss, the cross-entropy loss, and the IOU loss are 1, 5, and 2, respectively. 2.2.3 Model Training Transformer-based models are known to perform well in the presence of huge amounts of data. As we have only 2618 images for training and validation, training from scratch is impossible. As such, we utilized the trained DETR model with ResNet50 weights available at [ 6 ] to initialize all the weights of our model except the last FFN layer that was randomly initialized. Our model is then fine-tuned using the training set. The model was fine-tuned for 500 epochs using the AdamW optimizer [ 7 ] with a learning rate of 1 × 10 − 6 1\times 10^{-6} , 1 × 10 − 5 1\times 10^{-5} , and 5 × 10 − 5 5\times 10^{-5} for the ResNet50 backbone, the transformer encoder/decoder, and the final FCN prediction layer, respectively. A weight decay of 1 × 10 − 4 1\times 10^{-4}