FuseCaps: Investigating Feature Fusion Based Framework for Capsule Endoscopy Image Classification
Bidisha Chakraborty, Shree Mitra · 2024
Read FuseCaps: Investigating Feature Fusion Based Framework for Capsule Endoscopy Image Classification by Bidisha Chakraborty, Shree Mitra (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02637.txt.
Description
Abstract In order to improve model accuracy, generalization, and class imbalance issues, this work offers a strong methodology for classifying endoscopic images. We suggest a hybrid feature extraction method that combines convolutional neural networks (CNNs), multi-layer perceptrons (MLPs), and radiomics. Rich, multi-scale feature extraction is made possible by this combination, which captures both deep and handmade representations.
Catalog record
| CREATOR | Bidisha Chakraborty, Shree Mitra |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02637
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FuseCaps: Investigating Feature Fusion Based Framework for Capsule Endoscopy Image Classification Bidisha Chakraborty, Shree Mitra Abstract In order to improve model accuracy, generalization, and class imbalance issues, this work offers a strong methodology for classifying endoscopic images. We suggest a hybrid feature extraction method that combines convolutional neural networks (CNNs), multi-layer perceptrons (MLPs), and radiomics. Rich, multi-scale feature extraction is made possible by this combination, which captures both deep and handmade representations. These features are then used by a classification head to classify diseases, producing a model with higher generalization and accuracy. In this framework we have achieved a validation accuracy of 76.2% in the capsule endoscopy video frame classification task. K eywords Radiomics \cdot Projection Head \cdot Feature Fusion 1 Introduction Endoscopy is a procedure that is used to have a close look at the organs. This is done either to detect the diseases or observe the cellular patterns. Early detection of diseases is important as it helps to reduce the mortality rate and also improve the development of medicines. Here the subset of Machine Learning that is Deep Learning comes into play. Deep Learning has been implemented widely in the medical field to detect gastrointestinal and liver-related diseases. As a result, many models have been developed to classify capsule endoscopy images. Some of them involve the use of Convolution Neural Networks or Transfer Learning. But to ensure that the classification of the images is to the point, we have introduced the combination of Radiomics and Convolution Neural networks to enrich the feature dataset and lastly use that feature dataset to perform classification. 2 Methods In general a wide variety of techniques are used to determine the category of diseases. Some of them are Supervised Learning, Transformation Learning, Convolutional Neural Networks and so on. The above defined techniques have been also combined to enrich the feature dataset. In this we have proposed a methodology that will not only increase the accuracy of the model but will also improve the generalisation of the model and will also handle class imbalance. For extraction of features from the dataset, we have implemented a combination of Radiomics followed by Multi Layer Perceptron and Convolutional Neural Networks. The classification head is used for classification of diseases. 2.1 Radiomics Feature Extraction Radiomics [ 1 ] is a process in which we try to extract quantitative or handcrafted features from the images. This method is particularly useful in the medical field as medical images has wide variety of handcrafted features including shape, texture, density etc and this technique largely helps to derive those features using data characterisation algorithms. The algorithms also makes use of advanced mathematical concepts like Laplacian, Gaussian, Gradient formulas to apply the filters. Radiomics also makes use of advanced statistical methods like Gray Level co-occurrence matrix, Gray Level Run Length matrix, Gray Level Zone Size Matrix etc. But we Radiomics is particularly useful in segmentation where the Region of Interest is masked and the foreground is present but unmasked. In our case since there was no foreground present, so first we considered the centre of image and converted the pixel values in the central region to 1 while the rest to 0. We applied the Radiomics to extract features from the central part of the image. The results are stored in NPY file and is ultimately concatenated in a Comma Separated file that has features like mean, version etc. Now for the second case we are extracting features from the sides and excluding the central portion of the image. So basically we are creating our own foregrounds as per our needs, masking the Region of Interest and accordingly extracting the features. After concatenating the results, we combine the two csv files side by side and dropped the unnecessary columns. After that it is passed through the Multi Layer Perceptron. 2.2 Multi Layer Perceptron Multi-Layer Perceptrons [ 2 ] are Artificial Neural Networks that comprise an input layer, one or more hidden layers, and output layers. We used a multi-layer perceptron to preprocess the handcrafted features further. The MLP maps an input radiomics vector \mathbf{x}\in\mathbb{R}^{d_{\text{in}}} to a compact embedding \mathbf{z}\in\mathbb{R}^{d_{\text{embed}}} by first applying a linear transformation, \mathbf{h}_{1}=\text{ReLU\cite[cite]{[\@@bibref{}{DBLP:journals/corr/abs-1803-08375}{}{}]}}(\mathbf{W}_{1}\mathbf{x}+\mathbf{b}_{1}), where \mathbf{W}_{1}\in\mathbb{R}^{1024\times d_{\text{in}}} . Dropout [ 4 ] regularization, \mathbf{h}_{2}=\text{Dropout\cite[cite]{[\@@bibref{}{10.5555/2627435.2670313}{}{}]}}(\mathbf{h}_{1},p=0.5), mitigates overfitting by randomizing feature selection. Another linear transformation, \mathbf{z}=\text{Dropout\cite[cite]{[\@@bibref{}{10.5555/2627435.2670313}{}{}]}}(\mathbf{W}_{2}\mathbf{h}_{2}+\mathbf{b}_{2},p=0.5), with \mathbf{W}_{2}\in\mathbb{R}^{d_{\text{embed}}\times 1024} , completes the embedding. This process reduces dimensionality and enhances the representation of complex, non-linear feature relationships, producing a robust embedding that can be effectively fused with CNN features for multi-modal classification of capsule endoscopy images. 2.3 Convolutional Neural Networks For image classification tasks or processing tasks we mainly use Convolutional Neural Networks or CNNs [ 5 ] . The backbone CNN model of the proposed framework is the DenseNet [ 6 ] CNN architecture, which we used to extract features from the dataset’s complicated endoscopic pictures. DenseNet’s ability to retrieve feature maps from all previous levels is the primary driving force behind its use. Dense blocks, in which the output of every layer inside a block is concatenated with all of the inputs of its preceding layers, are used to accomplish this. In particular, X_{l}=H_{l}([X_{0};X_{1};\ldots;X_{l-1}])\in\mathbb{R}^{h_{b}\times w_{b}\times d_{b}} (1) where X_{0},X_{1},\ldots,X_{l}\in\mathbb{R}^{h_{b}\times w_{b}\times d_{b}} represent the output feature maps from the 0 -th to the l -th layers, ";" denotes the concatenation operation, d_{b} is the feature dimension of the dense block b , and the convolution function H_{l}(\cdot) consists of a 3 × 3 convolution layer, a ReLU activation function, and batch normalization. 2.4 Projection Head In the context of image classification, let x\in\mathcal{X}\subset\mathbb{R}^{H\times W\times C} be the input image and f_{\theta}(x)=z\in\mathcal{F}\subset\mathbb{R}^{h\times w\times d} represent the high-dimensional feature vector obtained from a convolutional neural network (CNN). The dimensionality of z is often large, leading to potential redundancies that can impede the learning of discriminative features. To mitigate this, a projection head g_{\phi}(z)=y\in\mathbb{R}^{k} is employed, where k\ll h\times w\times d . The projection head [ 7 ] incorporates an Adaptive Average Pooling operation, which reduces spatial dimensions while preserving essential information: z^{\prime}=\text{GAP}(z)\in\mathbb{R}^{d}. Following this, a fully connected layer with Batch Normalization and ReLU activation transforms the pooled representation: y=\sigma(\textbf{W}z^{\prime}+b)\in\mathbb{R}^{k}, where \sigma denotes the activation function. This transformation enhances feature disentanglement and regularizes the representation space, which is crucial for effective generalization. By constraining the representation to a lower-dimensional space, the projection head not only reduces the risk of overfitting but also helps the classifier focus on relevant features, minimizing the empirical loss: \mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\ell(\hat{y}_{i},y_{i}). Thus, the introduction of a projection head serves to amplify the discriminative power of the model while ensuring a compact, interpretable representation suitable for classification tasks. 2.5 Integration of CNN Extracted Features and Numerical Radiomics Features Let \mathbf{X}_{img} be the image input, and \mathbf{X}_{num} be the numerical features derived from radiomics. The model outputs are represented as follows: 1. DenseNet Feature Extraction: \mathbf{F}_{cnn}=\text{DenseNet}(\mathbf{X}_{img}) 2. MLP Feature Extraction: \mathbf{F}_{mlp}=\text{MLP}(\mathbf{X}_{num}) 3. Projection Head for Dimensionality Reduction: \mathbf{F}_{proj}=\text{ProjectionHead}(\mathbf{F}_{cnn}) 4. Feature Concatenation: The combined feature vector is given by: \mathbf{F}_{combined}=\mathbf{F}_{proj}\oplus\mathbf{F}_{mlp} where \oplus denotes the concatenation operation. 5. Classification Output: The final classification output is obtained by passing the combined features through the classifier: \mathbf{y}=\text{ClassificationHead}(\mathbf{F}_{combined}) The combined features capture both visual ( \mathbf{F}_{cnn} ) and numerical ( \mathbf{F}_{mlp} ) information, enhancing the representation power: \mathcal{L}=\text{Loss}(\mathbf{y},\mathbf{y}_{true}) By integrating