iPac: Incorporating Intra-image Patch Context into Graph Neural Networks for Medical Image Classification
Usama Zidan, Mohammed Medhat Gaber, Mohammed M. Abdelsamea · 2025
Read iPac: Incorporating Intra-image Patch Context into Graph Neural Networks for Medical Image Classification by Usama Zidan, Mohammed Medhat Gaber, Mohammed M. Abdelsamea (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23504.txt.
Introduction
Usama Zidan, Mohamed Gaber, and Mohammed M Abdelsamea present iPac, a graph neural network method for medical image classification that builds graphs from image patches with their context. Graph networks are promising for images, yet they often ignore the structure and relations among visual entities. iPac constructs a new graph representation of each image.
Its stages partition the image into patches, extract features, cluster them, and build a graph that carries intra-image patch context into the network. Classification then uses relations between regions, not just isolated features. Relational reading may catch disease patterns that flat models miss.
Catalog record
| CREATOR | Usama Zidan, Mohammed Medhat Gaber, Mohammed M. Abdelsamea |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23504
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
iPac: Incorporating Intra-image Patch Context into Graph Neural Networks for Medical Image Classification Usama Zidan, Mohammed Medhat Gaber, Mohammed M. Abdelsamea Abstract Graph neural networks have emerged as a promising paradigm for image processing, yet their performance in image classification tasks is hindered by a limited consideration of the underlying structure and relationships among visual entities. This work presents iPac, a novel approach to introduce a new graph representation of images to enhance graph neural network image classification by recognizing the importance of underlying structure and relationships in medical image classification. iPac integrates various stages, including patch partitioning, feature extraction, clustering, graph construction, and graph-based learning, into a unified network to advance graph neural network image classification. By capturing relevant features and organising them into clusters, we construct a meaningful graph representation that effectively encapsulates the semantics of the image. Experimental evaluation on diverse medical image datasets demonstrates the efficacy of iPac, exhibiting an average accuracy improvement of up to 5% over baseline methods. Our approach offers a versatile and generic solution for image classification, particularly in the realm of medical images, by leveraging the graph representation and accounting for the inherent structure and relationships among visual entities. 1 Introduction Image classification is a core task in computer vision, involving the assignment of labels to images based on their content. Traditional deep learning methods such as convolutional neural networks (CNNs) and transformers have shown significant success by leveraging the grid or sequence structure of images. However, there is growing interest in using graph neural networks (GNNs) for image classification due to their ability to capture relational information and handle nongrid structured data. GNNs are particularly beneficial for medical images, where objects of interest often have irregular shapes and overlapping characteristics. Several state-of-the-art methods have explored the use of GNNs for image classification, particularly in the medical domain. Recent approaches have tackled the challenge of representing images as graphs while preserving spatial and semantic information. Methods such as superpixel over-segmentation [ 6 ] and region proposals have been employed to convert images into graph representations. The efficacy of Graph Attention Networks (GATs) for image classification has been demonstrated in [ 2 ] , achieving high performance on standard datasets, but noting potential issues with information loss and memory usage during graph generation. Hierarchical GNN [ 5 ] was introduced to detect abnormal lesions in mammograms, combining CNNs for feature extraction with GATs for node and graph classification, achieving state-of-the-art performance on the INbreast dataset. Similarly, a synergic GNN for medical image classification was proposed in [ 13 ] , enhancing accuracy by combining features from different Graph Convolutional Network (GCN) variants. In [ 3 ] , a GNN-based model was proposed in a way to make use of a self-attention mechanism to classify digital breast tomosynthesis images and has shown superior performance compared to traditional CNN methods. Additionally, GNNs have been leveraged for detecting and segmenting lymph node tumors in oncology imaging, demonstrating significant improvements over CNN-based models [ 4 ] . An attention-guided deep GNN has been applied to analyze the progression of Alzheimer’s disease, achieving superior classification accuracy by processing brain network data from MRI images [ 11 ] . Furthermore, integration of a GNN module with a 3D UNet to segment tree-like structures on chest CT has showcased the benefits of graph neighborhood connectivity [ 7 ] . These recent methods highlight the advancements in using GNNs for complex image processing tasks, addressing challenges in capturing semantic and structural information of medical images. In medical imaging, accurate classification often requires capturing intricate details of cell/tissue morphology and spatial relationships, which traditional pixel-based methods may miss. To address this, we propose iPac, a novel approach that converts images into graphs that can model the relationships among different clusters of patches, to enhance classification accuracy using GNNs. iPac is evaluated on various medical image classification tasks, including histological images, skin lesion diagnosis, and retinal imaging. iPac outperforms state-of-the-art methods using GNNs on regular grids or similarly sized superpixels, demonstrating the effectiveness of our image-to-graph conversion and GNN-based classification approach. Figure 1 : iPac consists of three main stages: image patching, patch encoding and clustering, and graph formation. We divide an input image into several patches, encode them using a Swin transformer autoencoder, and cluster them. A GNN is constructed to represent the detected clusters in an image with layers iterating over the node representation of each cluster, taking into account the edge weights that represent the inherent relationship between patch and cluster pairs. 2 Methodology Our proposed approach, Figure 1 , is a multistage method to convert an image into a graph representation, addressing limitations of pixel-based approaches and super-pixel representations of current mode. Initially, the input image is partitioned into a fixed grid of patches. We train a Swin Transformer-based autoencoder to encode these patches into high-dimensional embeddings that capture spatial and semantic information. These embeddings are then clustered to form a compact set, each represented by its centroid. Let x_{i,j} be the j th patch in the i th image, with j\in\{1,\dots,P\} , and P being the total number of patches. We train an autoencoder with encoder f_{\theta} and decoder g_{\phi} to reconstruct x_{i,j} from a compressed vector representation z_{i,j} , optimizing the mean squared error. The complete pipeline is outlined in Algorithm 1 . Data: Set of images \mathcal{I} Result: Graph representations for each image in \mathcal{I} foreach image I in \mathcal{I} do P\leftarrow getPatches( I ) ; // Split image into patches E\leftarrow Encoder( P ) ; // Obtain embeddings for patches C\leftarrow detectClusters( E ) ; // Detect clusters using clustering algorithm G\leftarrow constructGraph( C , P ) ; // Construct Graph use G for graph-based learning and image classification ; end foreach Algorithm 1 iPac algorithm After training an encoder, we encode patches from the input image. The method for determining the number of patches per image is detailed in Section 3.3 . Each patch, now represented in a high-dimensional space, is clustered using an off-the-shelf clustering algorithm, such as k-means , to form a high-level representation of the objects in the image. We construct an adjacency matrix A based on the spatial distribution of these clustered patches (Figure 2 ). Formally, A is a C\times C matrix, where C is the number of clusters. Elements of A are defined as: A_{ij}=\begin{cases}\frac{n_{ij}}{n_{i,\cdot}},&\text{if }n_{ij}>0\\ 0,&\text{otherwise}\end{cases} (1) where n_{ij} is the number of patches from cluster j adjacent to patches of cluster i , and n_{i,\cdot} is the total number of patches adjacent to patches of cluster i . The image is divided into patches where each patch is assigned to a cluster based on clustering performed on the entire dataset. We aggregate each cluster’s patches as a node representation. In the second stage, we construct a graph in which the nodes represent clusters of patches identified within the image. To connect these nodes, we analyse the spatial relationships between patches belonging to different clusters. Specifically, we examine how frequently patches from one cluster are adjacent to patches from another cluster within the image. The more often these patches are neighbours, the stronger the connection between their respective cluster nodes in the graph. This adjacency-based weighting