ClusterViG: Efficient Globally Aware Vision GNNs via Image Partitioning

Dhruv Parikh 1 , Jacob Fein-Ashley 1 , Tian Ye 1 , Rajgopal Kannan 2 , Viktor Prasanna 1 · 2025

Read ClusterViG: Efficient Globally Aware Vision GNNs via Image Partitioning by Dhruv Parikh 1 , Jacob Fein-Ashley 1 , Tian Ye 1 , Rajgopal Kannan 2 , Viktor Prasanna 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-10640.txt.

Introduction

Dhruv Parikh and co-authors speed up graph-based vision models by cutting each image into blocks. Their 2025 paper notes that graph networks can model rich links, but vision versions pay a high cost to build their graphs by nearest-neighbor search. Past fixes hem in the graph and lose that free-form edge. Their method builds graphs in parallel inside each block, then learns both within blocks and across the whole image.

The block design forms the core of a mixed network named ClusterViG, part graph and part classic image network. It cut run time sharply against rival models of like size. Scores on sorting, spotting, and cutting out objects reached the top tier in their tests.

Catalog record

CREATORDhruv Parikh 1 , Jacob Fein-Ashley 1 , Tian Ye 1 , Rajgopal Kannan 2 , Viktor Prasanna 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-10640

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ClusterViG: Efficient Globally Aware Vision GNNs via Image Partitioning Dhruv Parikh 1 , Jacob Fein-Ashley 1 , Tian Ye 1 , Rajgopal Kannan 2 , Viktor Prasanna 1 Abstract Convolutional Neural Networks (CNN) and Vision Transformers (ViT) have dominated the field of Computer Vision (CV). Graph Neural Networks (GNN) have performed remarkably well across diverse domains because they can represent complex relationships via unstructured graphs. However, the applicability of GNNs for visual tasks was unexplored till the introduction of Vision GNNs (ViG). Despite the success of ViGs, their performance is severely bottlenecked due to the expensive k k -Nearest Neighbors ( k k -NN) based graph construction. Recent works addressing this bottleneck impose constraints on the flexibility of GNNs to build unstructured graphs, undermining their core advantage while introducing additional inefficiencies. To address these issues, in this paper, we propose a novel method called Dynamic Efficient Graph Convolution (DEGC) for designing efficient and globally aware ViGs. DEGC partitions the input image and constructs graphs in parallel for each partition, improving graph construction efficiency. Further, DEGC integrates local intra-graph and global inter-graph feature learning, enabling enhanced global context awareness. Using DEGC as a building block, we propose a novel CNN-GNN architecture, ClusterViG, for CV tasks. Extensive experiments indicate that ClusterViG reduces end-to-end inference latency for vision tasks by up to 5 × 5\times when compared against a suite of models such as ViG, ViHGNN, PVG, and GreedyViG, with a similar model parameter count. Additionally, ClusterViG reaches state-of-the-art performance on image classification, object detection, and instance segmentation tasks, demonstrating the effectiveness of the proposed globally aware learning strategy. Finally, input partitioning performed by DEGC enables ClusterViG to be trained efficiently on higher-resolution images, underscoring the scalability of our approach. ClusterViG enables real-time deployment of GNN-based models for CV, providing a new avenue for future research. I Introduction † † Distribution Statement A: Approved for public release. Distribution is unlimited. Graph Neural Networks (GNN) have demonstrated superior performance across diverse domains ranging from complex network analysis [ 1 , 2 ] , knowledge graphs [ 3 , 4 ] . Recommender systems [ 5 , 6 ] to chemistry [ 7 ] , physical systems [ 8 , 9 ] and brain network analysis [ 10 ] . GNNs utilize unstructured graphs comprising nodes (vertices) and edges (links) to represent data. This enables their performance on domains involving non-Euclidean data, using graphs to represent such complex data to perform feature representation learning at the node/edge/graph-level features [ 11 ] . Despite their success in the above domains, the applicability of GNNs across Computer Vision (CV) has been limited. Traditionally, CV has been dominated by Convolutional Neural Network (CNN) based architectures [ 12 , 13 ] . CNNs have been widely used for image classification [ 14 ] , object detection [ 15 ] , semantic segmentation [ 16 ] , instance segmentation [ 17 ] , etc. tasks in CV. Recently, Multi-Layer Perceptron (MLP) based vision models [ 18 , 19 ] have also performed well across several CV tasks. CNN and MLP-based models treat the input image as a structured grid of pixels and apply structured convolution and fully connected layers to the pixel grid for feature representation learning [ 12 , 18 ] . The transformer architecture, initially introduced for Natural Language Processing (NLP) in [ 20 ] , has significantly influenced advancements in CV [ 21 , 22 , 23 , 24 , 25 ] . Vision Transformers (ViT) [ 21 ] represent an input image as a sequence of patches and utilize the transformer architecture to process such a sequence. Unlike CNNs and MLPs, which operate with local receptive fields, Vision Transformers (ViTs) leverage the Multi-Headed Self-Attention (MHSA) mechanism to achieve a global receptive field [ 26 ] . While CNN, MLP, and ViT-based models have been successful in CV, they are limited by their reliance on rigid grid or sequence representations of images. Vision GNNs (ViG), proposed in [ 27 ] , address this limitation by using graphs to represent images, offering greater flexibility. Additionally, ViG utilizes GNNs to learn about such image graphs, extending the GNN framework to CV. Despite its superior performance across benchmark CV tasks, ViG suffers from two severe drawbacks: (i) High computational expense of dynamic graph construction. Specifically, ViG employs k -Nearest Neighbors ( k -NN) to connect similar image patches (nodes) with edges, forming an image graph. This k -NN graph construction is repeated at every model layer as patch (node) features are updated through the GNN layers. (ii) Limited flexibility of image graph construction. Specifically, ViG restricts edge construction in the image graph to pairwise, first-order similar nodes (patches), which reduces the model’s receptive field and undermines the primary advantage of using unstructured graphs to represent images. Several recent works [ 28 , 26 , 29 , 30 , 31 ] have been proposed to address the above two drawbacks of ViG. ViHGNN [ 28 ] and PVG [ 29 ] increase the flexibility of image graph construction while increasing its computational expense. Meanwhile, GreedyViG [ 26 ] and MobileViG [ 30 , 31 ] utilize structured graphs for image graph construction, reducing computational expense at the cost of flexibility. Thus, no prior work exists that addresses both drawbacks of ViG [ 27 ] and ViG-based models [ 28 , 26 , 29 , 30 , 31 ] , simultaneously . In light of this, in this paper, we propose a novel method called Dynamic Efficient Graph Convolution (DEGC) to design a new class of ViG-based models, ClusterViG. ClusterViG significantly reduces the computational expense of dynamic image graph construction without sacrificing its flexibility. DEGC, the main building block of ClusterViG, facilitates this improvement. DEGC partitions the input image patches and performs graph construction in parallel for each partition, reducing the computational expense of dynamic image graph construction. It then learns a global feature vector for each partition through inter-graph interactions among the constructed graphs. Finally, each partition’s local patch (node) features are updated in parallel, using both local and global features through intra-graph interactions. This enhances the receptive field of ClusterViG and fully leverages the flexibility of unstructured graph representation learning via GNNs. We summarize the contributions of our work below, • We propose a novel method, Dynamic Efficient Graph Convolution (DEGC), to build a new class of ViG models, ClusterViG. ClusterViG reduces the computational expense of dynamic image graph construction without sacrificing its flexibility for graph representation learning via GNNs. • DEGC is a general method for designing efficient Graph Convolutional Networks (GCNs) for CV. We demonstrate its generality by implementing DEGCs efficient dynamic image graph construction and flexible local-global graph representation learning with standard GCN backbones. • ClusterViG reaches state-of-the-art (SOTA) performance across three representative CV tasks: ImageNet image classification [ 32 ] , COCO object detection [ 33 ] , and COCO instance segmentation [ 33 ] , when compared to its SOTA counterparts (CNN, MLP, ViT or ViG based), with comparable total model parameters and GMACs. • ClusterViG reaches this SOTA performance with a hardware-friendly isotropic architecture. This contrasts with prior SOTA models, which typically have a pyramid architecture. • ClusterViG achieves up to 5\times reduction in

Read the complete text