GNN-MoE: Context-Aware Patch Routing using GNNs for Parameter-Efficient Domain Generalization
Mahmoud Soliman, Omar Abdelaziz, Ahmed Radwan, Anand, Mohamed Shehata · 2025
Read GNN-MoE: Context-Aware Patch Routing using GNNs for Parameter-Efficient Domain Generalization by Mahmoud Soliman, Omar Abdelaziz, Ahmed Radwan, Anand, Mohamed Shehata (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04008.txt.
Introduction
Mahmoud Soliman, Omar Abdelaziz, Ahmed Radwan, Anand, and Mohamed Shehata propose GNN-MoE for adapting vision transformers to new domains with few added parameters. Domain generalization needs strong performance on unseen domains, but full fine-tuning is costly and can itself harm generalization.
A graph neural network router works on relationships between image patches and assigns patches to specialised mixture-of-experts modules, replacing token-based routing. Kronecker adapters keep the added parameters efficient, and several graph network types serve as routers. Context-aware routing uses patch relationships to adapt better under domain shift. Benchmark results were state-of-the-art or competitive with high parameter efficiency, supporting graph-based routing for lightweight adaptation.
Catalog record
| CREATOR | Mahmoud Soliman, Omar Abdelaziz, Ahmed Radwan, Anand, Mohamed Shehata |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GNN-MoE: Context-Aware Patch Routing using GNNs for Parameter-Efficient Domain Generalization Mahmoud Soliman, Omar Abdelaziz, Ahmed Radwan, Anand, Mohamed Shehata Abstract Domain generalization (DG) seeks robust Vision Transformer (ViT) performance on unseen domains. Efficiently adapting pretrained ViTs for DG is challenging; standard fine-tuning is costly and can impair generalization. We propose GNN-MoE, enhancing Parameter-Efficient Fine-Tuning (PEFT) for DG with a Mixture-of-Experts (MoE) framework using efficient Kronecker adapters. Instead of token-based routing, a novel Graph Neural Network (GNN) router (GCN, GAT, SAGE) operates on inter-patch graphs to dynamically assign patches to specialized experts. This context-aware GNN routing leverages inter-patch relationships for better adaptation to domain shifts. GNN-MoE achieves state-of-the-art or competitive DG benchmark performance with high parameter efficiency, highlighting the utility of graph-based contextual routing for robust, lightweight DG. † † address: {mosama97,oabdelaz,ahmedm04,a}@student.ubc.ca, [email protected] 1 Introduction Vision Transformers (ViTs) [ 3 ] excel in computer vision but struggle with domain generalization (DG) [ 22 ] , often overfitting to source domains [ 17 ] when fully fine-tuned, which is also computationally expensive and prone to catastrophic forgetting. Parameter-Efficient Fine-Tuning (PEFT) methods like Adapters [ 8 ] and LoRA [ 9 ] offer lightweight adaptation by tuning only a small subset of parameters. Mixture of Experts (MoE) architectures [ 15 , 4 ] extend PEFT by routing inputs to specialized expert sub-networks. However, standard MoE routers typically operate on isolated token features, ignoring inter-patch relationships crucial for robust expert assignment under domain shifts. To address this, we propose GNN-MoE, a novel framework integrating Graph Neural Networks (GNNs) [ 10 ] for context-aware routing of ViT image patches to highly efficient Kronecker Adapter [ 7 ] experts. This GNN-based routing on inter-patch graphs enhances adaptation to domain shifts. Our key contributions are: • GNN-MoE: The first framework combining GNN-based routing with parameter-efficient Kronecker Adapters for ViT domain generalization. • A graph-based token routing mechanism capturing inter-patch relationships for improved patch-to-expert assignment. • State-of-the-art or competitive performance on DG benchmarks with high parameter efficiency. 2 Related Work This section reviews literature relevant to our GNN-MoE framework, covering Domain Generalization, Graph Neural Networks, Parameter-Efficient Fine-Tuning, and Mixture of Experts, highlighting their connection to our approach. 2.1 Domain Generalization (DG) Domain Generalization (DG) trains models for robust performance on unseen target domains using data from multiple source domains [ 22 ] , unlike domain adaptation which typically uses target data. The core challenge is learning domain-invariant yet task-relevant representations. DG strategies include domain alignment, meta-learning, learning invariant representations, and data augmentation, underscoring the complexity of out-of-distribution generalization. 2.2 Graph Neural Networks (GNNs) Graph Neural Networks (GNNs) model relationships in graph-structured data via message-passing paradigms. Architectures like Graph Convolutional Networks (GCNs) [ 10 ] , GraphSAGE [ 6 ] , and Graph Attention Networks (GATs) [ 19 ] learn node representations. In vision, GNNs can model contextual relationships between image patches, a principle we leverage for inter-patch routing. 2.3 Parameter-Efficient Fine-Tuning (PEFT) Parameter-Efficient Fine-Tuning (PEFT) adapts large pre-trained models like ViTs cost-effectively by freezing most parameters and training only a small subset [ 8 ] , mitigating catastrophic forgetting and computational demands. PEFT includes adapter modules and Low-Rank Adaptation (LoRA) [ 9 ] . Our work uses efficient Kronecker-factorized adapters [ 7 ] , relevant for DG by enabling domain-specific adaptation while preserving general pre-trained features [ 2 ] . Kronecker adapters reduce parameter count by decomposing the transformation matrix into compact, structured factors, enabling efficient specialization for each domain expert. 2.4 Mixture of Experts (MoE) Mixture of Experts (MoE) architectures enhance model capacity using multiple expert subnetworks and a router for dynamic input token assignment [ 16 ] . For DG, MoE allows experts (our GNN-MoE uses Kronecker adapters) to specialize in different domain characteristics or domain-invariant features. However, standard MoE routers often ignore inter-patch context, which GNN-MoE addresses. 3 Methodology Input Features \mathbf{X}_{\text{in}}\in\mathbb{R}^{B\times N\times D} (Batch, Nodes, Features) Frozen Backbone \mathbf{W}_{0}\in\mathbb{R}^{D\times D} Pretrained weights GNN Router \mathcal{R} Outputs: \mathbf{G}(\mathbf{X})\in\mathbb{R}^{B\times N\times N_{e}} Expert 1 \mathbf{A}_{1}\in\mathbb{R}^{D\times D} Expert 2 \mathbf{A}_{2}\in\mathbb{R}^{D\times D} Expert 3 \mathbf{A}_{3}\in\mathbb{R}^{D\times D} \vdots Expert {}_{N_{e}} \mathbf{A}_{N_{e}}\in\mathbb{R}^{D\times D} Domain-Specific Expert Set \mathcal{E} Weighted Aggregation \bigoplus (Element-wise Sum) Routing-weighted combination Output Features \mathbf{X}_{\text{out}}\in\mathbb{R}^{B\times N\times D} Domain-adapted features \mathbf{X}_{\text{in}} \mathbf{X}_{\text{in}} \mathbf{G}(\mathbf{X}) \mathbf{X}_{\text{in}}\mathbf{W}_{0} \sum_{i}\mathbf{G}_{i}\odot(\mathbf{X}\mathbf{A}_{i}) \mathbf{X}_{\text{out}} Trainable Figure 1 : GNN-Routed Mixture-of-Experts Architecture for Domain Generalization. The architecture combines a frozen pretrained backbone with trainable GNN-based routing and domain-specific expert adapters. The GNN router analyzes input structure to generate routing weights, enabling adaptive combination of domain experts for robust cross-domain performance. Figure 2 : A representation of an old kettle of OfficeHome dataset broken down into graph nodes. An optimal GNN router should be capable of understanding this relational graph in order to route patches to the corresponding experts. The proposed GNN-Routed Mixture-of-Experts Kronecker Adapter (GNN-MoE) module (Figure 1 ) replaces standard linear transformations (e.g., QKV, FFNs) in ViT encoder blocks. Figure 2 visualizes a target graph structure. Input \bm{X}_{\text{in}}\in\mathbb{R}^{S\times D} ( S tokens, dimension D ) is processed via two pathways: Frozen Pathway: Input \bm{X}_{\text{in}} is processed by the frozen weight matrix \bm{W}_{0}\in\mathbb{R}^{D\times D} (snowflake icon, Figure 1 ) yielding \bm{Y}_{\text{frozen}}=\bm{X}_{\text{in}}\bm{W}_{0} . Original bias \bm{b}_{0} , if present, is applied. MoE Adapter Pathway: Introduces trainable components: • GNN Layer (Router): Processes input \bm{X}_{\text{in}} (or patch subset \bm{X}_{\text{patches}} , Sec. 3.2 ) using a trainable GNN on a patch graph to generate routing weights \bm{G}(\bm{X}_{\text{in}})\in\mathbb{R}^{S\times N_{e}} for N_{e} experts. • Expert Adapters: N_{e} lightweight, trainable Kronecker adapters \mathcal{E}=\{\text{Adapter}_{1},\dots,\text{Adapter}_{N_{e}}\} . Each \text{Adapter}_{i} learns an update matrix \bm{H}_{i}\in\mathbb{R}^{D\times D} (Sec. 3.1 ). Input \bm{X}_{\text{in}} is processed as \operatorname{Expert}_{i}(\bm{X}_{\text{in}})=\bm{X}_{\text{in}}\bm{H}_{i} . • MoE Combination: Expert outputs are combined via routing weights \bm{G}(\bm{X}_{\text{in}}) . For token s : (\bm{Y}_{\text{MoE}}(\bm{X}_{\text{in}}))_{s,:}=\sum_{i=1}^{N_{e}}G(\bm{X}_{\text{in}})_{s,i}\cdot(\bm{X}_{\text{in}}\bm{H}_{i})_{s,:}. (1) Final Module Output: Outputs from both pathways are summed with a trainable bias \bm{b}_{\text{adapter}}\in\mathbb{R}^{D} : \bm{X}_{\text{out}}=\bm{X}_{\text{in}}\bm{W}_{0}+\sum_{i=1}^{N_{e}}\bm{G}(\bm{X}_{\text{in}})_{:,i}\odot(\bm{X}_{\text{in}}\bm{H}_{i})+\bm{b}_{\text{adapter}} (2) where \odot is element-wise multiplication with broadcasting of G(\bm{X}_{\text{in}})_{:,i} . The GNN router conditions adapter selection on \bm{X}_{\text{in}} ’s context. Trainable components (GNN router, Kronecker adapter factors \bm{H}_{i} , \bm{b}_{\text{adapter}} ) are optimized end-to-end. 3.1 Kronecker Adapter Experts ( \operatorname{Expert}_{i} ) Each \operatorname{Expert}_{i} computes an update matrix \Delta\bm{W}_{i} via Parameter-efficient Hypercomplex Multiplication (PHM). The effective update matrix \bm{H}_{i}\in\mathbb{R}^{D_{\text{in}}\times D_{\text{out}}} ( \Delta\bm{W}_{i}^{T}=\bm{H}_{i} ) is: \bm{H}_{i}=\operatorname{Dropout}\left(\sum_{k=1}^{d_{\text{phm}}}\left((\bm{H}_{\text{shared}})_{k}\otimes(\bm{H}_{\text{expert\_factors},i})_{k}\right)\right) (3) where \otimes is Kronecker product. (\bm{H}_{\text{shared}})_{k} is the k -th slice of a shared tensor from PHM rule matrices (e.g., \bm{P}_{L},\bm{P}_{R}\in\mathbb{R}^{d_{\text{phm}}\times d_{\text{phm}}\times 1} ). (\bm{H}_{\text{expert\_factors},i})_{k} \in\mathbb{R}^{(D_{\text{in}}/d_{\text{phm}})\times(D_{\text{out}}/d_{\text{phm}})} is the k -th slice from low-rank factors as \bm{L}_{i}\bm{R}_{i} ( d_{\text{phm}} is PHM dimension, r_{i} is expert rank). Summation is over PHM slices k . Expert output: \operatorname{Expert}_{i}(\bm{X})=\bm{X}\bm{H}_{i}. (4) 3.2 GNN-Based Router A GNN-based router determines weights \bm{G}(\bm{X}) for MoE, operating on a graph from patch tokens \bm{X}_{\operatorname{patches}}\in\mathbb{R}^{N_{p}\times D_{\text{in}}} (excluding class token \bm{z}_{\operatorname{cls}} ). Graph Construction Strategies: Defines connectivity \operatorname{EdgeIndex}\in\mathbb{Z}^{2\times N_{\text{edges}}} . Spatial Adjacency: Connects immediate spatial neighbors (e.g., 8-connectivity) for local context. Radius: Connects patches v_{j},v_{k} if Euclidean distance \|\operatorname{coord}(v_{j})-\operatorname{coord}(v_{k})\|_{2}\leq r_{\text{th}} (Eq. 5 ), found most effective. (v_{j},v_{k})\in\mathcal{E}\iff\|\operatorname{coord}(v_{j})-\operatorname{coord}(v_{k})\|_{2}\leq r_{\text{th}}. (5) Fully Connected: Connects all patch nodes for global context (higher cost). Self-loops are added for all types. Graph structure influences available context. GNN Architecture: We use GNNs for contextualized representations. Primarily Graph Convolutional Network (GCN) [ 10 ] . Given initial patch features \bm{h}_{v}^{(0)}=(\bm{X}_{\operatorname{patches}})_{v} , GCN layer l updates \bm{h}_{v}^{(l)} : \bm{h}_{v}^{(l)}=\sigma\left(\sum_{u\in\mathcal{N}(v)\cup\{v\}}\frac{1}{\sqrt{\deg(v)\deg(u)}}\bm{W}^{(l)}\bm{h}_{u}^{(l-1)}\right). (6) \mathcal{N}(v) are neighbors of v , \deg(v) is degree, \bm{W}^{(l)} is learnable