Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approach
Yinlin Zhu, Di Wu, Xianzhi Zhang, Yuming Ai, Xunkai Li, Miao Hu, Guocong Quan · 2026
Read Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approach by Yinlin Zhu, Di Wu, Xianzhi Zhang, Yuming Ai, Xunkai Li, Miao Hu, Guocong Quan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21369.txt.
Introduction
Yinlin Zhu and six co-authors rethink graph foundation models trained across split data. Graphs held by separate owners cannot be pooled, for privacy and legal reasons. A shared model must learn from those silos in place.
Earlier designs squeeze shared knowledge into a fixed set of discrete tokens. That squeezing loses information for good, and the loss can hurt the model. The authors instead align graph data with language in a way that keeps more of the signal. Their approach aims at shared graph models that respect data silos without paying a heavy quality price.
Catalog record
| CREATOR | Yinlin Zhu, Di Wu, Xianzhi Zhang, Yuming Ai, Xunkai Li, Miao Hu, Guocong Quan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21369
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approach Yinlin Zhu, Di Wu, Xianzhi Zhang, Yuming Ai, Xunkai Li, Miao Hu, Guocong Quan Abstract Recent studies of federated graph foundational models (FedGFMs) break the idealized and untenable assumption of having centralized data storage to train graph foundation models, and accommodate the reality of distributed, privacy-restricted data silos. Despite their simplicity and intuition, existing studies that project aligned generalizable knowledge onto a discrete token space via vector-quantized backbones suffer from irreversible knowledge loss during the quantization process. In this context, we argue that reconciling the semantic-structural orthogonality and integrity between pre-trained language models (PLMs) and graph neural networks (GNNs) is paramount for developing effective FedGFMs while simultaneously mitigating the severe data heterogeneity and communication constraints inherent in distributed, resource-limited environments. To address these issues, we propose FedGALA ( Fed erated G raph A nd L anguage A lignment), a framework that resolves graph-based semantic-structural orthogonality and integrity in federated settings by employing unsupervised contrastive learning to align GNNs and frozen PLMs within a continuous embedding space, thereby capturing robust, transferable general knowledge. Subsequently, FedGALA leverages a communication-efficient prompt tuning mechanism to steer these pre-aligned encoders and frozen PLMs, facilitating effective adaptation to diverse downstream tasks while circumventing the prohibitive overhead of full-parameter fine-tuning. The comprehensive experiments validate that FedGALA outperforms all competitive baselines across multi-domain datasets on multiple tasks with up to 14.37% performance improvement. 1 Introduction The success of pre-trained language models (PLMs) has catalyzed the rise of generalizable foundation models ( 19 ; 33 ; 3 ) across diverse downstream tasks ( 2 ; 42 ; 4 ) . This paradigm naturally extends to text-attributed graphs (TAGs) through the emergence of graph foundation models (GFMs). While GFMs leverage the synergy between textual semantics and complex topology, their reliance on centralized corpora is often untenable given the fragmented, privacy-restricted nature of real-world data silos ( 6 ; 40 ; 18 ) . Consequently, federated graph learning (FGL) has emerged as the imperative solution for collaborative training without raw data transfer ( 41 ; 44 ) , yet the development of robust federated graph foundation models (FedGFMs) remains a critical and underexplored frontier. Constructing effective FedGFMs requires addressing three fundamental dimensions: ❶ Semantic-Structural Orthogonality: The disconnect between sequential language and non-Euclidean topology necessitates a dual-encoder architecture that is difficult to synchronize in decentralized settings. PLMs lack structural inductive biases, while textual neighborhood summaries are limited by narrow receptive fields. In FedGFMs, topological heterogeneity ( 17 ; 16 ) across silos impedes the alignment of semantic and structural spaces without direct data access. This dichotomy complicates knowledge generalization, leading to cross-domain entanglement. ❷ Cross-domain Knowledge Entanglement: FedGFM generalizability is limited by fragmented local datasets. Severe non-IID distributions, manifested as semantic and structural divergence across clients, hinder the learning of unified, transferable representations. The divergence exacerbated by disparate domains causes conflicting local motifs that trigger overfitting and obstruct the acquisition of transferable knowledge, preventing robust global model convergence. ❸ Communication Overhead: Bandwidth bottlenecks in federated settings make exchanging massive PLM parameters infeasible. Developing practical FedGFMs requires bandwidth-efficient paradigms that adapt frozen models to downstream tasks while maintaining semantic-structural alignment under strict physical network constraints. Pioneering frameworks like FedGFM+ ( 43 ) and FedBook ( 31 ) utilize vector-quantized (VQ) methods ( 30 ) as backbones. However, these approaches reveal critical deficiencies when evaluated against the aforementioned fundamental challenges: Regarding Challenge 1 : Projecting high-dimensional semantic spaces onto discrete tokens induces irreversible quantization errors. This discretization loses the subtle interplay between textual semantics and complex topologies, which severely limits structural-semantic fidelity. Regarding Challenge 2 : VQ methods trade representational completeness for statistical generalizability by forcing diverse local distributions onto a constrained global codebook, leading to representation homogenization and neglecting nuanced client-specific long-tail knowledge. Regarding Challenge 3 : Transmitting discrete tokens creates a suboptimal equilibrium that sacrifices model convergence quality for communication efficiency. These limitations necessitate a strategic shift toward a new training paradigm of FedGFMs. To this end, we propose FedGALA ( Fed erated G raph A nd L anguage A lignment), which leverages unsupervised contrastive learning to bridge the semantic-structural gap in continuous latent spaces. By prioritizing representational fidelity through continuous alignment, FedGALA enables the framework to harness PLMs’ generalized knowledge for enhanced cross-domain transferability and robust few-shot performance, establishing a superior foundation for the two-stage training pipeline: Federated Pre-training Phase for cross-domain graph encoding and Prompt-based Fine-tuning Phase for task adaptation. Specifically, in order to resolve Challenge 1 , FedGALA aligns the frozen PLM textual features with the topology-oriented GNN features; similarly, to address Challenge 2 , it disentangles the graph encoder into local semantic and shared structural components, while also using semantic perturbation and a history-aware ensemble to stabilize domain-invariant global updates; and finally, to overcome Challenge 3 , it aggregates only the lightweight structural encoders and employs prompt tuning to efficiently adapt the frozen backbones. Our Contributions: ❶ New Perspective. We pinpoint representation loss in quantized FedGFMs and propose continuous structural-semantic alignment as a robust training alternative. ❷ New Framework. FedGALA bridges semantic-structural gaps through federated contrastive pre-training followed by communication-efficient local prompt fine-tuning. ❸ Superior Performance. Comprehensive evaluations confirm that FedGALA surpasses 22 SOTA baselines with significant performance gains reaching 14.37%. 2 Preliminaries 2.1 Problem Formalization The FedGFM framework comprises K distributed clients where each client k possesses a private subgraph G_{k} and textual attributes \mathcal{T}_{k} , aiming to collaboratively train a graph encoder f_{\theta} while the text encoder g_{\psi} remains frozen. Pre-training phase establishes a modality-invariant space by minimizing a global contrastive loss \mathcal{L}_{pre} that uses weights \alpha_{k} to aggregate across-domain structural knowledge. \min_{\theta}\mathcal{L}_{pre}=\sum_{k=1}^{K}\alpha_{k}\mathcal{L}_{local}(G_{k},\mathcal{T}_{k};\theta,\psi), (1) In the fine-tuning phase, backbone models, such as \theta_{frozen} , \psi_{frozen} remain frozen to shift optimization toward a personalized prompt pool \mathcal{P}=\{\phi_{1},\dots,\phi_{M}\} . Each client identifies the best-fitting prompt index m_{k}^{*} to minimize the task-specific loss \mathcal{L}_{task} for local data distributions D_{k} . \min_{\phi_{m_{k}^{*}}}\mathcal{L}_{task}(D_{k};\theta_{frozen},\psi_{frozen},\phi_{m_{k}^{*}}). (2) 2.2 Graph Foundation Model (GFM) GFMs ( 39 ; 19 ) utilize hybrid architectures where frozen PLMs extract semantic features and trainable GNNs encode topological structures for multi-modal expertise. Existing research employs self-supervised objectives like contrastive learning ( 38 ; 23 ) or cross-modal alignment ( 28 ; 15 ; 13 ) to unify textual and structural knowledge before downstream adaptation. Benchmark system ( 36 ) has recently been proposed to evaluate centralized GFMs systematically on their transferability across multiple domains. Centralized GFMs are unrealistic because single centers lack the massive data scale required for transferability, and privacy laws prevent sharing raw data. Therefore, a collaborative paradigm is essential to utilize distributed datasets without compromising privacy. 2.3 Federated Graph Foundation Model (FedGFM) FedGFMs integrate the multi-modal expertise of GFMs into a privacy-preserving collaborative training framework. FedGFM+ ( 43 ) utilizes anchor-based initialization and domain-sensitive prompt pools for personalization. FedBook ( 31 ) employs a unified federated codebook to model intra-domain and inter-domain knowledge. Both approaches using discrete vector quantization suffer from irreversible knowledge loss. To preserve representational fidelity, a redesign is necessary to align GNNs with frozen PLMs via self-supervised contrastive learning in a continuous latent space. FedGALA addresses this through a bandwidth-efficient framework that achieves multi-modal alignment for robust structural knowledge transfer. 3 Methods Figure 1 :