S 2 Aligner: Pair-Efficient Transferable Pre-Training for Sparse Text-Attributed Graphs

Yuhan Wang, Haopeng Zhang, Yibo Ding, Jiaqi Yu, Xinyu Zhao, Yuhang Liu, Ziwei Zhang, Xiao Wang, Ruijie Wang · 2026

Read S 2 Aligner: Pair-Efficient Transferable Pre-Training for Sparse Text-Attributed Graphs by Yuhan Wang, Haopeng Zhang, Yibo Ding, Jiaqi Yu, Xinyu Zhao, Yuhang Liu, Ziwei Zhang, Xiao Wang, Ruijie Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18579.txt.

Introduction

Yuhan Wang and eight co-authors present S2Aligner, a 2026 sparsity-aware LLM-as-Aligner system for pre-training on sparse text-attributed graphs. It treats meaning and structure separately, dividing representations into meaning and structure parts and adding topology cues to text vectors with consistency control. Cross-domain risk balancing tunes domain risks and downweights shaky sparse samples. Theory shows the goal narrows cross-domain generalization gaps, and tests across domains, sparsity levels, and tasks beat current baselines.

Catalog record

CREATORYuhan Wang, Haopeng Zhang, Yibo Ding, Jiaqi Yu, Xinyu Zhao, Yuhang Liu, Ziwei Zhang, Xiao Wang, Ruijie Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61644
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18579

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S 2 Aligner: Pair-Efficient Transferable Pre-Training for Sparse Text-Attributed Graphs Yuhan Wang, Haopeng Zhang, Yibo Ding, Jiaqi Yu, Xinyu Zhao, Yuhang Liu, Ziwei Zhang, Xiao Wang, Ruijie Wang Abstract Pre-training on text-attributed graphs (TAGs) is central to building transferable graph foundation models, where LLM-as-Aligner methods align graph and text representations through the semantic knowledge of large language models. However, these methods usually assume that node texts provide sufficient and reliable supervision, an assumption often violated in real-world sparse TAGs. When textual anchors are missing, noisy, or uneven across domains, graph structures must be aligned with weak semantic evidence, leading to unreliable structure–semantics correspondence and sparsity-induced transfer bias. This paper presents S 2 Aligner , a S parsity-aware and S tructure-enhanced LLM-as-Aligner framework for graph–text pre-training on sparse TAGs. The key idea is to decouple semantic alignment from structural modeling, allowing topology-aware signals to enhance alignment without contaminating the shared semantic space. Specifically, S 2 Aligner decomposes graph–text representations into semantic and structural components, uses structure-oriented reconstruction with consistency control to inject reliable topology cues into text representations, and suppresses inconsistent structural signals under textual sparsity. Moreover, S 2 Aligner introduces sparsity-aware cross-domain risk balancing, which calibrates domain risks through a global-domain density ratio and downweights unreliable sparse samples via graph reliability estimation. Theoretical analysis shows that this objective reduces cross-domain generalization gaps by controlling domain risk discrepancy. Extensive experiments across diverse graph domains, sparsity levels, and downstream tasks demonstrate that S 2 Aligner consistently outperforms existing baselines. 1 Introduction Graph foundation models (GFMs) have recently emerged as a promising paradigm for inductive graph learning. They aim to distill transferable structural and semantic priors from large-scale graph corpora, reducing the need for task-specific supervision and costly fine-tuning [ 18 , 12 , 27 ] . This paradigm is particularly suitable for text-attributed graphs (TAGs), common in citation networks, social platforms, and e-commerce systems [ 6 , 7 , 4 , 3 ] , where node texts provide rich semantics and edges capture relational structure. Recently, cross-modal graph-text alignment has become a central direction for TAG pretraining [ 37 , 43 , 19 ] , aligning graph or subgraph embeddings with textual embeddings, often leveraging large language models (LLMs) as aligners — the so-called LLM-as-Aligner paradigm (Figure 1 ). Despite its promise, the LLM-as-Aligner paradigm relies on a strong assumption: textual evidence should be sufficient to provide reliable supervision for graph-text alignment [ 37 , 43 , 19 ] . This assumption is often violated in real-world TAGs. Node texts can be missing, short, noisy, or highly uneven across domains. In this case, the text view becomes an incomplete anchor for the graph view. Rich graph structures must be aligned with weak and ambiguous semantic signals, making the learned structure–semantics correspondence unreliable. The problem is further amplified by recent methods that use LLM-generated subgraph summaries as supervision [ 43 , 19 ] . When textual contexts are sparse, LLMs have to infer missing semantics from incomplete evidence, which may introduce uncertainty and bias into the alignment signal. This raises a fundamental question: how can graph-text pretraining learn transferable structure–semantics correspondence when the cross-modal supervision itself is sparse and unreliable? This paper studies LLM-as-Aligner pretraining on sparse TAGs, preserving transferable pretraining while reducing dependence on dense textual supervision. It introduces two coupled challenges: Challenge 1: Unreliable structure–semantics alignment. In sparse text-attributed graphs, textual anchors are incomplete or ambiguous, while the graph structure remains rich. This imbalance can cause the model to misalign structural patterns with noisy text, failing to learn faithful structure–semantics correspondences. Figure 2 illustrates this issue: (a) text sparsity increases semantic uncertainty, weakening alignment anchors; (b) structural supplementation helps under dense text but may amplify mismatches under sparse text; (c) sparse-text graph–text retrieval performs substantially worse than full-text retrieval, highlighting the limited robustness of LLM-as-Aligner models. Challenge 2: Sparsity-induced transfer bias. Sparse texts also weaken zero-shot generalization. Different domains require different densities of alignment signals, since some domains can transfer with coarse semantic anchors while others depend on finer graph–text correspondences. Text sparsity makes such reliability uneven across domains. Standard pretraining may therefore overfit to domains with sufficient anchors or learn biased correlations from noisy ones, causing imbalanced domain risks and unstable transfer to unseen sparse TAGs. Thus, robust cross-domain generalization requires sparsity-aware risk balancing to extract domain-invariant representations under sparse supervision. Figure 1 : LLM-as-Aligner. (a) Uncertainty vs. Sparsity (b) Structural supplementation (c) Full vs. sparse retrieval Figure 2 : (a) Summary uncertainty increases. (b) Naive structural supplementation becomes harmful. (c) Full vs. Sparse alignment. To address the above challenges, we propose S 2 Aligner , a sparsity-aware and structure-enhanced LLM-as-Aligner framework for graph–text pretraining on sparse TAGs. The overall idea is to decouple semantic alignment from structural modeling, so that sparse and uncertain structural cues can be used to enhance graph–text alignment without directly contaminating the shared semantic space. For Challenge 1 , S 2 Aligner separates each graph–text representation into semantic and structural components. The semantic components define the main alignment space, while the structural components are modeled through structure-oriented reconstruction. By using reconstruction consistency to control structural enhancement, S 2 Aligner injects reliable topology-aware signals into text representations and suppresses inconsistent structural cues, thereby mitigating structural negative transfer under textual sparsity. For Challenge 2 , S 2 Aligner introduces a sparsity-aware cross-domain risk balancing mechanism. It calibrates domain risks with a global-domain density ratio and further uses graph reliability to reduce the impact of unreliable sparse samples. This encourages the model to focus on domain-invariant graph–text correspondences rather than sparse domain-specific correlations. Theoretical analysis suggests that this objective can reduce the cross-domain generalization gap by controlling domain risk discrepancy. Our contributions are summarized as follows: • We propose S 2 Aligner, a sparsity-aware and structure-enhanced LLM-as-Aligner framework for sparse TAGs. It transforms uncertain topological signals into verifiable structural enhancements and incorporates sparsity-aware risk balancing for better cross-domain generalization. • We design a sparsity-aware cross-domain risk balancing mechanism. It filters noisy samples using graph reliability evaluation and calibrates cross-domain risks via a global-domain density ratio, reducing domain-biased alignment and improving generalization. • Extensive experiments validate our method across sparsity levels, graph domains, and downstream tasks. Notably, even with only 10\% textual attributes, it surpasses baselines trained with full attributes, demonstrating superior sparsity generalization and data efficiency. 2 Preliminary LLM-as-Aligner for graph problems. Following GraphCLIP [ 43 ] , we define a text-attributed graph (TAG) as \mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{T}) , where \mathcal{V} and \mathcal{E} denote the node and edge sets, and \mathcal{T}=\{(g_{i},t_{i})\}_{i=1}^{M} denotes a collection of graph-text pairs. Here, g_{i} can be a node, a sampled subgraph, or an entire graph, and t_{i} is the associated textual information, such as a textual attribute, structural summary, or label prompt. Given a graph encoder f_{\mathrm{G}} and a text encoder f_{\mathrm{T}} , each pair is encoded as \mathbf{z}_{g,i}=\operatorname{READOUT}\!\left(f_{\mathrm{G}}(g_{i})\right),\qquad\mathbf{z}_{t,i}=\operatorname{READOUT}\!\left(f_{\mathrm{T}}(t_{i})\right), (1) For a mini-batch \mathcal{T} , graph and text representations are aligned by a symmetric contrastive loss: \mathcal{L}_{\mathrm{align}}(\mathcal{T})=\frac{1}{2|\mathcal{T}|}\sum_{(g_{i},t_{i})\in\mathcal{T}}\left[-\log\frac{\exp\left(\operatorname{sim}(\mathbf{z}_{g,i},\mathbf{z}_{t,i})/\tau\right)}{\sum_{j}\exp\left(\operatorname{sim}(\mathbf{z}_{g,i},\mathbf{z}_{t,j})/\tau\right)}-\log\frac{\exp\left(\operatorname{sim}(\mathbf{z}_{g,i},\mathbf{z}_{t,i})/\tau\right)}{\sum_{j}\exp\left(\operatorname{sim}(\mathbf{z}_{g,j},\mathbf{z}_{t,i})/\tau\right)}\right], (2) where \operatorname{sim}(\cdot,\cdot) denotes cosine similarity and \tau is a temperature parameter. This objective treats the matched graph–text pair as the positive pair and all other pairs in the mini-batch as negatives. Sparse text-attributed graph problem. In real-world TAGs, textual attributes are often missing, incomplete, or unreliable. We

Read the complete text