DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs

Lexuan Liang, Tao Zou, Xuxiang Ta, Zekun Qiu · 2026

Read DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs by Lexuan Liang, Tao Zou, Xuxiang Ta, Zekun Qiu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17411.txt.

Introduction

Liang, Zou, Ta, and Qiu propose DuConTE, a text encoder for graphs whose nodes carry text, such as documents linked by citations. Most methods encode words alone and ignore how texts on connected nodes relate. DuConTE cascades two pretrained language models, first at word-token level and then at node level, while graph connectivity shapes the attention masks. Tokens are also weighed under both the central node and its neighborhood, and tests on several benchmarks give state-of-the-art results on most of them.

Catalog record

CREATORLexuan Liang, Tao Zou, Xuxiang Ta, Zekun Qiu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73027
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17411

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs Lexuan Liang, Tao Zou, Xuxiang Ta, Zekun Qiu Abstract Text-attributed graphs integrate semantic information of node texts with topological structure, offering significant value in various applications such as document classification and information extraction. Existing approaches typically encode textual content using language models (LMs), followed by graph neural networks (GNNs) to process structural information. However, during the LM-based text encoding phase, most methods not only perform semantic interaction solely at the word-token granularity, but also neglect the structural dependencies among texts from different nodes. In this work, we propose DuConTE, a dual-granularity text encoder with topology-constrained attention. The model employs a cascaded architecture of two pretrained LMs, encoding semantics first at the word-token granularity and then at the node granularity. During the self-attention computation in each LM, we dynamically adjust the attention mask matrix based on node connectivity, guiding the model to learn semantic correlations informed by the graph structure. Furthermore, when composing node representations from word-token embeddings, we separately evaluate the importance of tokens under the center-node context and the neighborhood context, enabling the capture of more contextually relevant semantic information. Extensive experiments on multiple benchmark datasets demonstrate that DuConTE achieves state-of-the-art performance on the majority of them. 1 Introduction Text-attributed graphs ( 27 ; 21 ) have emerged as an increasingly significant research domain, with substantial applications in real-world scenarios such as social media analysis ( 21 ) , academic citation systems ( 25 ) , and knowledge base construction ( 29 ) . In such graphs, each node is associated with a piece of textual content, resulting in richly structured data that encapsulates both semantic text information and topological structure. Learning high-quality representations that effectively capture both the textual and structural characteristics of nodes is crucial for downstream tasks such as node classification ( 32 ) . Recently, a growing body of research ( 3 ; 4 ; 34 ) has begun leveraging Transformer-based language models (LMs) to model textual information in text-attributed graphs, aiming to enhance graph neural networks (GNNs). Thanks to their strong pre-trained understanding of natural language, LMs can produce highly expressive representations of textual content. For example, GraphBridge ( 24 ) attempts to combine the text from the center-node and its neighbors into the LM, enabling the model to jointly encode the central text and its contextual information from neighboring nodes. Current approaches ( 34 ; 9 ; 12 ) that jointly employ GNNs and LMs largely follow a common paradigm: the LM is responsible for encoding textual features, while the GNN focuses on capturing structural information. Figure 1: Overview of the text-attributed graph learning pipeline (top) and comparison between existing methods and the proposed DuConTE (bottom). However, existing approaches typically perform semantic interaction only at the word-token granularity when using LMs for text encoding, failing to capture meaningful node-granularity semantic interactions—where the textual content of different nodes is treated as holistic units and interacts across the graph. Moreover, current methods either do not incorporate structural information into the LM at all, or the injected structural signals are insufficient to guide the encoding process effectively. Additionally, existing methods lack an effective mechanism for composing node representations from word-token embeddings. To address these limitations, we propose DuConTE , a dual-granularity text encoder with topology-constrained attention for text-attributed graphs. As illustrated in the top panel of Figure 1 , the text-attributed graph learning pipeline consists of three stages, with DuConTE acting as a plug-and-play text encoder module. It takes as input the text of each node and its sampled neighborhood structure (e.g., from random walks or k-hop sampling), obtained through upstream preprocessing, and outputs enriched node representations for downstream GNN models. DuConTE performs dual-granularity semantic encoding , in which two pretrained LMs sequentially encode textual semantics at the word-token and node granularities, respectively. This design aligns with the inherent multi-granular nature of text-attributed graphs, allowing for a more complete capture of textual semantics. During the encoding process, DuConTE employs a topology-constrained attention mechanism to leverage graph structural information for enhanced text encoding. This is achieved through an attention masking strategy specifically designed for TAG, motivated by the homophily analysis in Section A , enabling pretrained LMs to better process graph-structured textual data without architectural modification. Furthermore, we design a node representation composer that assesses the importance of individual word tokens under both center-node and neighborhood semantic contexts. This enables the model to capture salient semantic information more effectively when composing node representations from word-token embeddings. • We propose DuConTE , a dual-granularity text encoder with topology-constrained attention for text-attributed graphs. It performs dual-granularity semantic encoding to model textual semantics at both the word-token granularity and node granularity , capturing a comprehensive, multi-scale understanding of the text-attributed graph. • We introduce a topology-constrained attention mechanism that leverages an attention masking strategy, specifically designed for TAGs and grounded in the homophily analysis in Section A , to effectively incorporate structural guidance into the textual encoding process. • We design a node representation composer that distinctly models token importance under center-node and neighborhood contexts, enabling effective fusion of word-token embeddings into comprehensive node representations. 2 Related Work 2.1 Text-attributed graph learning Learning on text-attributed graphs has evolved from employing simple text features like Bag-of-Words ( 31 ) to sophisticated methods centered on language models (LMs) ( 3 ; 4 ; 34 ) . These modern approaches generally follow two main paradigms. The first relies on a single, powerful LM to jointly process text and structure. For instance, LLaGA ( 2 ) injects structural information by mapping it into the LM’s token space and relies solely on the LM to generate predictions. While conceptually unified, this paradigm is often computationally demanding, suffers from poor scalability, and achieves limited effectiveness in leveraging structural information. The second, more common paradigm, employs a hybrid LM-GNN pipeline where an LM first serves as a text encoder, and a subsequent GNN performs the downstream task using the resulting node embeddings. Representative works like GraphBridge ( 24 ) enrich node text with neighbor semantics before encoding, whereas Engine ( 34 ) uses a GNN to process features from multiple LM layers. A critical limitation across most hybrid models is that the LM encoding process remains largely unaware of the graph topology. This decoupled approach hinders the deep fusion of structural and semantic information, a key challenge we address in this work. 2.2 Transformers for Modeling Structured Data In recent years, numerous studies have leveraged Transformers to process graph-structured data ( 22 ) . An early effort in this direction is Graph-BERT ( 30 ) , which applies a BERT-style Transformer to sampled subgraphs without relying on message passing. More recent approaches further enhance structural awareness: Graphormer ( 28 ) enhances the Transformer’s understanding of graph structures by introducing spatial encoding and degree encoding. Another work NeuralWalker ( 1 ) generates serialized representations of graphs through random walks to exploit the self-attention mechanism of Transformers for modeling purposes. Edge-augmented methods ( 17 ; 19 ) explicitly model edge features to enhance the Transformer’s sensitivity towards different edge types. Masked Graph Modeling ( 10 ; 23 ) employs a masking strategy to learn structural information by predicting masked

Read the complete text