LLM-driven Knowledge Distillation for Dynamic Text-Attributed Graphs

Amit Roy, Ning Yan, Masood Mortazavi · 2025

Read LLM-driven Knowledge Distillation for Dynamic Text-Attributed Graphs by Amit Roy, Ning Yan, Masood Mortazavi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10914.txt.

Introduction

Amit Roy, Ning Yan, and Masood Mortazavi teach graph neural networks to read networks that change over time and carry text, in a paper from 2025. Such graphs appear as social, citation, and review networks, where each post, paper, or comment brings words and where links keep forming. Tasks like predicting future links need features that blend structure, timing, and text.

Their method borrows knowledge from a large language model and distills it into the graph model. A simple trick stamps each link with its time so that graph layers absorb timing and structure together. The language model supplies the text understanding that graph networks lack. The authors test the design on the standard prediction and classification tasks for these evolving networks.

Catalog record

CREATORAmit Roy, Ning Yan, Masood Mortazavi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10914

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-driven Knowledge Distillation for Dynamic Text-Attributed Graphs Amit Roy, Ning Yan, Masood Mortazavi Abstract Dy namic T ext- A ttributed G raphs ( DyTAG s) have numerous real-world applications, e.g., social, collaboration, citation, communication, and review networks. In these networks, nodes and edges often contain text descriptions, and the network structure can evolve over time. Future link prediction, edge classification, relation generation, and other downstream tasks on DyTAGs require powerful representations that encode structural, temporal, and textual information. Graph Neural Networks (GNNs) are adept at managing structural data, yet encoding temporal information in dynamic graphs has proven challenging. In this work, we propose L LM-driven K nowledge D istillation for Dy namic T ext A ttributed G raph ( LKD4DyTAG ) with temporal encoding to address those challenges. We first use a simple yet effective approach to encode temporal information in edges so that graph convolution can simultaneously capture both temporal and structural information in the hidden representations. To leverage LLM’s text processing capabilities to learn richer representations on DyTAGs, we distill knowledge from LLM-driven edge representations (based on a neighborhood’s text attributes) into saptio-temporal representations learned by a lightweight GNN model that encodes temporal and structural information. Our knowledge distillation objective enables the GNN to learn representations that more effectively encode available structural , temporal and textual information in DyTAGs. Extensive experimentation conducted on six real-world dynamic text-attributed graph datasets prove the efficacy of our approach LKD4DyTAG for the future link prediction and edge classification task. The results show that our approach significantly improves the performance of downstream tasks compared to baseline models. Figure 1: Example Dynamic Text-Attributed Graph in the context of 2024 Presidential Election in USA. Each entity is described with text, while the relations between them are represented by edges including temporal information. Introduction Dy namic T ext- A ttributed G raph ( DyTAG ) structures change over time, incorporating text descriptions in both nodes and edges. DyTAGs have a wide range of applications across various domains ( Tang et al. 2023 ; Luo et al. 2023 ; Zhang, Shao, and Cui 2023 ; Cai et al. 2022 ; Skarding, Gabrys, and Musial 2021 ; Kazemi et al. 2020 ) , such as email networks, political event networks, online Q&A forums, and review networks for products and movies. For instance, in an email network, nodes can represent email users with their profile information, while edges can represent the emails exchanged between them at specific times. Similarly, in a movie or item review network, nodes can describe users, movies, or items, while edges may denote reviews between users and movies or items at different times. Again, in a political event network nodes can denote the political figures, places, institution and the edges may indicate the interrelationships between those entities with temporal information attached. Handling both the dynamically changing graph structure and the semantic information in nodes and edges described in natural language is a challenging task since structure and text have different modality. When designing deep learning models for various downstream tasks for such text attributed dynamic graph data, it is crucial to carefully consider structural, temporal, and textual information. Dynamic networks are often represented as snapshots with discrete timestamps. Recent works propose various methods to capture temporal information in dynamic networks, such as anonymous random walk-based approach ( Wang et al. 2021b ) and window-based approach ( Alomrani et al. 2023 ) . However, when the number of timestamps is extensively large and the downstream task depends on earlier timestamps, it becomes challenging to efficiently encode the complete temporal information into the latent representation. Additionally, as the graph structure evolves over time, it is also essential to include temporal information in graph convolution as well as structural information. In addition to temporal and structural information found in dynamic graphs, Dynamic Text-Attributed Graphs(DyTAGs) also incorporate textual information. This added text provides a more comprehensive representation of real-world systems compared to dynamic graphs that only contain structural and temporal data. With the increasing capability of LLMs to produce semantic representations of text, numerous works have emerged that utilize them for graph tasks ( Fang et al. 2024b ; Xu et al. 2024 ) . Most of these works rely on prompt engineering to describe the graph structure and the downstream task in order to “tune” the LLMs, expecting the LLMs to generate the desired results for the given task. However, despite their advanced capabilities, LLMs face challenges in scalability and are limited in their ability to process input prompts of longer sequence lengths. Consequently, their applicability to diverse downstream tasks is constrained. Furthermore, despite LLMs versatility in capturing semantic information from text, they often try to memorize the common structural motifs/patterns and are not effective in comprehending the underlying evolving structure and the complex temporal dynamics of Dynamic Text-Attributed Graphs (DyTAGs). To address these challenges, one can deploy the capabilities of LLMs for scalable processing of all locally-scoped text attributes in DyTAGs and transfer representations thus learned by heavyweight large language models to lightweight graph neural networks models (GNNs) for downstream tasks involving DyTAGs. Here, text-based edge representation generated by the LLM serves as a guide for the spatio-temporal edge representation produced by temporally encoded GNNs. This approach ensures that, among the three types of information present in DyTAGs, temporal and structural information are captured by the temporally encoded GNN, while the LLMs are utilized to encode the textual information of edges for distillation purposes. This approach leads to our design of a novel LLM-driven knowledge distillation framework for processing Dynamic Text-Attributed Graphs (DyTAGs) with temporal encoding. The structure and temporal information of edges are transformed into spatio-temporal edge representations using GNNs with a simple yet efficient encoding of time. For the textual edge representation, we describe the neighbors of adjacent nodes to the LLMs one edge at a time and sum these descriptions to obtain neighborhood textual embeddings. Next, we add up the neighborhood textual embeddings of adjacent node with the LLM output for the corresponding edge’s description to obtain a text-based edge representation. Next, we bring into alignment the spatio-temporal edge representation and the text-based edge representation. In short, we are using knowledge distillation method to transfer knowledge from the text-based edge representation from a teacher LLM model to the spatio-temporal representation of a student GNN model with temporal encoding for dynamic text attributed graph. Our contributions in this work are as follows: • We propose a novel framework LKD4DyTAG which integrates time, structure, and text information in Dynamic Text-Attributed Graphs to produce suitable representations for downstream tasks. • We designed a simple yet effective approach to produce spatio-temporal edge representations of DyTAG that capture the temporal and structural information using temporal edge encoding with GNNs. • To make the best utilization of textual information from DyTAG and the text processing capability from LLMs, we perform knowledge distillation from text-based edge representation to spatio-temporal edge representation. • We perform extensive experiments on six real-world benchmark dynamic text-attributed graph datasets and show that our proposed method LKD4DyTAG can outperform state-of-the-art approaches in certain downstream tasks such as future link prediction and edge classification . Figure 2: LKD4DyTAG : LLM-driven Knowledge Distillation for Dynamic Text-Attributed Graph with temporal

Read the complete text