Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformer
Chaoqun Cui, Caiyan Jia · 2026
Read Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformer by Chaoqun Cui, Caiyan Jia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22854.txt.
Introduction
Cui and Jia tackle over-smoothing in graph-based rumor detection on social media. Most nodes in rumor spread trees sit at level one, they show. That shape makes graph networks blur node features and miss long-range links. Their fix, P2T3, is a pure transformer pretrained on large unlabeled data. It pulls all conversation chains from each tree, in reply order. Token embeddings carry the connection data. Tests beat prior top methods on several benchmarks. The method also works well with few labeled examples.
Catalog record
| CREATOR | Chaoqun Cui, Caiyan Jia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41834 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22854
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformer Chaoqun Cui, Caiyan Jia Abstract Deep learning techniques for rumor detection typically utilize Graph Neural Networks (GNNs) to analyze post relations. These methods, however, falter due to over-smoothing issues when processing rumor propagation structures, leading to declining performance. Our investigation into this issue reveals that over-smoothing is intrinsically tied to the structural characteristics of rumor propagation trees, in which the majority of nodes are 1-level nodes. Furthermore, GNNs struggle to capture long-range dependencies within these trees. To circumvent these challenges, we propose a Pre-Trained Propagation Tree Transformer (P2T3) method based on pure Transformer architecture. It extracts all conversation chains from a tree structure following the propagation direction of replies, utilizes token-wise embedding to infuse connection information and introduces necessary inductive bias, and pre-trains on large-scale unlabeled datasets. Experiments indicate that P2T3 surpasses previous state-of-the-art methods in multiple benchmark datasets and performs well under few-shot conditions. P2T3 not only avoids the over-smoothing issue inherent in GNNs but also potentially offers a large model or unified multi-modal scheme for future social media research. 1 Introduction Currently, deep learning-based methods for rumor detection can be broadly classified into four categories: time-series based methods 39 ; 66 ; 37 , propagation structure learning methods 41 ; 5 ; 62 , multi-source integration methods 24 ; 7 , and multi-modal fusion methods 61 ; 22 . Among these, propagation structure learning methods, which leverage the wisdom of crowds to model the relations between posts, have shown promising results in debunking rumors 26 ; 5 ; 56 . Numerous studies highlight the value of propagation structures in revealing inter-post relations for rumor detection 56 ; 12 . Propagation structure learning methods represent the propagation process of each claim as a tree, which can be a top-down or bottom-up directed or undirected graph with the source post as the root node, comments as other nodes, and reply relations as edges. Graph Neural Networks (GNNs) are then used to learn the representations of rumor propagation trees (RPTs). Two examples of rumor propagation trees are shown in Fig. 6 . However, our investigation reveals that certain structural characteristics of RPTs pose serious over-smoothing risks for GNNs. Indeed, in our experiments, GNNs showed over-smoothing signs when handling RPTs. This manifests as a decline in the performance of the GNNs when faced with the undirected graph of RPT, as opposed to adopting top-down or bottom-up directed graph. This is particularly contradictory since earlier research 5 has already proven the effectiveness of bidirectional information in RPTs. Moreover, some GNNs, or those dedicated rumor detection models, display a phenomenon where the model’s accuracy decreases as the model depth increases. These experimental phenomena demonstrate that GNNs tend to encounter over-smoothing problems when dealing with RPTs. It especially hinders the expansion of model scale during pre-training on large-scale unlabeled social media data. We identified the causes of over-smoothing by investigating RPT structures. We found that in the propagation process of social posts, the vast majority of comments are direct replies to source post. This results in the majority of nodes in a RPT being 1-level nodes directly connected to the root, while the proportion of nodes at deeper levels is quite small. Because the scope of a node’s neighborhood view in GNNs is tied to the model depth 29 ; 63 , all 1-level nodes in a tree-like structure like a RPT are in each other’s 2-hop neighborhoods. If neighborhood aggregation is conducted indiscriminately, it leads to over-smoothing. In other words, the structure of graphs like RPTs inherently predisposes GNNs to over-smoothing. In addition, GNNs struggle to capture long-range dependencies 64 in RPTs, making it challenging to leverage information from the few deep nodes in the tree. Specifically, GNNs find it hard to learn from complete conversation chains in RPTs (conversation threads from 1-level node to leaf node), and such chains are better suited for Transformer 58 architecture designed for sequential structure. As a result, we chose Transformer as underlying architecture for our model to harness its self-attention mechanism, avoiding potential over-smoothing issues and facilitating modeling of user interactions throughout complete conversation chains. In this study, we propose P re- T rained P ropaga- tion T ree T ransformer (P2T3). P2T3 extracts conversation chains from deep structure of RPTs following the node propagation direction to form sequential structures. It uses token-wise embedding to infuse connection information into the sequence tokens, introduces necessary inductive bias, and pre-trains on large-scale unlabeled datasets to enhance the performance. Experiments shows that P2T3 outperforms previous state-of-the-art (SOTA) methods on multiple benchmark datasets. In summary, this study contributes as follows: • We ran extensive experiments to reveal over-smoothing phenomenon and its intrinsic causes on RPTs. • We released two large unlabeled topic datasets, which may promote semi-supervised rumor detection research. • We proposed the P2T3 method, where the special token-wise embedding enables Transformer to handle RPTs. • Experiments show that P2T3 outperforms current SOTA methods and performs well in few-shot scenarios. 2 Empirical Investigation 2.1 Over-smoothing Phenomenon The over-smoothing phenomena of GNNs when handling RPTs mainly manifest in two aspects: (1) Useing undirected graphs of RPTs to learn bidirectional propagation information actually impedes the improvement of model performance; (2) The expansion of the model scale, by increasing in the number of model layers, leads to a performance decrease. We will elaborate on these two aspects. 2.1.1 Impact of Infomation Flow Direction We conducted experiments on four rumor detection datasets including Weibo 39 , DRWeibo 12 , Twitter15, and Twitter16 40 , following Bian et al. (2020) 5 . We employed GCNs in four settings: undirected graphs, top-down directed graphs, bottom-up directed graphs, and a combination of top-down and bottom-up graphs, labeled as UD-GCN, TD-GCN, BU-GCN, and Bi-GCN respectively. UD-GCN uses one GCN encoder, while Bi-GCN uses two GCN encoders on top-down and bottom-up graphs. Results is shown in Fig. 1 . (a) Weibo (b) DRWeibo (c) Twitter15 (d) Twitter16 Figure 1: The impact of information flow in different directions. The results reveal two phenomena: (1) Bi-GCN shows a significant improvement compared to TD-GCN and BU-GCN, which only use top-down or bottom-up information. (2) UD-GCN does not show a substantial improvement over TD-GCN and BU-GCN, and performance even decreases on certain datasets (DRWeibo and Twitter15). The improved performance of Bi-GCN compared to TD-GCN and BU-GCN suggests that both top-down and bottom-up information in RPTs are useful. However, the effect of using UD-GCN to learn this bidirectional information is not ideal and may even result in performance degradation. These phenomena reflect a contradictory fact, namely, while bidirectional information is beneficial, learning from undirected graphs can lead to adverse effects. The ablation studies by BiGCN 5 and RAGCL 12 also demonstrate similar phenomena. 2.1.2 Deep Model Performance Degradation We explored the effect of varying GNN layer numbers on performance using undirected RPT graphs (see Fig. 2 ). It suggests that the optimal GNN layer count for Weibo and DRWeibo is three, while it’s two for Twitter15 and Twitter16. Performance declines to different degrees as layer number increases. In other words, the model scale expansion actually hinders model performance improvement, which is a typical over-smoothing phenomenon 31 ; 9 . Among the three