Data-Driven Self-Supervised Graph Representation Learning

Ahmed E. Samy, Zekarias T. Kefato, Šarūnas Girdzijauskas · 2024

Read Data-Driven Self-Supervised Graph Representation Learning by Ahmed E. Samy, Zekarias T. Kefato, Šarūnas Girdzijauskas (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18316.txt.

Introduction

Ahmed E Samy and co-authors present a data-driven way to learn from graphs without labels in a 2024 paper. Graphs model networks of linked things, and self-supervised learning trains on the graph itself to avoid costly hand labeling. A key step distorts the graph to make extra training views, but the usual tricks are hand-picked and can ruin the signal for some tasks.

Their method learns the distortion from the graph's own data instead. One part learns how to vary node features, and another learns a higher-order view of the link pattern, both trained jointly with the main model. Across many data sets for node and whole-graph tasks, the method matched or beat leading rivals and neared methods that use some labels.

Catalog record

CREATORAhmed E. Samy, Zekarias T. Kefato, Šarūnas Girdzijauskas
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18316

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Data-Driven Self-Supervised Graph Representation Learning Ahmed E. Samy, Zekarias T. Kefato, Šarūnas Girdzijauskas Abstract Self-supervised graph representation learning (SSGRL) is a representation learning paradigm used to reduce or avoid manual labeling. An essential part of SSGRL is graph data augmentation. Existing methods usually rely on heuristics commonly identified through trial and error and are effective only within some application domains. Also, it is not clear why one heuristic is better than another. Moreover, recent studies have argued against some techniques (e.g., dropout : that can change the properties of molecular graphs or destroy relevant signals for graph-based document classification tasks). In this study, we propose a novel data-driven SSGRL approach that automatically learns a suitable graph augmentation from the signal encoded in the graph (i.e., the nodes’ predictive feature and topological information). We propose two complementary approaches that produce learnable feature and topological augmentations. The former learns multi-view augmentation of node features, and the latter learns a high-order view of the topology. Moreover, the augmentations are jointly learned with the representation. Our approach is general that it can be applied to homogeneous and heterogeneous graphs. We perform extensive experiments on node classification (using nine homogeneous and heterogeneous datasets) and graph property prediction (using another eight datasets). The results show that the proposed method matches or outperforms the SOTA SSGRL baselines and performs similarly to semi-supervised methods. The anonymised source code is available at https://github.com/AhmedESamy/dsgrl/ 1 Introduction Figure 1: The Architecture of dsgrl . The left figure (a) is based on learnable feature augmentation, where we only augment node features, \mathbf{X} . Two learnable augmenters f_{\Theta_{1}}(\mathbf{X}) and f_{\Theta_{2}}(\mathbf{X}) are applied on \mathbf{X} to obtain the learned augmentations \mathbf{X}_{1} and \mathbf{X}_{2} , respectively. The middle figure (b) is used for learning topological augmentation. Two augmentations, an identical, t_{\Phi}(\mathbf{A}) , and learned ones, t_{\Phi^{\prime}}(\mathbf{A}) , are respectively applied on the adjacency matrix to obtain \mathbf{A} and a high-order network \mathbf{A}^{\prime} . The right figure (c) is used for combining learnable feature and topological augmentations. The augmentation results in two views G_{1}=(\mathbf{A}_{1},\mathbf{X}_{1}) and G_{2}=(\mathbf{A}_{2},\mathbf{X}_{2}) from the augmentation space, where \mathbf{A}_{1},\mathbf{X}_{1},\mathbf{A}_{2},\mathbf{X}_{2} are set to different values as shown in the above figures based on the augmentation type. Finally, a shared GNN encoder h_{\Theta} is applied on the views, h_{\Theta}(\mathbf{A}_{1},\mathbf{X}_{1}) and h_{\Theta}(\mathbf{A}_{2},\mathbf{X}_{2}) , respectively, to obtain the latent representations \mathbf{Z}_{1} and \mathbf{Z}_{2} . The data modality ( \mathbf{A} or \mathbf{X} ) that is not affected by an augmenter is blurred. Self-supervised graph representation learning (SSGRL) has been successfully used for graph representation learning (GRL) [ 26 , 12 , 37 , 17 , 29 ] in various domains where labeled data is scarce and manual label is expensive. It has recently attracted interest across domains by achieving a competitive performance when compared to semi-supervised approaches. Considering the scarcity of labeled data, SSGRL has emerged as a new paradigm that narrows down the performance gap between the unsupervised and semi-supervised learning methods. Self-supervised learning (SSL) is commonly formulated as a predictive or contrastive learning [ 46 ] . For predictive models [ 7 ] , one first defines a related task on which an SSL model is pre-trained to extract meaningful patterns. The pre-trained model is subsequently refined (fine-tuned) on a relevant but specific task of interest. Typically, an SSL model is pre-trained over large data as a starting point. The quintessential models, particularly from NLP, are the ones that are pre-trained on masked word prediction tasks and are fine-tuned on other relevant tasks, such as text classification or translation. On the other hand, contrastive models learn based on augmented views of a data point (e.g., image, graph) that are generated by applying a meaningful perturbation on the original data point. The representation of a data point is then learned by maximizing the mutual information between latent representations obtained from its augmented views. The main challenge here is to produce augmented views of the data points. The key to learning high-quality representations based on augmentation is that the perturbations should preserve semantics [ 49 , 1 , 38 , 6 ] . For instance, a perturbation applied to an image of a dog should preserve “dogness” . Effective augmentation techniques for images (e.g., rotation, flipping, resizing) allow learning high-quality visual representations because they preserve the semantics of the original image. This is also true for SSL techniques in NLP [ 7 , 23 , 21 ] , (e.g. synonym augmentation and word masking), such techniques do not alter the meaning of the original sentence. Due to the complex nature of graph data, it is much more challenging to find appropriate techniques for augmenting graphs. While some techniques are proposed, there is no standard technique that works well for graphs in different domains [ 39 , 53 , 35 , 48 , 38 , 40 , 14 , 18 ] . Consequently, most efforts rely on finding a heuristic by trial and error to identify a suitable augmentation for the graph at hand. Generally, there are two classes of perturbations, which either corrupt the topology of the graph or node features. The topology can be corrupted by dropping nodes and edges or adding new edges either randomly or through a diffusion process [ 40 , 53 , 35 , 14 ] . Similarly, dropout, masking, and permutation techniques have been used for corrupting node features [ 35 , 18 ] . Nonetheless, it is unclear why a particular augmentation technique works better. A study [ 39 ] has shown that these strategies are susceptible to destroying task-relevant information. Furthermore, in some cases, e.g., for molecular graphs, dropout techniques alter the semantics of the graph [ 46 ] . The effectiveness of such techniques usually comes not from the graph augmentations but from the strong inductive bias of the underlying learning algorithm, particularly Graph Neural Networks (GNNs) [ 39 ] . In this paper, we follow a data-driven approach, where the augmentation process is guided by the inherent signal encoded in the graph. Such an approach establishes obvious benefits, first as one can avoid trial and error in identifying a suitable augmentation mechanism. Second, it provides a flexible framework that can be adapted to different domains. Thus, we propose a novel D ata-driven S elf-supervised G raph R epresentation L earning ( dsgrl ) method. dsgrl is data-driven because it jointly learns the augmentation with the representation. Similar to existing methods, we aim to augment either the topology or node features; however, unlike them dsgrl learns both augmentations from the data. dsgrl is a general approach that can be applied to both homogeneous and heterogeneous graphs (i.e., graphs containing multiple node/edge types). Generally, for a given graph G , and a family of augmentation heuristics \mathcal{A} , existing methods apply either a topological, A_{t}\sim\mathcal{A} , or feature, A_{f}\sim\mathcal{A} , augmentation sampled from \mathcal{A} . However, dsgrl does not rely on \mathcal{A} , instead it learns A_{t_{\Phi}} and A_{f_{\Theta}} ; where \Phi and \Theta are the learnable parameters of the topological and feature augmenters, respectively. We materialize t_{\Phi} using a GNN to obtain high-order node features; based on similarity scores between these features, we compute a high-order weighted network as a topological augmentation. The feature augmentation, f_{\Theta} , is simply

Read the complete text