Token-Level Graphs for Short Text Classification
Gregor Donabauer, Udo Kruschwitz · 2024
Read Token-Level Graphs for Short Text Classification by Gregor Donabauer, Udo Kruschwitz (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12754.txt.
Introduction
Donabauer and Kruschwitz sort short texts with token-level graphs in a 2024 paper. Short posts are hard to classify with few examples. Graph methods help in low-data cases but often freeze word meaning or lock in test data. Their method builds the graph straight from tokens of a pretrained language model, so each word node carries its context.
The setup dodges fixed word lists and lets one word hold many senses. It also trains with fewer weights than full model tuning, which steadies learning from small samples. Across tests the method matched or beat prior graph and tuning routes. Code is shared for others to check and reuse.
Catalog record
| CREATOR | Gregor Donabauer, Udo Kruschwitz |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12754
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Token-Level Graphs for Short Text Classification Gregor Donabauer, Udo Kruschwitz Abstract The classification of short texts is a common subtask in Information Retrieval (IR). Recent advances in graph machine learning have led to interest in graph-based approaches for low resource scenarios, showing promise in such settings. However, existing methods face limitations such as not accounting for different meanings of the same words or constraints from transductive approaches. We propose an approach which constructs text graphs entirely based on tokens obtained through pre-trained language models (PLMs). By applying a PLM to tokenize and embed the texts when creating the graph(-nodes), our method captures contextual and semantic information, overcomes vocabulary constraints, and allows for context-dependent word meanings. Our approach also makes classification more efficient with reduced parameters compared to classical PLM fine-tuning, resulting in more robust training with few samples. Experimental results demonstrate how our method consistently achieves higher scores or on-par performance with existing methods, presenting an advancement in graph-based text classification techniques. To support reproducibility of our work we make all implementations publicly available to the community 1 1 1 https://github.com/doGregor/TokenGraph . 1 Introduction Short text classification is considered an important subtask in many Information Retrieval (IR) pipelines, and examples include assigning a label or category to snippets in search results or to tweets, status updates, comments, and reviews from social platforms [ 19 ] . Over the years, various methods have been proposed to improve the accuracy and efficiency of this task and fine-tuning of pre-trained language models (PLMs) is among the most popular approaches nowadays [ 3 ] . However, in situations where labeled data is scarce, such models often perform poorly or unstable [ 28 ] . Driven by progress in graph machine learning [ 11 , 7 ] , it has recently become increasingly popular to employ graph-based approaches on the task of short text classification that show more robust performance in low resource settings e.g., [ 23 , 31 , 27 ] . However, many existing approaches are facing certain limitations: Some rely on large networks of text documents to classify nodes within these graphs, making the approach transductive [ 13 , 23 , 12 , 26 ] . Others adopt an inductive approach but are constrained by different drawbacks, such as stopword removal, which further truncates already concise short texts [ 30 , 31 , 25 ] . Additionally, most methods depend on word representations (e.g., pre-trained embeddings like GloVe [ 16 ] and word2vec [ 14 ] , or one-hot vectors), lacking the ability to capture context-dependent word meanings [ 30 , 4 ] . To address these constraints, we introduce an approach wherein text preprocessing for constructing text graphs is based on PLM tokenization. While the idea of combining PLMs and graph-based text classification is not novel [ 12 , 26 ] our method allows to go beyond limitations of existing approaches (such as transductive settings) by breaking the construction of text graphs down to the token level of PLMs. We propose to feed text samples through the PLM’s tokenizer, breaking them into word units. Subsequently, these tokens are linked within a n -hop fully connected graph, with each text sample representing a distinct graph. The embeddings of token nodes within each graph are derived by feeding the tokens through the PLM. As a result, our approach represents an inductive setting (each sample is represented by a distinct graph) and maintains the positive implications of using PLMs for text representation. We can embed nearly any text since the vocabulary of the PLM is more fine-grained (tokens instead of words) which allows us to also tokenize unknown words into smaller word units. Furthermore, the embeddings of these tokens are context-dependent, as they are embedded within the complete sequence using the PLM. The embeddings also represent semantic information between words, inherent in the PLM’s weights. During training our approach easily allows to reduce the number of parameters in the graph neural network (GNN) that can be used to classify the token-graphs compared to fine-tuning a PLM. This makes overfitting less probable and improves the training robustness. Nonetheless, the aggregation process remains similar to fine-tuning a PLM, as it has been shown that Transformers operating on token sequences are essentially a variant of GNNs [ 9 , 20 ] . We conduct multiple experiments on diverse datasets and compare our approach to state-of-the-art systems in the field. Our findings demonstrate that we constantly achieve better or competitive performance compared to existing methods in the domain. 2 Related Work 2.1 Graph Neural Networks (GNNs) Graphs, or networks, offer a unique way to model any interactions between components in a system [ 1 , Ch.1] . This idea goes beyond well-defined inputs, such as sequences (e.g. text) or grid-structures (e.g. images) and thus requires another type of encoder than well-defined neural networks, such as recurrent or convolutional neural networks [ 8 , Ch.5] . As a solution, graph neural networks that can operate on graph-structured data have been proposed [ 8 , Ch.5] . Many variations of GNNs have been introduced over the last years, this includes Graph Convolutional Networks [ 2 , 11 ] , GraphSAGE [ 7 ] and Graph Attention Networks [ 21 ] . Due to their unique abilities, they have also sparked interest in the NLP/IR community where a variety of problems can also benefit from being modelled as a graph structure [ 24 , 25 , 5 ] . This also includes text classification [ 25 ] . 2.2 GNNs for Short Text Classification Several recent approaches have explored graph-based techniques for short text classification. Linmei et al. [ 13 ] use heterogeneous graph attention networks to aggregate information over a heterogeneous network where nodes represent text samples, entities and topics. Similarly, Wang et al. [ 23 ] use heterogeneous networks comprising word-level graphs (including POS-tag and entity graphs) as well as a document-level graph for short text similarity to capture similarities between short texts in the corpus-level graph. Ding et al. [ 4 ] represent documents as hypergraphs where nodes represent words and different hyperedge types account for co-occurrences of words and topic-related high-order correlations. Another approach based on homogeneous word graphs also modelling co-occurrences of words as edges within a fixed-size sliding window is presented by Zhang et al. [ 30 ] . To inject more corpora-level knowledge into graph-based text classification, Zheng et al. [ 31 ] construct a word co-occurrence graph of common words based on such a corpus. Text samples can then be modelled as the average of these embeddings of words appearing in the corpus graph and then be classified. Based on that approach, Yatadi [ 27 ] introduces a refined GNN architecture that calculates a node-specific radius during aggregation going beyond fixed-hop neighborhoods. Finally, Lin et al. [ 12 ] first fine-tune BERT and then use the model to encode document nodes in a heterogeneous graphs of words and documents. Due to the transductive setup, the GCN used to aggregate the information can also observe test nodes during training. In conclusion, research in graph machine learning has recently pushed graph-based short text classification and resulted in various new approaches. However, these methods are often