Graph-Convolutional Networks: Named Entity Recognition and Large Language Model Embedding in Document Clustering

Imed Keraghel, Mohamed Nadif · 2024

Read Graph-Convolutional Networks: Named Entity Recognition and Large Language Model Embedding in Document Clustering by Imed Keraghel, Mohamed Nadif (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14867.txt.

Introduction

Imed Keraghel and Mohamed Nadif group documents with a graph method that blends named entity links and large language model embeddings. Rich embeddings from models like BERT help text work, but most grouping methods skip entity ties. Their work came out in twenty twenty-four.

The method builds a graph where documents are nodes and edges weigh how alike their named entities are. A graph convolutional network then tunes the grouping. Tests show the approach beats classic co occurrence methods, most of all for documents rich in named entities. The authors argue entity aware graphs give a truer sense of which documents belong together.

Catalog record

CREATORImed Keraghel, Mohamed Nadif
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14867

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Graph-Convolutional Networks: Named Entity Recognition and Large Language Model Embedding in Document Clustering Imed Keraghel, Mohamed Nadif Abstract Recent advances in machine learning, particularly Large Language Models (LLMs) such as BERT and GPT, provide rich contextual embeddings that improve text representation. However, current document clustering approaches often ignore the deeper relationships between named entities (NEs) and the potential of LLM embeddings. This paper proposes a novel approach that integrates Named Entity Recognition (NER) and LLM embeddings within a graph-based framework for document clustering. The method builds a graph with nodes representing documents and edges weighted by named entity similarity, optimized using a graph-convolutional network (GCN). This ensures a more effective grouping of semantically related documents. Experimental results indicate that our approach outperforms conventional co-occurrence-based methods in clustering, notably for documents rich in named entities. Keywords Large Language models, Named Entity Recognition, Graph Convolutional Networks, Node Embedding, Node Clustering. 1 Introduction Document clustering is widely used in data analytics, especially in fields like information retrieval and natural language processing (NLP). It groups documents into categories based on shared characteristics, which is crucial for tasks like topic modeling and recommendation systems. Traditional methods depend on lexical features [ 29 , 39 ] , co-occurrences, or TF-IDF matrices [ 1 , 27 ] . Although effective in various situations, these methods face challenges in recognizing semantic connections that extend beyond basic word frequency analysis [ 13 ] . Recent advances in machine learning, particularly with the emergence of Large Language Models (LLMs) such as BERT [ 35 , 18 ] and GPT [ 24 ] , have revolutionized text representation. These models offer rich contextual embeddings that capture the nuances of word meanings in context. However, many clustering methods still use traditional approaches like k-Nearest Neighbors (KNN) to construct graphs [ 14 , 23 ] , which depend on shallow lexical similarities and often fail to capture deeper semantic relationships between documents. To illustrate the limitations of traditional methods, Figure 4 compares two graph structures generated from the BBC News dataset: one using a KNN-based graph and the other using our proposed NER-based graph construction method. In the KNN graph, documents are connected based on lexical similarity, leading to overlapping and indistinct clusters. In contrast, the NER-based graph leverages named entity similarities, resulting in a clearer separation of clusters that correspond to distinct semantic topics. Figure 1: Comparison of graph structures for the BBC News dataset. (Left) KNN-based graph with lexical similarity; (Right) NER-based graph capturing entity similarities. In this paper, we propose a novel approach that leverages Named Entity Recognition (NER) alongside LLM embeddings in a graph-based framework for document clustering. Our method builds a graph where nodes represent documents and edges are weighted by the similarity of named entity contexts in each document. Named entities in similar contexts show strong semantic connections, and combining NER with LLM embeddings constructs a graph that better represents document similarities. To optimize this, we use a Graph Convolutional Network (GCN) [ 11 , 15 ] , which enables the joint optimization of the embeddings and the clustering objectives. This procedure guarantees that semantically related documents are grouped more effectively, addressing the shortcomings of conventional clustering techniques that do not incorporate global contextual information. Our main contributions include: • A document clustering method that improves accuracy using NER, LLM embeddings, and graph-based representations. • A technique to create an adjacency matrix based on named entity similarity for precise document relationship identification. • Experiments demonstrating our approach outperforms co-occurrence-based techniques, especially for documents rich in named entities. The remainder of this paper is organized as follows. Section 2 reviews the literature on graph representation learning, graph clustering, GCNs, LLMs, and NER. Section 3 details our method, including adjacency matrix construction and GCNs. Section 4 presents the experimental framework and results, followed by a conclusion and future perspectives in Section 5. 2 Related work Unsupervised graph representation learning has seen remarkable progress in recent years, primarily through two key approaches: contrastive learning and autoencoders. Contrastive learning has emerged as a powerful method due to its ability to differentiate between positive and negative pairs in a self-supervised manner. Techniques such as GraphCL [ 42 ] have introduced graph augmentations that improve graph representations by maximizing the agreement between different augmented views of the same graph. Similarly, MVGRL [ 12 ] enhances the embeddings of node- and graph-level by contrasting views at both levels. On the other hand, autoencoder-based methods provide an alternative by reconstructing specific graph properties, such as the adjacency matrix, to learn embeddings. For example, Graph Autoencoders (GAEs), introduced by [ 37 ] , propose an architecture designed for large-scale graph data, enabling tasks such as node classification and link prediction. Further developments, such as Variational Graph Autoencoders (VGAE) [ 16 ] , extend this approach by incorporating variational autoencoders with GCNs, while Linear Variational Graph Autoencoders (LVGAE) [ 28 ] simplify this architecture using a linear transformation with a one-hop propagation matrix. Building on the foundation of graph representation learning, graph clustering aims to group nodes into clusters based on either graph structure or node-level features, or both. When focusing solely on node attributes, traditional clustering algorithms can be applied to graph data. Spectral clustering [ 22 ] , for example, leverages the graph structure exclusively and optimizes the ratio and normalized cut criteria to form clusters. Deep Graph Infomax (DGI) [ 36 ] , on the other hand, improves graph clustering by maximizing mutual information between graph embeddings and substructures. In addition to these approaches, more recent methods have explored joint optimization of node embeddings and clustering objectives. An example is Graph Convolutional Clustering (GCC) [ 11 ] , which simultaneously learns node embeddings and clusters, resulting in enhanced performance and reduced computational costs. These methods show that incorporating clustering into the learning process can result in more meaningful representations and improved clustering outcomes. In parallel to advancements in graph learning, recent progress in LLMs has significantly improved document representation. Models such as BERT or GPT provide deep contextual embeddings that are particularly well suited for tasks such as clustering, as they capture semantic relationships between entities and documents that traditional methods often miss [ 21 , 13 ] . Sentence-BERT [ 25 ] , for example, fine-tunes BERT to specialize in sentence similarity, making it an ideal candidate for clustering semantically related texts. Incorporating clustering algorithms such as k-means directly into LLM embeddings has also been explored, as seen with BERT-Kmeans [ 31 , 13 ] . Furthermore, the combination of LLMs with graph-based models, such as Graph-BERT [ 43 ] , provides a promising avenue for more accurate and context-aware document clustering by integrating semantic knowledge and graph structures. 3 Model and algorithm The use of NEs in clustering, although not yet widely adopted, presents a promising approach to improve document clustering. By focusing on key entities, such as medical concepts, NE-based clustering can group documents that share relevant entities. For example, in sports news, named entities such as "Kylian Mbappé" and "Cristiano Ronaldo" can link documents about their performances, such as a match in which Mbappé leads PSG to victory or Ronaldo secures Portugal’s World Cup semi-final spot. Furthermore, their involvement in

Read the complete text