Knowledge Graphs for Enhancing Large Language Models in Entity Disambiguation
Gerard Pons, Besim Bilalli, Anna Queralt · 2025
Read Knowledge Graphs for Enhancing Large Language Models in Entity Disambiguation by Gerard Pons, Besim Bilalli, Anna Queralt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-02737.txt.
Introduction
Gerard Pons, Besim Bilalli, and Anna Queralt present a computing paper from 2025 on naming things in text. A name like Jordan can mean a person, a country, or a firm. Software must pick the right sense from context. Language models can do this with no task training, but they can invent facts or miss field facts.
The team feeds the model facts from a knowledge graph, a web of linked entries. Class layers in the graph prune the list of candidate senses step by step. Short entries describe each candidate to enrich the prompt. On popular test sets, the graph-fed model beat plain and description-only models. It also adapted to new tasks better than models trained for one task. Richer graphs gave stronger gains.
Catalog record
| CREATOR | Gerard Pons, Besim Bilalli, Anna Queralt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-02737
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowledge Graphs for Enhancing Large Language Models in Entity Disambiguation Gerard Pons, Besim Bilalli, Anna Queralt Abstract Recent advances in Large Language Models (LLMs) have positioned them as a prominent solution for Natural Language Processing tasks. Notably, they can approach these problems in a zero or few-shot manner, thereby eliminating the need for training or fine-tuning task-specific models. However, LLMs face some challenges, including hallucination and the presence of outdated knowledge or missing information from specific domains in the training data. These problems cannot be easily solved by retraining the models with new data as it is a time-consuming and expensive process. To mitigate these issues, Knowledge Graphs (KGs) have been proposed as a structured external source of information to enrich LLMs. With this idea, in this work we use KGs to enhance LLMs for zero-shot Entity Disambiguation (ED). For that purpose, we leverage the hierarchical representation of the entities’ classes in a KG to gradually prune the candidate space as well as the entities’ descriptions to enrich the input prompt with additional factual knowledge. Our evaluation on popular ED datasets shows that the proposed method outperforms non-enhanced and description-only enhanced LLMs, and has a higher degree of adaptability than task-specific models. Furthermore, we conduct an error analysis and discuss the impact of the leveraged KG’s semantic expressivity on the ED performance. 1 Introduction The association of textual mentions in a document to the entities they refer to in a Knowledge Graph (KG) is crucial for many Natural Language Processing (NLP) applications, such as question answering or information retrieval. This task is known as Entity Linking (EL), and it is a fundamental step in the transformation of unstructured text into structured knowledge. EL is usually performed as a pipeline with three different steps. The first one is Mention Detection, which detects the text spans that could possibly be linked to entities. It is followed by the Candidate Generation stage, which selects for each mention the top k entities from the KG that could refer to it, usually based on precomputed probability distributions from entity-mention hyperlink pairs. Finally in the Entity Disambiguation (ED) step, a final entity is selected from the previously generated set. Usually, the ED problem is tackled by designing and training task-specific models with large amounts of data (e.g., Wikipedia dumps) [ 10 , 5 ] . In recent years, language models have been used for this task by making use of the mention’s context in the document to disambiguate between the possible solutions [ 21 , 10 , 7 ] . Additionally, some approaches incorporate the candidates’ descriptions [ 25 ] , classes [ 32 ] (e.g., the categories they are tagged with in Wikipedia) or both [ 5 ] in the model’s input, by generating encodings for these text items. The addition of this knowledge allows zero-shot ED, enabling the models to classify entities that may have not been seen during training time. Lately, new advances in Large Language Models (LLMs) such as GPT-3 [ 8 ] , GPT-4 [ 2 ] or LLaMA-2 [ 40 ] , have demonstrated remarkable performance in numerous NLP problems [ 44 ] . Given their large-scale and diverse training corpus, they are good candidates to perform tasks, even zero-shot ones, where general knowledge is needed for language processing, such as ED [ 12 ] . However, these LLMs still face some challenges, such as hallucination (i.e., the generation of statements that are factually incorrect) [ 20 ] , and the lack of knowledge about concepts outside their training corpus. To mitigate these issues, the use of KGs to enhance LLMs has been proposed to address different problems [ 33 ] . There exist a large variety of KGs, storing information which can be encyclopedic (e.g., DBpedia [ 3 ] or YAGO [ 38 ] , which extract information from Wikipedia), commonsense knowledge (e.g., ConceptNet [ 37 ] , with information such as \langle house,has,door\rangle or \langle bed,usedFor,sleep\rangle ) or domain specific [ 1 ] . The explicit and structured knowledge they contain can be used to enhance the performance of LLMs, by leveraging it either during pre-training by enriching the training data [ 19 ] , or during the inference stage [ 42 , 6 , 39 ] . Following the nomenclature proposed in [ 33 ] , in this work we focus on KG-enhanced LLM inference, and apply it to the ED task. Our approach takes advantage of KGs to avoid re-training the LLM, and improves the effectiveness of zero-shot LLM approaches for ED. Figure 1: Overview of the two steps of our approach. Solving the ED problem using only LLMs would require to instruct them to choose one of the entities from the candidate set given the document containing the mention. Instead, we propose to extract the candidates’ class taxonomy from a KG and use it to guide the disambiguation. For example, taking Query 1 in Figure 1 , given ‘MTV awards’ appearing in the context, the entity ‘Justin’ is more likely to be a Musician than a Politician . Thus, we can use this context to guide the LLM by eliminating invalid solutions such as ‘Justin Trudeau’, rather than letting the LLM directly predict the entity. Moreover, when all the remaining candidates fall directly under the same class, we retrieve the candidates’ descriptions from a Knowledge Base (KB), such as Wikipedia, and append them to the disambiguation prompt (see Figure 1 , query 2). With this Retrieval Augmented Generation (RAG) [ 23 ] stage, we provide the LLM with reliable information, reducing hallucination and enabling the LLM to perform predictions over new or unusual entities which may not have been present in the training corpus. Therefore, our contributions are as follows: • We present a method to enhance LLMs in the ED task by leveraging the candidate entity class taxonomies available in KGs. Moreover, we also augment the prompt with the entity descriptions, in order to allow the disambiguation of unseen or difficult entities. • We evaluate the method against non-enhanced LLMs, description-only enhanced LLMs and a task-specific model by using ten ED datasets. The results show that our approach improves the disambiguation capabilities of LLMs and has a higher degree of adaptability to different domains than the task-specific model. • We discuss how using KGs with different levels of semantic expressivity (e.g., YAGO and DBpedia) affects the proposed pruning algorithm, by studying both the ED results and the algorithm’s performance. • We study and classify the cases in which our method fails to correctly disambiguate the mention and provide insights on the possible improvements. The remainder of the paper is structured as follows. In Section 2 we discuss the Related Work. In Section 3 we formalize the problem and present the proposed approach. In Section 4 we describe the different experiments, discuss the results, and conduct an error analysis. Finally, in Section 5 we provide our conclusions and ideas for future work. 2 Related Work This section begins with an overview of different ED methods which leverage external information to improve their predictions. Then, the recently emerged concept of KG-enhanced LLMs is introduced, enumerating some of the proposed methods for tackling NLP tasks. 2.1 Knowledge-augmented