Filter-then-Generate: Large Language Models with Structure-Text Adapter for Knowledge Graph Completion

Ben Liu 1 † \dagger, Jihai Zhang 2, Fangquan Lin 2, Cheng Yang 2, Min Peng 1 · 2024

Read Filter-then-Generate: Large Language Models with Structure-Text Adapter for Knowledge Graph Completion by Ben Liu 1 † \dagger, Jihai Zhang 2, Fangquan Lin 2, Cheng Yang 2, Min Peng 1 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09094.txt.

Introduction

Ben Liu, Jihai Zhang, Fangquan Lin, Cheng Yang, and Min Peng present Filter-then-Generate in a 2024 paper on knowledge graph completion with large language models. A knowledge graph stores facts as links between entities, and completion means guessing the links that are missing. Language models hold vast facts, yet they trail classic tools at this task.

Three blocks stand out: the huge pool of candidate entities, the models' urge to hallucinate, and their weak use of graph shape. Their method first filters the pool down, then lets the model generate the pick. An adapter joins graph structure to text for the model.

Catalog record

CREATORBen Liu 1 † \dagger, Jihai Zhang 2, Fangquan Lin 2, Cheng Yang 2, Min Peng 1
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-09094

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Filter-then-Generate: Large Language Models with Structure-Text Adapter for Knowledge Graph Completion Ben Liu 1 † \dagger, Jihai Zhang 2, Fangquan Lin 2, Cheng Yang 2, Min Peng 1 Abstract Large Language Models (LLMs) present massive inherent knowledge and superior semantic comprehension capability, which have revolutionized various tasks in natural language processing. Despite their success, a critical gap remains in enabling LLMs to perform knowledge graph completion (KGC). Empirical evidence suggests that LLMs consistently perform worse than conventional KGC approaches, even through sophisticated prompt design or tailored instruction-tuning. Fundamentally, applying LLMs on KGC introduces several critical challenges, including a vast set of entity candidates, hallucination issue of LLMs, and under-exploitation of the graph structure. To address these challenges, we propose a novel instruction-tuning-based method, namely FtG. Specifically, we present a filter-then-generate paradigm and formulate the KGC task into a multiple-choice question format. In this way, we can harness the capability of LLMs while mitigating the issue casused by hallucinations. Moreover, we devise a flexible ego-graph serialization prompt and employ a structure-text adapter to couple structure and text information in a contextualized manner. Experimental results demonstrate that FtG achieves substantial performance gain compared to existing state-of-the-art methods. The instruction dataset and code are available at https://github.com/LB0828/FtG . 1 Introduction Knowledge graphs (KGs) encode and store abundant factual knowledge in the format of triples like (head entity, relation, tail entity) , which provide faithful knowledge source for downstream knowledge-intensive tasks ( Pan et al. 2023 ; Luo et al. 2023b ) . However, due to the evolving nature of knowledge, real-world KGs often suffer from incompleteness, urging auto-completion of them. Therefore, knowledge graph completion (KGC), which aims to infer missing triples from the existing KG, has been a fundamental and challenging problem in artificial intelligence research. Figure 1: Unsatisfactory performance (Hits@1 metric) of LLMs on the test set of FB15k-237 compared to conventional KGC method RotatE ( Sun et al. 2019 ) . Recently, large language models (LLMs) have demonstrated impressive memorization and reasoning abilities through pre-training on massive text corpora ( Zhang et al. 2024 ; He et al. 2024 ) . Despite their success, LLMs are limited in insufficient knowledge and prone to generate hallucinations ( Huang et al. 2023 ; Li et al. 2023 ) . Multiple attempts that exploit KGs to address the hallucinations issues of LLMs have demonstrated promising achievements on various natural language processing tasks ( Luo et al. 2023b ; Xu et al. 2024 ) . However, the integrating LLMs with KGs for KGC remains underexplored, which present a significant opportunity to revolutionize the approach to KGC by leveraging the contextual understanding and reasoning abilities of LLMs. Zhang et al. 2023 and Shu et al. 2024 apply LLMs to relatively simple KG triple classification (i.e., determining if it is true) has achieved promising results. For more practical and challenging KGC task, Zhu et al. 2023 exploit ChatGPT ( OpenAI 2023 ) with in-context learning ( Wei et al. 2022b ) to transform the KGC task into a text-based prediction. However, contrary to expectations, ChatGPT, equipped with hundreds of billions of parameters, lags behind the performance achieved by conventional structure-based methods (e.g., RotatE ( Sun et al. 2019 ) ). As shown in Figure 1 , on widely-used dataset FB15k-237 Toutanova and Chen 2015 , ChatGPT only achieves 0.097 on the Hits@1 metric while RotatE achieves 0.241 . Furthermore, KG-LLaMA ( Yao et al. 2023 ) utilizes instruction tuning ( Wei et al. 2022a ) to adapt LLaMA2-7b ( Touvron et al. 2023 ) to KGC task, the performance achieved still is not satisfactory ( 0.165 Hits@1 on FB15k-237). Considering the powerful abilities of LLMs, yet they perform worse than conventional methods on KGC task, a pertinent question arises: Why LLMs cannot present satisfactory performance on KGC task? We speculate LLMs struggle in the KGC task from two aspects: (1) Large entity candidate set. The KGC task can essentially be regarded as a classification task, where the label is all entities in KG. Recent works ( Ma et al. 2023 ) evaluating the performance of LLMs on classification tasks have shown that LLMs struggle in datasets with a large number of labels. The result is consistent with Yao et al. 2023 , where LLMs perform relatively better on the KG with a smaller entity set. The enormity of the entity candidate set in the KG poses a challenge for LLMs. Furthermore, LLMs are known for their propensity to hallucinate content, generating information not grounded by world knowledge ( Ji et al. 2023 ) . Directly applying LLMs to elicit missing entities from the entire token space often leads to generating invalid entities outside the entity set. (2) Inherent Graph Structure of KGs. Different from text-based NLP tasks, KGs demonstrate unorganized and complicated graph structures. Current efforts ( Fan et al. 2024 ) have revealed that performance of LLMs on fundamental graph structural understanding tasks is subpar. Effectively guiding LLMs to comprehend the structural information of KGs remains a considerable challenge. Building on these findings, we propose a novel instruction-tuning based method, namely FtG, which harnesses and unleashes the capability of LLMs for KGC task. To address the issues of enormous entity candidate set, we present a filter-then-generate paradigm, where FtG first employs a conventional KGC method as the filter to eliminate unlikely entities and retain only the top- k candidates. Then we formulate the KGC task into a multiple-choice question format and construct instruction template to prompt LLMs to generate target entity from the top- k candidates. Essentially, this paradigm mirrors human behavior. For example, when answering a question, humans would eliminate obviously wrong answers and find the answer from few remaining candidates. In this way, we narrow the candidate set significantly, and the multiple-choice question format effectively avoids LLMs to output uncontrollable text. Moreover, to incorporate graph structure information into LLMs, we devise a structure-aware ego-graph serialization prompt and propose a light-weight structure-text adapter to map the graph features into the text space. Comparison experiment results show that our proposed FtG greatly improves performance of LLMs on KGC task. In summary, our contributions are: • We propose a novel instruction-tuning based method FtG to enhance the performance of LLMs in the KGC task. In which, our proposed filter-then-generate paradigm can effectively harness the powerful capability of LLMs while mitigating the issues caused by hallucinations. • To bridge the gap between the graph structure and LLMs, we introduce a structure-aware ego-graph prompt and devise a structure-text adapter in a contextualized way. • Our FtG outperforms the state-of-the-art methods over three widely used benchmarks, which demonstrates our model’s superiority. Further results reveal that our FtG paradigm can enhance existing KGC methods in a plug-and-play manner. Figure 2: The overall framework of FtG. For a query triple, we employ a KGC filter to obtain top- k candidates and construct corresponding multiple-choice question instruction. Then we sample the ego-graph of query entity and prune irrelevant neighbors. To bridge the gap between graph and text, we encode the pruned ego-graph into a soft graph token and map the graph token into text embedding space with a lightweight adapter. The target entity name is generated with soft graph token, textual serialization of

Read the complete text