CypherBench : Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era
Yanlin Feng, Simone Papicchio, Sajjadur Rahman · 2024
Read CypherBench : Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era by Yanlin Feng, Simone Papicchio, Sajjadur Rahman (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18702.txt.
Introduction
Yanlin Feng, Simone Papicchio, and Sajjadur Rahman build a benchmark for querying huge knowledge graphs with language models in a paper from 2024. Graphs like Wikidata can feed models private and public facts, but their schemas are too large for a model's context window. Identifiers and tangled relation types add to the problem.
Their fix lays a property graph view over the raw graph, which models can query in the Cypher language. The benchmark holds eleven such graphs across domains, with millions of entities and over ten thousand questions. Building it needed a converter engine and new scoring metrics, both described in the paper.
Catalog record
| CREATOR | Yanlin Feng, Simone Papicchio, Sajjadur Rahman |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CypherBench : Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era Yanlin Feng, Simone Papicchio, Sajjadur Rahman Abstract Retrieval from graph data is crucial for augmenting large language models (LLM) with both open-domain knowledge and private enterprise data, and it is also a key component in the recent GraphRAG system [ 1 ] . Despite decades of research on knowledge graphs and knowledge base question answering, leading LLM frameworks ( e.g. , Langchain and LlamaIndex) have only minimal support for retrieval from modern encyclopedic knowledge graphs like Wikidata. In this paper, we analyze the root cause and suggest that modern RDF knowledge graphs ( e.g. , Wikidata, Freebase) are less efficient for LLMs due to overly large schemas that far exceed the typical LLM context window, use of resource identifiers, overlapping relation types and lack of normalization. As a solution, we propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher . We instantiated this idea on Wikidata and introduced CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. To achieve this, we tackled several key challenges, including developing an RDF-to-property graph conversion engine, creating a systematic pipeline for text-to-Cypher task generation, and designing new evaluation metrics. Dataset https://huggingface.co/datasets/megagonlabs/cypherbench Code https://github.com/megagonlabs/cypherbench 1 Introduction Graphs, as a natural modality for modeling entity-relation data, have been widely used for storing both large-scale encyclopedic knowledge and domain-specific enterprise data. Compared to raw textual documents, graphs enable efficient processing of complex multi-hop aggregation queries ( e.g. , What is the average height of point guards who have played for the Toronto Raptors? ), where the answer might depend on information spread across thousands of documents or even the entire corpus. Graphs also provide a more compact representation of knowledge. For example, Wikidata [ 2 ] contains on average 4.6 times the entities covered by Wikipedia across the domains we experimented with. These advantages has motivated decades of research in knowledge graphs and knowledge base question answering (KBQA) [ 3 , 4 , 5 , 6 , 7 ] , as well as the recent proposal of GraphRAG [ 1 , 8 ] . Figure 1: An illustration of Cypher as a unified interface for retrieval over both RDF and property graphs. A typical graph retrieval or RAG workflow involves: 1) text-to-Cypher translation using an LLM, 2) Cypher query execution, and optionally, 3) final answer generation. However, retrieval 1 1 1 Graph retrieval can be considered as a broader task than KBQA, as it is not only essential for question answering but also for other tasks such as fact checking [ 9 ] . from modern encyclopedic knowledge graphs [ 2 , 10 , 11 , 12 ] , which are predominantly based on RDF, has proven challenging even with the use of LLMs, unlike the success achieved with relational databases using text-to-SQL. Previous studies in KBQA (detailed in section 7 ) typically focused on simplified settings for evaluating algorithmic improvements, using either smaller subgraphs [ 7 , 13 , 14 ] , simple queries without aggregation or grouping [ 3 , 5 , 15 , 13 ] , or assuming the entity identifiers are provided [ 16 , 17 , 18 ] , which limits their practical application in real-world scenarios. As a result, leading LLM frameworks, including LangChain and LlamaIndex, have only minimal support for retrieval from modern RDF knowledge graphs as of the writing of this paper. Instead, they have prioritized retrieval from property graphs , which are typically domain-specific with smaller schemas [ 19 , 20 ] , using text-to-Cypher generation. We analyze the root cause of why retrieval from RDF knowledge graphs is challenging in section 2 and propose transforming them into multiple smaller property graphs. These property graphs function as domain-specific views (analogous to views in relational databases) that can be efficiently queried by LLMs. This approach not only simplifies retrieval from modern RDF knowledge graphs, but also enables the use of Cypher as a unified query language for both RDF graphs and property graph databases like Neo4j that are widely used in enterprise. As a proof of concept, we introduced CypherBench , a collection of 11 property graphs transformed from Wikidata (detailed in section 3 ). Each graph contains the complete set of entities and relations from Wikidata that conform to a domain-specific schema. Together, these graphs include a total of 7 million entities, covering roughly 25% of Wikipedia and 6% of Wikidata. In addition, we constructed over 10,000 natural language questions that spans 12 types of graph matching patterns (detailed in section 4 ). Notably, we include global queries which have been largely overlooked by prior KBQA benchmarks. The benchmark presents significant challenges, with gpt-4o achieving 60.18% execution accuracy, and no LLMs with <10B parameters surpassing 20%. In summary, this paper makes the following main contributions: • A novel methodology to enable efficient and accurate text-to-Cypher retrieval over modern RDF knowledge graphs. • A collection of 11 large-scale property graphs with 7 million entities, serving as groundwork for future graph retrieval research and as a high-quality locally-deployable knowledge source (with a knowledge cutoff of April 2024) for augmenting LLMs. • An RDF-to-property-graph transformation engine for Wikidata that creates the aforementioned graphs. It handles triple transformation, datatype conversion, and unit standardization to produce clean, schema-enforced property graphs as output. • A text-to-Cypher / KBQA benchmark with over 10,000 instances spanning 12 types of graph patterns, covering global queries, multi-hop queries, temporal queries and aggregation queries. • An automatic text-to-Cypher task generation pipeline that creates the aforementioned benchmark. It can be used to generate (question, Cypher) pairs for any Neo4j graph database endpoint. • A set of related tools, including graph deployment Docker, evaluation scripts, and graph visualization tools. 2 Knowledge Graph Modeling in the LLM Era 2.1 Preliminaries: Knowledge Graphs, RDF and Property Graphs The Resource Description Framework (RDF) and property graph are two approaches to modeling and querying knowledge graphs. We begin with an abstract definition of a knowledge graph and then discuss how it is implemented in RDF and property graphs. In its most basic form, a knowledge graph is a list of relations (also called triples) in the format (subject entity, relation type, object entity), such as ("LeBron James", playsFor , "LA Lakers"). 2 2 2 Note that relation types are also called properties in Wikidata or predicates in KBQA literature. In this paper, we use ”properties” specifically to denote entity and relation attributes in property graphs. Additionally, entities are often assigned entity types (e.g., Person ). Entities and relations can also be associated with literal values as properties ( e.g. , receivesAward.year ). The most popular public knowledge graphs to date ( e.g. , Wikidata, Freebase, and DBpedia) are predominantly based on the RDF and queried using SPARQL. In RDF, entities are stored and accessed using Internationalized Resource Identifiers (IRIs). Entity properties, including entity names, are stored as relations, with the subject being the entity IRI and the object being a literal value. To store relation properties, RDF uses a process called reification, which creates a copy of the relation as a