LLM-KG-Bench 3.0: A Compass for Semantic Technology Capabilities in the Ocean of LLMs
Lars-Peter Meyer, Johannes Frey, Desiree Heim *, Felix Brei *, Claus Stadler, Kurt Junghanns, Michael Martin · 2025
Read LLM-KG-Bench 3.0: A Compass for Semantic Technology Capabilities in the Ocean of LLMs by Lars-Peter Meyer, Johannes Frey, Desiree Heim *, Felix Brei *, Claus Stadler, Kurt Junghanns, Michael Martin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13098.txt.
Introduction
Lars-Peter Meyer and co-authors present version three of a benchmark that tests how well language models handle knowledge graph work, in a 2025 paper. Such work is the plumbing of the semantic web, and model skill at it varies widely.
The framework runs a set of auto-graded tasks, so answers need no hand checking. This release adds a more flexible task interface, revised tasks, and support for many open models. The team ran over thirty current models, open and commercial, through the bench. Results ship as a data set with model cards comparing skill at graph formats and query languages. The authors call the bench a compass for the field.
Catalog record
| CREATOR | Lars-Peter Meyer, Johannes Frey, Desiree Heim *, Felix Brei *, Claus Stadler, Kurt Junghanns, Michael Martin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13098
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-KG-Bench 3.0: A Compass for Semantic Technology Capabilities in the Ocean of LLMs Lars-Peter Meyer, Johannes Frey, Desiree Heim *, Felix Brei *, Claus Stadler, Kurt Junghanns, Michael Martin Abstract Current Large Language Models (LLMs) can assist developing program code beside many other things, but can they support working with Knowledge Graphs (KGs) as well? Which LLM is offering the best capabilities in the field of Semantic Web and Knowledge Graph Engineering (KGE)? Is this possible to determine without checking many answers manually? The LLM-KG-Bench framework in Version 3.0 is designed to answer these questions. It consists of an extensible set of tasks for automated evaluation of LLM answers and covers different aspects of working with semantic technologies. In this paper the LLM-KG-Bench framework is presented in Version 3 along with a dataset of prompts, answers and evaluations generated with it and several state-of-the-art LLMs. Significant enhancements have been made to the framework since its initial release, including an updated task API that offers greater flexibility in handling evaluation tasks, revised tasks, and extended support for various open models through the vllm library, among other improvements. A comprehensive dataset has been generated using more than 30 contemporary open and proprietary LLMs, enabling the creation of exemplary model cards that demonstrate the models’ capabilities in working with RDF and SPARQL, as well as comparing their performance on Turtle and JSON-LD RDF serialization tasks. Resource type: evaluation benchmark framework License: MPL 2.0 DOI: DOI:10.5281/zenodo.15100803 URL: https://github.com/AKSW/LLM-KG-Bench peer reviewed and to appear in proceedings of ESWC 2025 Resources Track 1 Introduction In the rapidly developing field of Large Language Models (LLMs), it is difficult to keep up with the latest developments and put them into context of prior work. Several LLMs are released every month and some of them are advertised as "better" , "faster" , "cheaper" or with better "reasoning capabilities" . With the work on our benchmarking framework LLM-KG-Bench , we are particularly interested in making it possible to assess and compare LLMs by their capabilities to cope with Semantic Web technology. The framework features support for open source and top-ranking commercial LLMs and includes a set of highly relevant tasks specific to Knowledge Graph Engineering (KGE). For instance, there are specialized tasks related to SPARQL and RDF serialization. In this work, we present Version 3.0 of LLM-KG-Bench. Our latest advancements offer the following contributions: • A major update of the task api which makes writing of new task easier as the overhead is reduced and the framework can handle the evaluation orchestration. • An RDF repair task where the goal is to detect and fix errors across several RDF serialization formats, such as Turtle, JSON-LD, N-Triples. • Improved analytics and visualization: Combined scores can be computed and visualized in a capability compass for task categories such as RDF syntax , RDF analytics , SPARQL syntax , SPARQL semantics or brevity . • Support for encrypted task data to prevent test data leakage into LLM training data. • A new connector for vllm 1 1 1 https://github.com/vllm-project/vllm , a popular high throughput LLM serving framework. The extensions and refinement of the framework were guided by making the KGE related comparison of LLMs easier and broader in aspects of LLM size and task areas covered. The remainder of this work is structured as follows: In Section 2 we present related work. An overview of our LLM-KG-Bench system and its latest improvements is given in Section 3 . In Section 4 we apply the framework to create a big dataset of evaluation results for more than 30 open and proprietary LLMs. Section 5 concludes this work and points out directions for future work. 2 Related Work In order to explore and navigate the vast amount of LLMs, there are several LLM leaderboards, which rank LLMs based on a selection of benchmarks or workloads. For commercial (and a set of open) models, the Chatbot Arena 2 2 2 Leaderboard: https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leaderboard [ 5 ] is popular. While it lists scores for MMLU 3 3 3 Description: https://crfm.stanford.edu/2024/05/01/helm-mmlu.html and MT-bench [ 2 ] , it also calculates its own arena-score which is based on arbitrary tasks, that are processed by two models side-by-side and then evaluated by the same user voting for their preferred answer. For open models the OpenLLM-Leaderboard [ 7 ] provides the most exhaustive list of benchmark results with over 2,000 tested models, and with scores for IfEval, BBH, MATH, GPQU, MUSR, MMLU, and a carbon dioxide emission estimate. HELM 4 4 4 Leaderboard: https://crfm.stanford.edu/helm/lite/latest/#/leaderboard [ 17 ] comprises the most exhaustive list, including also domain-specific tests like LegalBench and MedQA. In contrast to a set of other leaderboards that just collect published or reported test results, those leaderboards provide evaluation as a service. While a comparison of the individual benchmark suites is out of scope of this paper, we see in all of them a gap in addressing Knowledge Graph engineering (KGE) tasks. We also see a gap in a benchmark execution framework that helps to deal with the particularities of RDF and KG-related workloads (format parsing, syntax check feed back loops, execution and evaluation of queries towards KGs, etc.) While Big Bench [ 25 ] was an initial inspiration for the LLM-KG-Bench framework, and we tried to be compatible in the beginning, we realized that the Task API was not sufficient for our KGE benchmarking efforts. Both HELM and BigBench have a strong focus on multiple choice tasks and use scores based on string or document similarities. In contrast, the LLM-KG-Bench framework focuses on the syntactically and semantically correct generation of RDF (i.e. in Turtle) and SPARQL. The LLM-KG-Bench framework aims to reduce complexity and technological burdens to create, execute, evaluate, and analyze KG-related tasks. In the area of benchmarking coding capabilities, we observe, at a conceptual level, characteristics (e.g., with respect to output format requirements, instruction complexity, and response evaluation strategies) that are more closely related to KGE capabilities benchmarking. In this domain, several leaderboards exist. The Big Code Models Leaderboard 5 5 5 https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard evaluates over 60 base models using the HumanEval and MultiPL-E configured for Java, Javascript, and CPP. The EvalPlus Leaderboard 6 6 6 https://evalplus.github.io/leaderboard.html ranks more than 100 models using HumanEval and the Mostly Basic Python Programming (MBPP) Benchmark. These datasets combine human-written programming problems with basic Python challenges to assess coding proficiency. The CanAiCode Leaderboard 7 7 7 https://huggingface.co/spaces/mike-ravkine/can-ai-code-results focuses on programming-related tasks, benchmarking more than 300 models using the custom CanAICode Benchmark. This test suite is specifically designed for testing small text-to-code LLMs with less complex tasks compared to HumanEval and MBPP. However, these efforts are very specialized towards coding and as such also hard to adopt for our use case. In literature, many efforts 8 8 8 Updated overview: https://github.com/zjukg/KG-LLM-Papers explore the combination of LLMs and KGs [ 23 ] . Several of them are evaluating the application of LLMs for KG related tasks. However, these LLM evaluations are often focused on a very specific problem in a specific task area like Text to RDF (e.g. [ 22 , 29 ] ) or Knowledge Graph Question Answering (KGQA, e.g. [ 26 ] ) or Text to SPARQL (e.g. [ 15 , 28 ]