ARK-V1: An LLM-Agent for Knowledge Graph Question Answering Requiring Commonsense Reasoning*

Jan-Felix Klein, Lars Ohnemus · 2025

Read ARK-V1: An LLM-Agent for Knowledge Graph Question Answering Requiring Commonsense Reasoning* by Jan-Felix Klein, Lars Ohnemus (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18063.txt.

Introduction

Jan-Felix Klein and Lars Ohnemus build a simple agent that walks a knowledge graph to answer hard queries. Their 2025 system, ARK-V1, pairs a language model with a graph of linked facts. Model memory alone can be stale or wrong, while graph facts are firm but hard to search across many hops.

The agent steps through the graph bit by bit, guided by common sense as well as graph links. Backbones tested were large models with no extra tuning. On a bench that needs both graph facts and common sense, scores beat step-by-step text prompts by a wide margin. Bigger backbones gave broader and steadier answers.

Catalog record

CREATORJan-Felix Klein, Lars Ohnemus
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-18063

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ARK-V1: An LLM-Agent for Knowledge Graph Question Answering Requiring Commonsense Reasoning* Jan-Felix Klein, Lars Ohnemus Abstract Large Language Models (LLMs) show strong reasoning abilities but rely on internalized knowledge that is often insufficient, outdated, or incorrect when trying to answer a question that requires specific domain knowledge. Knowledge Graphs (KGs) provide structured external knowledge, yet their complexity and multi-hop reasoning requirements make integration challenging. We present ARK-V1, a simple KG-agent that iteratively explores graphs to answer natural language queries. We evaluate several not fine-tuned state-of-the art LLMs as backbones for ARK-V1 on the CoLoTa dataset, which requires both KG-based and commonsense reasoning over long-tail entities. ARK-V1 achieves substantially higher conditional accuracies than Chain-of-Thought baselines, and larger backbone models show a clear trend toward better coverage, correctness, and stability. I INTRODUCTION Large Language Models (LLMs) that are pre-trained on massive text corpora have demonstrated remarkable reasoning capabilities across a wide range of natural language tasks. They are able to answer questions in a zero-shot, closed-book manner by relying on their internalized knowledge. However, this knowledge is often insufficient, outdated, or even incorrect. As a result, LLMs frequently struggle with hallucinations, factual inaccuracies, and a lack of specialized domain knowledge. Knowledge Graphs (KGs) offer a promising solution by serving as structured, external sources of factual knowledge. Yet, effectively integrating KGs into reasoning tasks remains challenging [ 10 ] . KGs can be large and complex, containing numerous irrelevant relations, and many queries require multi-hop reasoning across entities. Furthermore, most LLMs are not fine-tuned to operate directly on KG structures, limiting their effectiveness in leveraging such resources. These challenges are particularly evident in closed-domain applications, such as business settings, where knowledge graphs capture highly specialized and constantly evolving information. In such cases, a general-purpose LLM cannot reliably provide accurate or up-to-date knowledge unless it is explicitly connected to the KG. This highlights the importance of developing AI agents that can effectively explore and utilize KGs as external knowledge sources. In this work, we present an A gent for R easoning on K nowledge Graphs ARK-V1 2 2 2 https://github.com/JaFeKl/ark_v1 , a simple yet effective agent that iteratively explores a knowledge graph and formulates intermediate reasoning steps to answer natural language queries. We evaluate ARK-V1 on the CoLoTa dataset, a newly created benchmark which requires multi-hop reasoning combined with commonsense reasoning. The remainder of this paper is structured as follows. Section II reviews related work. Section III introduces the agent architecture of ARK-V1. Section IV motivates and describes our evaluation experiments, followed by a discussion of results in Section V . Finally, Section VI outlines limitations and directions for future work. II Related Work LLMs have recently been integrated into Knowledge Graph Question Answering (KGQA), with methods broadly categorized into semantic parsing (SP) and Information Retrieval (IR) methods [ 2 ] . SP methods translate natural language (NL) queries into formal logical expressions (e.g., SPARQL), whereas IR methods aim to retrieve and utilize relevant KG information in natural language form. Among the IR-based methods, Li et. al [ 7 ] address multiple-choice QA by first linearizing KG triples into natural language passages using templates, then applying hybrid retrievers to retrieve relevant passages. After reranking with a pre-trained cross-encoder, the top passages are fed into a fine-tuned language model to score answer choices. Guo et al. [ 4 ] propose KnowledgeNavigator , a three-stage framework combining hop-number estimation, iterative triple retrieval, and LLM-based answer generation. Similarly, Baek et al. [ 1 ] study single-shot prompting where the prompt includes pre-retrieved KG triples, evaluating how the quality of the provided triples influence the result. A complementary line of research explores agent-based approaches for KG reasoning. Jiang et al. [ 6 ] introduce a KG agent that employs a toolbox of extraction, logic, and semantic tools, with a fine-tuned LLaMA7-7B model as backbone. Luo et al. [ 8 ] propose RoG, which uses an instruction-fine-tuned LLaMA2-Chat-7B to plan relation paths before retrieval and reasoning. Sun et al. [ 11 ] present Think-on-Graph (ToG), where an LLM agent iteratively generates reasoning paths that are aggregated into answers. While these methods achieve strong results, they are typically evaluated on KGQA datasets where the graph encodes all the information needed to answer and where entities are already familiar to the LLM. By contrast, our work focuses on the CoLoTa dataset, which requires combining KG-based reasoning with commonsense reasoning over long-tail entities. This setting highlights different strengths and weaknesses of LLM-based agents. III AGENT ARCHITECTURE Fig. 1: Agent Architecture of ARK-V1 Given a natural language question Q , the goal of the ARK-V1 agent is to compute an answer A by performing multi-hop reasoning over a knowledge graph (KG). The architecture, illustrated in Figure 1 , achieves this by iteratively constructing reasoning steps and summarizing them to produce the final answer. We represent the KG as a property graph [ 5 ] , defined as \mathcal{G}=\{(h,r,t,\phi)\mid h,t\in\mathcal{E},\;r\in\mathcal{R},\;\phi\in\Phi\}, (1) where \mathcal{E} is the set of entities, \mathcal{R} the set of relations, and \Phi the space of edge properties. Each tuple (h,r,t,\phi) represents a directed relation r from head entity h to tail entity t , optionally augmented with key–value attributes \phi (e.g., temporal information, confidence scores, or provenance). III-A Initialization The agent is initialized ( 1 ) by adding the system prompt and the query Q to the message context. III-B Reasoning Step At its core, the agent produces a sequence of reasoning steps R^{(k)} , where k indexes the k -th step. Each R^{(k)} is initialized in 2 , and its generation follows a fixed procedure outlined below. Select Anchor The first sub-step is to select a single anchor entity from the knowledge graph \mathcal{G} to begin exploration. An explanatory prompt is added to the message context ( 3 ), and the LLM proposes an anchor candidate a^{(k,c)} ( 4 ), where c denotes the c -th attempt to select an anchor within reasoning step k . An anchor candidate is considered valid if it corresponds to a head entity in the graph: a^{(k,c)}\in\mathcal{E}_{\mathrm{head}}=\{h\in\mathcal{E}\mid\exists(h,r,t,\phi)\in\mathcal{G}\}\subseteq\mathcal{E}. (2) The agent is routed according to the attempt counter c and the validity of the selected anchor: \text{route}(a^{(k,c)})=\begin{cases}\hbox to10pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\enskip\lower-5.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke} \lxSVG@begingroup@{fill} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} { {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{fill} {{}{{{}}}{{}}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{}{}{}{}{}{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{fill} \lxSVG@fill\lxSVG@drawpath@unclipped{M 6.92 0 C 6.92 3.82 3.82 6.92 0 6.92 C -3.82 6.92 -6.92 3.82 -6.92 0 C -6.92 -3.82 -3.82 -6.92 0 -6.92 C 3.82 -6.92 6.92 -3.82 6.92 0 Z M 0 0}{stroke:none} \lx@inpgf@ignorespaces \lxSVG@closescope }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-4.25006pt}{-2.57777pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope {{{}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}},&c\geq C_{\max},\\ \hbox to10pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\enskip\lower-5.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke} \lxSVG@begingroup@{fill} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} { {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{fill} {{}{{{}}}{{}}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{}{}{}{}{}{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{fill} \lxSVG@fill\lxSVG@drawpath@unclipped{M 6.92 0 C 6.92 3.82 3.82 6.92 0 6.92 C -3.82 6.92 -6.92 3.82 -6.92 0 C -6.92 -3.82 -3.82 -6.92 0 -6.92 C 3.82 -6.92 6.92 -3.82 6.92 0 Z M 0 0}{stroke:none} \lx@inpgf@ignorespaces \lxSVG@closescope }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-2.12503pt}{-2.57777pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope {{{}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}},&a^{(k,c)}\in\mathcal{E}_{\mathrm{head}},\\ \hbox to10pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\enskip\lower-5.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke} \lxSVG@begingroup@{fill} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} { {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{fill} {{}{{{}}}{{}}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{}{}{}{}{}{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{fill} \lxSVG@fill\lxSVG@drawpath@unclipped{M 6.92 0 C 6.92 3.82 3.82 6.92 0 6.92 C -3.82 6.92 -6.92 3.82 -6.92 0 C -6.92 -3.82 -3.82 -6.92 0 -6.92 C 3.82 -6.92 6.92 -3.82 6.92 0 Z M 0 0}{stroke:none} \lx@inpgf@ignorespaces \lxSVG@closescope }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-2.12503pt}{-2.57777pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope

Read the complete text