A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness

Maruf Ahmed Mridul, Rohit Kapa, Oshani Seneviratne · 2026

Read A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness by Maruf Ahmed Mridul, Rohit Kapa, Oshani Seneviratne (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10853.txt.

Introduction

Mridul, Kapa, and Seneviratne offer a runnable test of gap and overlap analysis for graph readiness, built on policy texts like insurance contracts. It asks which texts back a case (overlap) and which do not (gap), with reasons tied to true gaps, not lost data. It lines up plain contract text with an ontology and evidence-linked ground truth: ten checked life-insurance contracts, a filled base, and 58 cases paired with SPARQL queries. A text-only baseline loses to an ontology-driven pipeline on stable scores and diagnosis.

Catalog record

CREATORMaruf Ahmed Mridul, Rohit Kapa, Oshani Seneviratne
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50089
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10853

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness Maruf Ahmed Mridul, Rohit Kapa, Oshani Seneviratne Abstract Task-oriented evaluation of knowledge graph (KG) quality increasingly asks whether an ontology-based representation can answer the competency questions that users actually care about, in a manner that is reproducible, explainable, and traceable to evidence. This paper adopts that perspective and focuses on gap and overlap analysis for policy-like documents (e.g., insurance contracts), where given a scenario, which documents support it (overlap) and which do not (gap), with defensible justifications. The resulting gap/overlap determinations are typically driven by genuine differences in coverage and restrictions rather than missing data, making the task a direct test of KG task readiness rather than a test of missing facts or query expressiveness. We present an executable and auditable benchmark that aligns natural-language contract text with a formal ontology and evidence-linked ground truth, enabling systematic comparison of methods. The benchmark includes: (i) ten simplified yet diverse life-insurance contracts reviewed by a domain expert, (ii) a domain ontology (TBox) with an instantiated knowledge base (ABox) populated from contract facts, and (iii) 58 structured scenarios paired with SPARQL queries with contract-level outcomes and clause-level excerpts that justify each label. Using this resource, we compare a text-only LLM baseline that infers outcomes directly from contract text against an ontology-driven pipeline that answers the same scenarios over the instantiated KG, demonstrating that explicit modeling improves consistency and diagnosis for gap/overlap analyses. Although demonstrated for gap and overlap analysis, the benchmark is intended as a reusable template for evaluating KG quality and supporting downstream work such as ontology learning, KG population, and evidence-grounded question answering. The benchmark and accompanying codebase are available at https://github.com/brains-group/gap_and_overlap_analysis_insurance_contract_benchmark . [ [email protected], orcid=0009-0003-7501-4714 ] [ [email protected] ] [ [email protected], orcid=0000-0001-8518-917X ] keywords life insurance dataset, knowledge graph quality, task-oriented evaluation, ontology engineering, OWL (TBox/ABox), gap and overlap analysis, competency questions, evidence-grounded evaluation, SPARQL query answering, scenario-based testing † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: QKG @ ESWC’26 † † address: Rensselaer Polytechnic Institute, Troy, New York, United States † † address: Prudential Financial, Inc. 1 Introduction Assessing and improving the quality of Knowledge Graphs (KGs) is an active research area, motivated by the growing use of KGs as reusable and interoperable resources for downstream analysis and AI systems 23 ; 20 . In addition to structural and logical criteria, many evaluation efforts now take a task-oriented view of quality: a representation is valuable if it supports the competency questions 5 ; 2 that users care about, with answers that are reproducible, explainable, and traceable to evidence. This paper adopts that perspective and studies a concrete task family centered on gap and overlap analysis . In many domains, important rules and exceptions are written in policy-like documents , such as contracts, regulations, service terms, or product specifications. These documents state what is supported under certain conditions, what is excluded or denied, and what is outside scope. Stakeholders frequently need to compare multiple documents: for a given scenario, which documents support it (overlap), and which do not (gap). For example, a scenario in which an insured dies by suicide 13 months after policy issuance would be covered by contracts with a 12-month exclusion period but not by those with a 24-month exclusion. The overlapping contracts are those that agree on covering it, while the gap is the subset that do not. Here, gaps and overlaps are usually substantive : they occur because documents differ in their intended coverage and restrictions, not because a KG is missing facts or because a query cannot be written. From a KG perspective, gap and overlap analysis is therefore a useful test of task readiness: can an ontology-based representation show these differences consistently, and can it provide evidence that supports each outcome? Reproducible evaluation of such analyses is nevertheless difficult without aligned resources that connect natural language documents to formal representations and queryable ground truth. In practice, it is common for such document collections to be complex, hard to share, or insufficiently annotated for detailed benchmarking, which limits the ability to evaluate both explainability and correctness of the results produced by different methods. It also limits the evaluation of automatic ontology generation and refinement methods, since their outputs cannot be tested against shared, executable competency questions with evidence. To address this barrier, we contribute a benchmark dataset that supports such evaluation purposes, and we demonstrate its use through a workflow that makes gap and overlap analysis executable and auditable. In this paper, we focus on the life insurance domain, where contracts are legally structured yet heterogeneous in their benefits, exclusions, conditions, and temporal constraints, making them a suitable setting for auditable gap and overlap analysis. Our contributions are four-fold. • First, we release a manually curated corpus of ten simplified but diverse life insurance contracts , verified by a domain expert (who is also a co-author on this paper), designed to cover a range of clause patterns while remaining manageable for systematic study. • Second, we provide a domain ontology (TBox) that formalizes the main contract concepts and relations, together with an aligned ABox populated from the contract facts, enabling deterministic, query-based analysis. • Third, we release a suite of 58 structured scenarios and a corresponding ground truth for gap and overlap analysis, which is also reviewed and verified by a domain expert. For each scenario, we include contract-level outcomes and clause-level source excerpts that justify those outcomes, supporting traceability and diagnosis. • Fourth, we evaluate gap and overlap determination using two approaches: a text-only LLM pipeline that infers contract responses for each scenario, and an ontology-driven pipeline that answers the same scenarios using SPARQL over the instantiated ontology. This comparison measures the value of explicit ontological modeling for producing consistent and auditable gap and overlap analyses. Although the benchmark is built in the life insurance domain, the main goal is more general. It shows how aligned document text, an ontology with instances, scenario probes, and evidence-linked labels can form a reusable resource for KG quality evaluation and for downstream tasks such as ontology learning, KG population, and explainable question answering. More broadly, this benchmark aims to address key challenges in modern AI: the limits of LLM reasoning, the need for explainable and auditable outputs, and the integration of structured-logic based and neural methods. By providing a dataset where every answer is traceable to structured evidence, we enable faithful evaluation of reasoning systems beyond surface-level accuracy. Our results demonstrate that while LLMs can approximate coverage judgments, they are less reliable when a scenario requires strict handling of structural applicability, contractual scope, or clause-level justification. By contrast, the ontology-backed pipeline makes these commitments explicit and executable. In this sense, the benchmark is not only a resource for KG quality evaluation, but also a controlled testbed for studying how structured, logic-based representations can complement neural methods in high-stakes domains where correctness and justification matter. 2 Related Work KG quality evaluation has received attention since Zaveri et al. 24 systematized 18 quality dimensions for linked data, and Xue and Zou 23 extended

Read the complete text