Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs
Dipak Meher, Carlotta Domeniconi · 2025
Read Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs by Dipak Meher, Carlotta Domeniconi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26512.txt.
Introduction
Dipak Meher and Carlotta Domeniconi pull apart a system they studied before, to see which parts do the work. The system, CORE-KG, builds knowledge graphs from legal case files about human smuggling. Such files are dense, loose in form, and full of shifting references to the same people, which tends to yield noisy graphs with repeated nodes.
Meher and Domeniconi run an ablation study, removing one part at a time. Without the step that resolves shifting references, node duplication rose by about twenty-eight percent and noisy nodes by about four percent. Without the domain-guided structured prompts, duplication rose by about four percent, while noisy nodes jumped by about seventy-three percent. The results tell builders of such pipelines where the gains come from.
Catalog record
| CREATOR | Dipak Meher, Carlotta Domeniconi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26512
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs Thanks: All example names are fictitious and used solely to protect privacy. Dipak Meher, Carlotta Domeniconi Abstract Human smuggling networks are increasingly adaptive and difficult to analyze. Legal case documents offer critical insights but are often unstructured, lexically dense, and filled with ambiguous or shifting references, which pose significant challenges for automated knowledge graph (KG) construction. While recent LLM-based approaches improve over static templates, they still generate noisy, fragmented graphs with duplicate nodes due to the absence of guided extraction and coreference resolution. The recently proposed CORE-KG framework addresses these limitations by integrating a type-aware coreference module and domain-guided structured prompts, significantly reducing node duplication and legal noise. In this work, we present a systematic ablation study of CORE-KG to quantify the individual contributions of its two key components. Our results show that removing coreference resolution results in a 28.32% increase in node duplication and a 4.32% increase in noisy nodes, while removing structured prompts leads to a 4.34% increase in node duplication and a 73.33% increase in noisy nodes. These findings offer empirical insights for designing robust LLM-based pipelines for extracting structured representations from complex legal texts. I Introduction Human smuggling has become a complex operation involving dynamic networks of actors, routes, vehicles, and intermediaries [ 1 ] . These networks facilitate illicit human mobility for payment, exploiting legal loopholes and adapting rapidly to enforcement efforts to avoid detection. Often linked to transnational crime, they pose serious security and humanitarian risks due to their covert, profit-driven nature. Understanding their structure is critical for shaping effective policy responses and disrupting illicit operations. However, much of the relevant intelligence remains locked within unstructured legal texts such as court rulings and case transcripts, making systematic analysis challenging. Despite growing interest from legal and social science communities, computational approaches for analyzing such documents remain underdeveloped. Entity references in unstructured legal text are often inconsistent, appearing as varying coreferent mentions (e.g., “Defendant Lewis,” “smuggler,” “driver Lewis,” or simply “Lewis”), abbreviations, or role-based titles (e.g., “Officer Lewis” vs. “Defendant Lewis”). These inconsistencies complicate coreference resolution, entity normalization, and downstream tasks such as information extraction and knowledge graph construction. Prior work has shown the value of knowledge graphs in legal investigations. Mazepa et al. [ 2 ] and Shi et al. [ 3 ] , for instance, used rule-based methods to build graphs from legal cases, but their reliance on static templates limits flexibility in handling aliasing and surface-level entity variation. Large language models (LLMs) have shown promise in knowledge extraction [ 4 ] , yet their use in human smuggling case narratives remains limited. A major challenge is merging semantically equivalent mentions of typed entities in complex legal texts [ 5 ] , which, if left unresolved, creates fragmented and redundant graphs. Additionally, LLM-based extractors often misclassify or hallucinate entities and relations [ 6 ] , adding noise that hinders downstream analysis. To address these challenges, we introduced CORE-KG [ 7 ] , a modular framework that builds clean and interpretable knowledge graphs from legal case documents using large language models. CORE-KG combines two main components: (1) a type-aware coreference resolution module that sequentially resolves contextually similar mentions within each entity type, and (2) a structured prompting strategy that guides the model to extract relevant entities and relationships while filtering legal boilerplate and reducing ambiguity. Compared to a GraphRAG-based baseline, CORE-KG reduces node duplication by 33.28% and legal noise by 38.37%, showing strong improvements in graph quality. In this paper, we conduct an ablation study on CORE-KG to quantify the individual contributions of its two key components: type-aware coreference resolution and structured prompting. In the first setting ( CoreKG-no-coref ), we disable the coreference module while keeping structured prompts to assess its role in entity consolidation. In the second ( CoreKG-no-structprompts ), we retain coreference resolution but remove structured prompts to evaluate their impact on reducing noise and improving extraction precision. We adopt the same setup as CORE-KG, using legal case documents related to human smuggling from publicly available U.S. federal and state court proceedings. Since no gold-standard graphs exist, we compare the outputs of each ablation with CORE-KG and a GraphRAG-based baseline [ 8 ] , both of which generate knowledge graphs from raw legal text using large language models. Our ablation study reveals that removing either coreference resolution or structured prompts significantly degrades KG quality. Specifically, removing coreference resolution ( CoreKG-no-coref ) results in a 28.32% increase in node duplication and a 4.32% increase in noisy nodes. In contrast, removing structured prompts ( CoreKG-no-structprompts ) leads to a 73.33% increase in noisy nodes and a 4.34% increase in node duplication. On average, this corresponds to a 38.83% degradation in noise reduction and a 16.33% degradation in entity resolution, underscoring the necessity of both modules for accurate and interpretable knowledge graph construction. 1 1 1 Our code is available at https://github.com/dipakmeher/corekg-ablation-study.git. To summarize, our key contributions are: • We present the first systematic ablation study of the CORE-KG framework, isolating the effects of its two central components: coreference resolution and structured prompting. • We quantitatively show that removing the coreference module results in a 28.32% increase in node duplication and 4.32% increase in noise, while removing structured prompting leads to a 4.34% increase in duplication and a 73.33% spike in noise, demonstrating the distinct and complementary impact of each component. • Our findings empirically validate that both components are essential and complementary for constructing accurate, interpretable knowledge graphs from unstructured legal texts, offering design guidance for future LLM-based KG pipelines. Fig. 1 : Overview of the CORE-KG Pipeline. The pipeline begins with legal text inputs, processed via a type-aware coreference resolution module using sequential per-type prompting. The final resolved text is then passed through a structured prompting stage for entity and relationship extraction. The resulting triples are used to construct a coherent knowledge graph with significantly reduced duplication and noise. II Related Work Knowledge graphs enable structured representation of unstructured text for tasks like reasoning, retrieval, and analysis [ 8 ] . They have seen wide adoption across domains including education [ 9 ] , life sciences [ 10 ] , and construction safety [ 11 ] . However, poorly constructed graphs often exhibit node duplication and structural fragmentation, which hinder their effectiveness [ 12 ] . II-A Knowledge Graph Construction Traditionally, knowledge graphs are constructed by extracting key entities and their relationships from text and linking them sequentially. Entity extraction relies on rule-based, statistical, and domain-specific methods [ 13 ] , while relation extraction uses syntactic, lexical, and semantic features [ 14 ] , ontology-based systems like HowNet [ 15 ] , and semi-supervised techniques [ 16 ] . Though effective in narrow domains, these approaches often rely on hand-crafted rules and focus on maintaining close alignment with the source text. More recently, LLM-based frameworks now enable prompt-driven knowledge graph construction with minimal manual effort [ 6 , 17 ] . Kommineni et al. [ 18 ] combined competency question generation, ontology design, and RAG-based triple extraction, while Zhang et al. [ 6 ] proposed a modular pipeline with open IE, schema definition, and canonicalization. However, these methods often assume clean input