Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs

Alberto Cattaneo, Carlo Luschi, Daniel Justus · 2025

Read Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs by Alberto Cattaneo, Carlo Luschi, Daniel Justus (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04473.txt.

Description

Alberto Cattaneo and two colleagues build ground-truth data for knowledge graph question answering. Language models grounded in knowledge graphs promise more factual answers. Yet few datasets reveal the exact facts a system should retrieve, so methods are hard to compare.

Their SynthKGQA framework generates questions from any knowledge graph with full ground-truth fact sets. Applied to a Wikidata-derived graph and checked against a current Wikidata dump, it yields the GTSQA dataset. The set is built to test zero-shot skill on unseen material. With known targets, retrieval methods can finally be scored on what they fetch, not just final answers.

Catalog record

CREATORAlberto Cattaneo, Carlo Luschi, Daniel Justus
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04473

Browse the full library · About the Tower of Babel