UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions

Chuanyuan Tan, Wenbiao Shao, Hao Xiong, Tong Zhu, Zhenhua Liu, Kai Shi, Wenliang Chen · 2025

Read UAQFact: Evaluating Factual Knowledge Utilization of LLMs on Unanswerable Questions by Chuanyuan Tan, Wenbiao Shao, Hao Xiong, Tong Zhu, Zhenhua Liu, Kai Shi, Wenliang Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23461.txt.

Description

Tan and six co-authors build a test for how language models handle questions that cannot be answered, in this two thousand twenty-five paper. A model that guesses anyway can mislead, so knowing when to stop matters, and older test sets gave models no facts to lean on.

Their bilingual dataset, UAQFact, pairs unanswerable questions with supporting facts drawn from a knowledge graph. Two new tasks check whether models use facts they already know and facts handed to them. Across several model families, performance stayed shaky even when the needed knowledge was present, and adding external facts helped only partly. The benchmark exposes a gap between having knowledge and using it under uncertainty.

Catalog record

CREATORChuanyuan Tan, Wenbiao Shao, Hao Xiong, Tong Zhu, Zhenhua Liu, Kai Shi, Wenliang Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23461

Browse the full library · About the Tower of Babel