The CitizenQuery Benchmark: A Novel Dataset and Evaluation Pipeline for Measuring LLM Performance in Citizen Query Tasks
Neil Majithia, Rajat Shinde, Zo Chapman, Prajun Trital, Jordan Decker, Manil Maskey, Elena Simperl, Sir Nigel Shadbolt · 2026
Read The CitizenQuery Benchmark: A Novel Dataset and Evaluation Pipeline for Measuring LLM Performance in Citizen Query Tasks by Neil Majithia, Rajat Shinde, Zo Chapman, Prajun Trital, Jordan Decker, Manil Maskey, Elena Simperl, Sir Nigel Shadbolt (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04064.txt.
Description
Citizens asking government questions need answers that are right, not just fluent. Neil Majithia and 7 co-authors built CitizenQuery-UK, a benchmark of twenty-two thousand question-answer pairs drawn from official UK government information. Their 2026 paper covers benefits, taxes, immigration, employment, and health topics.
Pairs were synthetically generated from gov.uk content, with a documented curation method. An evaluation pipeline adapts FActScore to grade factuality across eleven language models. Wrong answers here can quietly cost someone benefits or legal standing, so invisible errors matter more than obvious ones. The benchmark gives developers and agencies a shared yardstick before deployment.
Catalog record
| CREATOR | Neil Majithia, Rajat Shinde, Zo Chapman, Prajun Trital, Jordan Decker, Manil Maskey, Elena Simperl, Sir Nigel Shadbolt |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04064