An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Syste
Anna Martin-Boyle, William Humphreys, Martha Brown, Cara Leckey, Harmanpreet Kaur · 2026
Read An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Syste by Anna Martin-Boyle, William Humphreys, Martha Brown, Cara Leckey, Harmanpreet Kaur (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21059.txt.
Description
Automated metrics for language-model reliability miss how scientists actually judge answers. Anna Martin-Boyle, William Humphreys, Martha Brown, Cara Leckey, and Harmanpreet Kaur build an expert-grounded alternative in this 2026 arXiv paper: a schema for evaluating errors in scholarly question-answering systems. Working with domain experts, they identified twenty error patterns across seven categories from thematic analysis of sixty-eight question-answer pairs.
Validation through contextual inquiries with ten additional scientists showed which errors experts spot naturally and how structured schemas help them catch issues they had missed. Experts work through orderly routines when they judge: they test fine technical detail, weigh results against their own values, and turn back to review their own judging. The paper points toward tools built on the schema and tuned to each expert's habits and skill.
Catalog record
| CREATOR | Anna Martin-Boyle, William Humphreys, Martha Brown, Cara Leckey, Harmanpreet Kaur |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 126 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-21059