Qworld: Question-Specific Evaluation Criteria for LLMs

Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik, [2mm] Department of Biomedical Informatics, Harvard Medical School · 2026

Read Qworld: Question-Specific Evaluation Criteria for LLMs by Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik, [2mm] Department of Biomedical Informatics, Harvard Medical School (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23522.txt.

Introduction

Gao and four co-authors argue that grading open answers needs rules written for each question. Fixed rubrics miss what a given question really demands. Their Qworld method grows a tree of scenarios and viewpoints from the question. The tree bottoms out in fine binary checks an answer must pass. On HealthBench, Qworld covers 89 percent of expert-written checks. It also adds many new checks that experts later validate. Applied to 11 frontier models, it exposes gaps in equity, error handling, and long-term thinking.

Catalog record

CREATORShanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik, [2mm] Department of Biomedical Informatics, Harvard Medical School
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE111389
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23522

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Qworld: Question-Specific Evaluation Criteria for LLMs Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik, [2mm] Department of Biomedical Informatics, Harvard Medical School Abstract Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question’s context. Binary scores and static rubrics fail to capture these context-dependent requirements. Existing methods define criteria at the dataset level or generate them in a single pass, which limits their ability to explore the evaluation space implied by each question. We introduce One-Question-One-World (Qworld), a method that generates question-specific evaluation criteria using a recursive expansion tree. Given a question, Qworld decomposes it into scenarios, perspectives, and fine-grained binary criteria through structured hierarchical and horizontal expansion. The resulting criteria specify what a high-quality answer must address for that question. On HealthBench, Qworld covers 89% of expert-authored criteria and generates 79% novel criteria validated by human experts. Experts rate Qworld criteria higher in insight and granularity than those produced by prior methods. When applied to 11 frontier LLMs on HealthBench and Humanity’s Last Exam, Qworld reveals capability differences in dimensions such as long-term impact, equity, error handling, and interdisciplinary reasoning that coarse rubrics do not distinguish. By formulating criteria generation as structured coverage of question-implied evaluation axes, Qworld enables evaluation that adapts to each question rather than relying on fixed task-level criteria. 1 Introduction Large language models (LLMs) now answer open-ended questions in settings such as scientific reasoning Gao et al. 2025b and clinical decision support ( Xu et al. 2021 ; Singhal et al. 2025 ; Zhao et al. 2026 ; Gao et al. 2025a ) . These uses make evaluation harder ( Li et al. 2025 ) . Closed-ended benchmarks have a single correct answer, but open-ended questions allow multiple valid responses and require judging qualities that depend on the question’s context and intent ( Zheng et al. 2023 ; Bedi et al. 2026 ) . Standard metrics such as accuracy or BLEU ( Papineni et al. 2002 ) do not measure these context-dependent requirements. Current evaluations rely on human judgment, LLM-as-a-Judge, and Agent-as-a-Judge frameworks ( Li et al. 2025 ; Gu et al. 2024 ) guided by predefined task-level criteria. These criteria help, but they assume that questions within a task share the same evaluation needs. They miss requirements that vary with the question and its context. For example, a medical diagnosis question requires attention to safety, uncertainty, and risk communication, while a scientific explanation requires different checks, even when both appear under the same task label ( Bedi et al. 2026 ) . Task-level criteria are also hard to write well: manual curation often omits subtle but important dimensions ( Ye et al. 2023 ) , and expert-written question-level criteria are expensive to produce at scale ( Arora et al. 2025 ) . Figure 1: Given a question, Qworld generates question-specific evaluation criteria, which can be used by downstream evaluators (e.g., LLM-as-judge) to assess responses across diverse contexts. Recent work has automated criteria construction with LLM-generated checklists ( Kim et al. 2024 ; Ye et al. 2023 ) , single-pass prompting ( Cook et al. 2024 ; Wei et al. 2025 ) , contrastive or preference-based induction ( Liu et al. 2025 ; Xie et al. 2026 ; Shen et al. 2026 ) , and retrieval-grounded generation ( Wadhwa et al. 2025 ; Li et al. 2026b ) . These approaches scale better than expert authoring, but most produce criteria from one viewpoint or from a fixed set of dimensions. They do not search for missing evaluation axes implied by a question, and they often miss context-dependent requirements that human experts would consider. Present work. We introduce One-Question-One-World (Qworld), a method that generates evaluation criteria tailored to each question (Figure 1 ). Rather than applying a fixed rubric across questions, Qworld constructs criteria specific to the question at hand. It uses a recursive expansion tree to decompose a question into scenarios, perspectives, and fine-grained binary criteria grounded in the question’s content. We refer to this set of elements as a question’s “world,” which specifies what a high-quality answer must address. Qworld identifies evaluation dimensions that existing criteria overlook while retaining alignment with expert standards. On HealthBench, Qworld achieves Coverage of 0.89 and Uniqueness of 0.79, covering 89% of expert-authored criteria and generating 79% novel criteria. Human evaluators rate Qworld criteria higher in Insight and Granularity than those produced by prior methods. These results show that Qworld generates criteria that extend expert coverage without sacrificing evaluability. The criteria integrate with standard evaluation pipelines, including human review, LLM-as-a-Judge, and agentic frameworks. We apply Qworld to evaluate frontier LLMs on HealthBench ( Arora et al. 2025 ) and Humanity’s Last Exam (HLE) ( Phan et al. 2025 ) , where question-specific requirements shape what counts as a correct answer ( Li et al. 2026a ) . Aggregating Qworld-generated criteria reveals distinctions that generic rubrics collapse, such as separating patient-facing communication from safety-critical risk management ( Ramaswamy et al. 2026 ) , or pedagogical clarity from mathematical rigor ( Bedi et al. 2026 ) . 2 Related Work Scalable evaluation of LLMs and assessment of their capabilities. Traditional benchmarks focused on closed-ended tasks ( Hendrycks et al. 2021 ; Yue et al. 2024 ) , but real-world deployment requires assessing open-ended questions across multiple dimensions of quality ( Singhal et al. 2025 ; Zhang et al. 2025 ) that n-gram metrics like BLEU ( Papineni et al. 2002 ) and ROUGE ( Lin 2004 ) fail to capture ( Li et al. 2025 ) . The community addressed this through LLM-as-a-Judge ( Gu et al. 2024 ) , extended with specialized judge training ( Zhu et al. 2025 ; Wang et al. 2023 ) , factual verification ( Wei et al. 2024 ) , reasoning and planning improvements ( Whitehouse et al. 2025 ; Chen et al. 2025 ; Ko et al. 2025 ; Saha et al. 2025 ) , and agent-as-a-judge settings ( Zhuge et al. 2025 ; Yu 2025 ; You et al. 2026 ) . These approaches optimize how evaluation is executed, but all assume a fixed criterion provided to the judge; Qworld instead generates question-specific criteria that can be used by any of these systems for LLM output scoring. Evaluation Criteria for LLMs. Existing methods for constructing evaluation criteria differ in their level of granularity. Broadly, criteria are defined either at the dataset level or at the question level. Dataset-level criteria: Methods at this level define criteria once for the full dataset, drawing on expert-authored criteria ( Min et al. 2023 ; Qin et al. 2024 ) , human-defined taxonomies in checklist form ( Ye et al. 2023 ; Lee et al. 2025 ) , or task-level decompose-then-prune strategies ( Liu et al. 2024 ) . These capture broad benchmark properties but assume questions within the same dataset share the same evaluation needs. Question-level criteria: More recent work constructs criteria per question. Human-expert benchmarks such as HealthBench ( Arora et al. 2025 ) , ProfBench ( Wang et al. 2025 ) , and PaperBench ( Starace et al. 2025 ) provide high-quality question-level criteria but require expert annotation for every question, limiting scalability. LLM methods lower

Read the complete text