Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity’s Last Exam
Michael Majurski, Cynthia Matuszek · 2026
Read Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity’s Last Exam by Michael Majurski, Cynthia Matuszek (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04454.txt.
Introduction
Majurski and Matuszek show query rewriting with answer free context lifts hard exam accuracy. Question phrasing shapes answer quality for models and people alike, yet grounding context and query form interact in underexplored ways. Combining grounded dynamic context construction with rewriting cuts ambiguity and raises accuracy without changing the answer itself, beating merely prepending context. Rewriting Humanity's Last Exam questions with answer free context more than doubles a small model's accuracy. Prompting alone cannot recover the gain: distinct rewriting and answering phases are required.
Catalog record
| CREATOR | Michael Majurski, Cynthia Matuszek |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58483 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04454
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity’s Last Exam Michael Majurski, Cynthia Matuszek Abstract How carefully and unambiguously a question is phrased has a profound impact on the quality of the response, for Language Models (LMs) as well as people. While model capabilities continue to advance, the interplay between grounding context and query formulation remains under-explored. This work investigates how the quality of background grounding information in a model’s context window affects accuracy. We find that combining well-grounded dynamic context construction (i.e, RAG) with query rewriting reduces question ambiguity, resulting in significant accuracy gains. Given a user question with associated answer-free grounding context, rewriting the question to reduce ambiguity produces benchmark improvements without changing the answer itself, even compared to prepending that context before the question. Using gpt-oss-20b to rewrite a subset of Humanity’s Last Exam using answer-free grounding context improves gpt-5-mini accuracy from 0.14 to 0.37. We demonstrate that this accuracy improvement cannot be fully recovered just through prompting at inference time; rather, distinct rewriting and answering phases are required. Code and data are available at https://github.com/mmajurski/lm-rewrite-uplift 1 Introduction The ongoing explosion of Language Model (LM) capability is largely a consequence of increases in training compute, model parameters, and dataset size as described by scaling laws [ 4 ] ; which have demonstrated relationships between performance and parameter count, dataset size, and computation. This capability growth has opened a widening chasm between what modern LMs can demonstrably do and how we measure their spiky intelligence. Leaderboard-style benchmarks have been a primary engine and evaluator of progress in machine learning, providing an objective scalable way to measure and compare capabilities. However, human-curated static benchmarking is a brittle standard. Figure 1: When RAG systems surface relevant information, LM performance can be enhanced by rewriting the initial query using context —added information that, without providing the answer, gives relevant background knowledge and direction. Figure 2: Rewriting questions using answer-free grounding context yields significant accuracy improvement (distance above line) over the original questions, as evaluated on a subset of Humanity’s Last Exam (HLE). See Figure 4 for a complete legend. LM users often implicitly assume the LM shares their mental model, including their background knowledge, context, and intent. This can cause them to omit critical background and related information when formulating a query, believing it to be self-evident. The model, lacking this context, operates instead on training data patterns and the explicit text of the prompt. When faced with an under-specified query, its response will reflect implicit (or explicit) assumptions. If these assumptions do not align with the user’s unstated background expectations, the resulting answer (while potentially factually correct under that assumed interpretation) will be perceived as incorrect or irrelevant.We focus on the specific version of this problem where implicit information can be clarified within RAG search systems. In this work, we explore a the impact on system performance of adding Answer-Free Context ( AFC )—background information that is relevant to the question but which does not contain the actual answer. We demonstrate that even when RAG fails to surface a direct answer, the retrieved background information can be used to perform a contextually grounded disambiguation rewrite ( Figure 1 ) to improve an underspecified query before it is answered. This method can meaningfully improve accuracy on the human validated subset of Humanity’s Last Exam (HLE) ( Figure 2 ), producing better results than simply including the AFC in the question. An example of an original question (from Squadv2 [ 19 ] ), associated AFC , and rewritten question are shown below. Original question: What kind of lasers are crystals of zinc suflde (sic) used in? Answer-free context: Zinc chloride is often added to lumber as a fire retardant and can be used as a wood preservative. It is also used to make other chemicals. Zinc methyl (CH 3 Zn) is used in a number of organic syntheses. Zinc sulfide (ZnS) is used in luminescent pigments such as on the hands of clocks, X-ray and television screens, and luminous paints. Crystals of ZnS are used in lasers. Zinc sulfate is a chemical in dyes and pigments. Zinc pyrithione is used in antifouling paints. Rewritten question: Which portion of the electromagnetic spectrum do lasers that incorporate zinc sulfide (ZnS) crystals generally operate in? (Prompts are given in the Appendix.) This work makes the following contributions: 1. Introduces a method for leveraging Answer-Free Context ( AFC ) to disambiguate queries, yielding significant accuracy gains. 2. Analyzes the performance differential in RAG systems when retrieving direct answers versus purely auxiliary background information. 3. Demonstrates that this accuracy improvement necessitates a distinct rewriting phase, and cannot be replicated by simply prepending the retrieved context to a prompt. 2 Related Works While LMs store significant parametric knowledge, their responses are not grounded in reliable external information sources. Considerable background information lets these models score highly on tests of general knowledge, but they often struggle with domain-specific or proprietary data absent from public training sets. Dynamic context construction approaches like Retrieval-Augmented Generation (RAG) address this by retrieving external evidence during the generation process, which significantly improves factual accuracy [ 23 ] . The RAG pipeline introduces its own optimization challenges, spanning document chunking, search, ranking [ 34 ] , and post-retrieval processing [ 6 ] . A particularly critical component, with a long history in information retrieval, is the transformation, expansion, and normalization of user queries [ 20 , 21 ] . In professional and enterprise workflows, system performance can depend on RAG to dynamically construct context for the LM based upon user queries [ 14 , 34 ] . However, current RAG evaluation focuses on retrieval ranking [ 31 , 22 ] rather than how models utilize retrieved documents—information that might be relevant without explicitly containing the answer. The initial user query is often an imperfect expression of the information needed, suffering from ambiguity, missing context, or poorly aligned terminology with a target document corpus. To bridge this lexical gap, modern RAG systems employ query expansion and rewriting [ 9 , 36 , 15 ] . Query expansion reformulates the question to better surface relevant documents and improve the signal-to-noise ratio of the retrieved context [ 7 ] . LMs have proven adept at this rewriting, expansion, and contextualization task, disambiguating queries using carefully designed prompts [ 29 , 7 , 26 ] . Other approaches explore post-RAG retrieval optimizations for improving signal to noise in the surfaced documents, shifting from a retrieval-generate paradigm to retrieval-note-generate in which the discovered documents are summarized into high level notes [ 6 ] . Other dynamic context and prompting strategies focus on getting the most out of the context. For example, Step-Back Prompting attempts to have LMs derive high-level concepts from first principles to guide reasoning in query answering [ 35 ] . Frameworks like AGREE have the LM cite its sources within the context window (drawn from grounding documents) [ 33 ] . Other approaches involve extensions to chain-of-thought, including Meta-CoT, which models how to determine what underlying reasoning is required to arrive at a particular CoT [ 30 ] . CoT-type reasoning approaches enable the LM to employ self-directed question rewriting during the process of figuring out