Halo : Domain-Aware Query Optimization for Long-Context Question Answering
Pramod Chunduri, Francisco Romero, Ali Payani, Kexin Rong, Joy Arulraj · 2026
Read Halo : Domain-Aware Query Optimization for Long-Context Question Answering by Pramod Chunduri, Francisco Romero, Ali Payani, Kexin Rong, Joy Arulraj (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17668.txt.
Introduction
Chunduri and co-authors present Halo, a long-context question answering system that uses domain knowledge as code. Whole-document calls degrade as text grows, and chunk retrieval still costs many passes. Halo extracts user hints about where to look, what to skip, and how to check answers. Each hint runs as an operator at the pipeline stage where it helps most. A fallback shuts off bad operators at run time. Accuracy rose up to 13 percent.
Catalog record
| CREATOR | Pramod Chunduri, Francisco Romero, Ali Payani, Kexin Rong, Joy Arulraj |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73708 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17668
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Halo : Domain-Aware Query Optimization for Long-Context Question Answering Pramod Chunduri, Francisco Romero, Ali Payani, Kexin Rong, Joy Arulraj Abstract . Long-context question answering (QA) over lengthy documents is critical for applications such as financial analysis, legal review, and scientific research. Current approaches, such as processing entire documents via a single LLM call or retrieving relevant chunks via RAG have two drawbacks: First, as context size increases, response quality can degrade, impacting accuracy. Second, iteratively processing hundreds of input documents can incur prohibitively high costs in API calls. To improve response quality and reduce the number of iterations needed to get the desired response, users tend to add domain knowledge to their prompts. However, existing systems fail to systematically capture and use this knowledge to guide query processing. Domain knowledge is treated as prompt tokens alongside the document: the LLM may or may not follow it, there is no reduction in computational cost, and when outputs are incorrect, users must manually iterate. We present Halo, a long-context QA framework that automatically extracts domain knowledge from user prompts and applies it as executable operators across a multi-stage query execution pipeline. Halo identifies three common forms of domain knowledge – where in the document to look, what content to ignore, and how to verify the answer – and applies each at the pipeline stage where it is most effective: pruning the document before chunk selection, filtering irrelevant chunks before inference, and ranking candidate responses after generation. To handle imprecise or invalid domain knowledge, Halo uses a fallback mechanism that detects low-quality operators at runtime and selectively disables them. Our evaluation across finance, literature, and scientific datasets shows that Halo achieves up to 13% higher accuracy and 4.8 × \times lower cost compared to baselines, and enables a lightweight open-source model to approach frontier LLM accuracy at 78 × \times lower cost. † † authors: . PVLDB Reference Format: Halo : Domain-Aware Query Optimization for Long-Context Question Answering. PVLDB, 14(1): XXX-XXX, 2020. doi:XX.XX/XXX.XX † † This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected] . Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 14, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX 1. Introduction Long-Context Question Answering (QA), an important task in document understanding, aims to answer fact-based questions over lengthy documents. Real-world applications, such as financial analysts querying 10-K filings, legal professionals reviewing contracts, and researchers surveying scientific literature, regularly demand accurate answers from documents spanning hundreds of thousands of tokens ( Reddy et al. 2024 ; Wang et al. [n. d.] ) . Recent advances in large language models (LLMs) have significantly expanded their context windows , with some models such as Gemini 3 Pro and Sonnet 4.5 supporting a million tokens ( Google 2026 ; Anthropic 2025 ) . However, efficiently leveraging these contexts for QA remains challenging. Currently, there are two standard approaches for tackling long-context QA, yet neither achieves a satisfactory accuracy-cost tradeoff. The first feeds the entire document to a state-of-the-art (SoTA) LLM like Sonnet 4.5 in a single call, which we refer to as VanillaLLM . However, this approach is both expensive and unreliable: processing hundreds of long documents can cost over $500 in API calls ( §7.2 ) and even SoTA LLMs exhibit significant hallucination over long contexts due to issues like “lost in the middle” ( Liu et al. 2024 ; Liu et al. 2025a ) . The second approach is Retrieval-Augmented Generation (RAG), which attempts to reduce cost by selecting only the most relevant document chunks before inference ( Lewis et al. 2020 ) . Although RAG reduces token usage compared to VanillaLLM , it introduces top-K selection errors that could hurt accuracy: key evidence may be missed during the similarity search, and fragmented or reordered chunks can introduce incoherent context. As a result, RAG still hallucinates or overlooks important nuances in the provided context ( Leng et al. 2024 ; Jin et al. 2025 ) . To improve the accuracy-cost tradeoff, domain experts naturally turn to domain knowledge. However, the primary mechanism available to them is prompt engineering, where such knowledge is embedded directly into the model’s prompt input, as shown in Figure 1 . For example, a financial analyst may instruct the LLM to focus only on tables or ignore boilerplate disclosures when processing SEC 10-K filings. While such instructions can sometimes improve answer quality, they remain embedded as free-form text within the prompt. The model may inconsistently follow the instructions, especially in long contexts where instruction-following degrades ( He et al. 2024 ; Yen et al. 2025 ) . As seen in Figure 1 , VanillaLLM fabricates incorrect values even when domain knowledge is provided in the prompt as the instructions are lost in the 200K-token context. RAG retrieves relevant context but hallucinates the final response, reporting basic instead of diluted computations, due to the incoherent top-K chunks. In both cases, the domain knowledge yields no cost reduction. Moreover, when such errors occur, users must iteratively refine prompts through trial and error. The core limitation is that domain knowledge is expressed implicitly as tokens rather than captured explicitly as structured guidance that shapes how the system processes the query, limiting improvements in both accuracy and cost. Figure 1. VanillaLLM and RAG cannot utilize user’s domain knowledge effectively for long-context QA. Halo extracts this knowledge and systematically applies it in a multi-stage query optimization pipeline. Figure 2. Workflow of Halo : (1) the KnowledgeParser extracts domain directives from the user prompt, (2) the ContextSelector prunes structural elements using the Structural operator and identifies the top- k relevant chunks, (3) the InferenceEngine applies the Filter operator to discard irrelevant chunks and generates candidate responses on the remaining chunks, (4) the Verifier re-ranks the responses using Validate operator–demoting hallucinatory responses. Our Operator-Based Approach . To address this limitation, we introduce Halo , a long-context QA framework that extracts domain knowledge from free-form prompt text into explicit directives that directly shape execution plans. Our key observation is that domain knowledge should not be consumed as prompt tokens for the LLM to implicitly interpret and instead be extracted into structured directives that are applied at the pipeline stage where each is most effective. Halo identifies three common forms of domain knowledge from users’ natural language prompts: \bullet Structural — where to look : structural elements likely to contain the answer ( e . g ., tables, MD&A sections) \bullet Filter — what to ignore : semantic content irrelevant to the query ( e . g ., legal disclosures, boilerplate text) \bullet Validate — how to verify : domain-specific constraints to demote hallucinated responses ( e . g ., “NOT basic computations”) The KnowledgeParser first extracts structured directives from the user’s unstructured prompt. Halo then leverages operators to apply these directives across a multi-stage pipeline ( Figure 2 ), with each operator designed around a specific insight about where domain knowledge has the highest impact. The ContextSelector stage applies the Structural operator to prune the document to only the user-specified structural elements