QUEST: Query Optimization in Unstructured Document Analysis Conference: Make sure to enter the correct conference title from your rights confirmation email; June 22–27, 2025; Berlin, Germany DOI: XX.XXXX/XXXXXXX.XXXXXXX ISBN: 979-8-4007-0422-2/24/06
Zhaoze Sun, Qiyan Deng, Chengliang Chai, Kaisen Jin, Xinyu Guo † , Han Han † , Ye Yuan, Guoren Wang, Lei Cao ∗† · 2025
Read QUEST: Query Optimization in Unstructured Document Analysis Conference: Make sure to enter the correct conference title from your rights confirmation email; June 22–27, 2025; Berlin, Germany DOI: XX.XXXX/XXXXXXX.XXXXXXX ISBN: 979-8-4007-0422-2/24/06 by Zhaoze Sun, Qiyan Deng, Chengliang Chai, Kaisen Jin, Xinyu Guo † , Han Han † , Ye Yuan, Guoren Wang, Lei Cao ∗† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06515.txt.
Introduction
Sun and colleagues present QUEST, a tool that cuts the cost of asking language models to mine plain text files, in a paper from twenty twenty-five. New data tools let users query text files like a database. A model reads each file and pulls out facts. That reading is slow and costs a lot, so it slows the whole query.
QUEST first finds the short text parts that hold the needed facts. It feeds just those parts to the model, which cuts waste. It also plans each file on its own, since cost can vary a lot by file. Tests on three real data sets saved thirty percent to six times the cost. Scores for correct facts rose as well.
Catalog record
| CREATOR | Zhaoze Sun, Qiyan Deng, Chengliang Chai, Kaisen Jin, Xinyu Guo † , Han Han † , Ye Yuan, Guoren Wang, Lei Cao ∗† |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06515
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QUEST: Query Optimization in Unstructured Document Analysis Conference: Make sure to enter the correct conference title from your rights confirmation email; June 22–27, 2025; Berlin, Germany DOI: XX.XXXX/XXXXXXX.XXXXXXX ISBN: 979-8-4007-0422-2/24/06 Zhaoze Sun, Qiyan Deng, Chengliang Chai, Kaisen Jin, Xinyu Guo † , Han Han † , Ye Yuan, Guoren Wang, Lei Cao ∗† Abstract . Most recently, researchers have started building large language models (LLMs) powered data systems that allow users to analyze unstructured text documents like working with a database because LLMs are very effective in extracting attributes from documents. In such systems, LLM-based extraction operations constitute the performance bottleneck of query execution due to the high monetary cost and slow LLM inference. Existing systems typically borrow the query optimization principles popular in relational databases to produce query execution plans, which unfortunately are ineffective in minimizing LLM cost. To fill this gap, we propose 𝚀𝚄𝙴𝚂𝚃 {\tt QUEST} , which features a bunch of novel optimization strategies for unstructured document analysis. First, we introduce an index-based strategy to minimize the cost of each extraction operation. With this index, 𝚀𝚄𝙴𝚂𝚃 {\tt QUEST} quickly retrieves the text segments relevant to the target attributes and only feeds them to LLMs. Furthermore, we design an evidence-augmented retrieval strategy to reduce the possibility of missing relevant segments. Moreover, we develop an instance-optimized query execution strategy: because the attribute extraction cost could vary significantly document by document, 𝚀𝚄𝙴𝚂𝚃 {\tt QUEST} produces different plans for different documents. For each document, 𝚀𝚄𝙴𝚂𝚃 {\tt QUEST} produces a plan to minimize the frequency of attribute extraction. The innovations include LLM cost-aware operator ordering strategies and an optimized join execution approach that transforms joins into filters. Extensive experiments on 3 real-world datasets demonstrate the superiority of 𝚀𝚄𝙴𝚂𝚃 {\tt QUEST} , achieving 30%-6 × \times cost savings while improving the F1 score by 10% -27% compared with state-of-the-art baselines. 1. Introduction Modern corporations often maintain a large amount of unstructured data including text documents such as web pages. In fact, according to IDC research ( int 2019 ) , unstructured data accounts for 80%-90% of the data. Recently, researchers have started building LLMs-powered systems, such as UQE ( Dai et al. 2024a ) , ZenDB ( Lin et al. 2024 ) , Lotus ( Patel et al. 2024 ) , and Palimpzest ( Liu et al. [n.d.] ) , to analyze the valuable information hidden in text documents. These systems allow a user to select a set of documents, specify some attributes that can be extracted from them, and apply some database analytical operations on these attributes, e.g., filter, aggregation, or join. The core is to leverage LLMs ( PengLi and ect all 2023 ; Sainz et al. 2023 ; Ashok and Lipton 2023 ) to effectively extract out the attribute values that users are interested in because recent research ( Arora et al. 2023 ) shows that LLMs are remarkably good at data extraction. These efforts, if successful, can turn unstructured data into actionable insights. For instance, a lawyer may employ this system to swiftly locate legal cases about murder with a minimum of three charges and a 15-year sentence. Similar to traditional databases, a successful unstructured data analysis system relies on a query optimizer that automatically produces a plan, minimizing query execution costs. However, LLMs play a central role in such a system , raising unique challenges and optimization opportunities. More specifically, compared to traditional database operations, LLM inferences are much more expensive in both execution time and monetary cost ( Liu et al. [n.d.] ) , no matter whether using commercial LLM services or deploying open source LLMs on high-performance thus expensive GPUs. Because unstructured data analysis relies on LLMs to extract attributes, the extraction operation thus constitutes its performance bottleneck. Therefore, the key optimization objective in this scenario is to minimize the LLM cost incurred during extraction, equivalent to (1) minimizing the LLM cost of each extraction operation, which depends on the number of input tokens to an LLM, and (2) minimizing the frequency of invoking the data extraction operations. To achieve the above optimization objective, we propose {\tt QUEST} with two key components: (1) an index-based attribute extraction strategy to minimize the number of input tokens per extraction; (2) an instance-optimized query execution strategy to minimize the frequency of attribute extraction. Crucially, these optimization strategies are model-agnostic. Although the cost and performance may vary across different LLMs, the core principles of {\tt QUEST} remain effective in reducing the number of input tokens and extraction frequency. {\tt QUEST} thus offers an affordable and scalable approach that allows users to select thousands of documents from a large document collection, specify any attributes out of tens or even hundreds of attributes that these documents potentially contain, and analyze these attributes. (1) Index-based Attribute Extraction. {\tt QUEST} designs an index for LLMs to efficiently and effectively extract attributes from documents. Rather than using LLMs to scan each document routinely , it only feeds LLMs the relevant text segments, thus significantly reducing the number of input tokens. This retrieval augmented generation (RAG) inspired solution features two innovative designs: a two-level index and evidence augmented retrieval. Two-level Index. First, {\tt QUEST} summarizes the subject of each document and indexes this information to filter documents irrelevant to the target attributes. Then {\tt QUEST} builds a segment level index to retrieve relevant segments from the remaining documents. Compared to existing document analysis systems such as ZenDB ( Lin et al. 2024 ) , which filter data only at the segment level, our two-level strategy more accurately identifies the relevant segments because it avoids erroneously recognizing some segments as relevant when they are from documents in different subjects. Evidence Augmented Retrieval. Second, when searching the index, existing RAG style solutions tend to miss relevant segments, because the query, which typically contains the attributes and their text description, is often not informative enough. To solve this problem, one can rely on users to manually enhance text descriptions like prompt engineering, but it is tedious and time-consuming. We propose a sampling-based approach to automatically collect the evidence on what the relevant segments should look like and use this evidence to augment the retrieval. (2) Instance-optimized Query Execution. Rather than first extracting out the attribute values involved in a query and then employing the database query optimizer to produce an execution plan, {\tt QUEST} adopts a lazy extraction strategy. That is, it interleaves attribute extraction and analytical operations (e.g., filter or join), only extracting an attribute when an analytical operation has to evaluate it. In this way, {\tt QUEST} opens optimization opportunities to minimize the frequency of data extraction. For example, if a legal case in a document is not about murder, {\tt QUEST} does not have to extract other attributes, e.g., the number of charges or the years of the sentence. Intuitively, similar to relational databases, appropriately ordering the filters or the joins in a query could effectively avoid extracting the values that have no chance to appear in the final query results. Guided by the optimization objective of {\tt QUEST} , we revisit the query optimization principles in databases, e.g., filter ordering, predicate pushdown, and join ordering. As detailed in Sec. 3 , rather than simply ordering