Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06
Sepanta Zeighami, Shreya Shankar, Aditya Parameswaran · 2025
Read Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 by Sepanta Zeighami, Shreya Shankar, Aditya Parameswaran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02896.txt.
Introduction
Sepanta Zeighami, Shreya Shankar, and Aditya Parameswaran introduced a 2025 system called BARGAIN for running language models over large text collections. Top models are accurate but pricey at scale, while cheaper models make more mistakes. Their goal is to cut bills without letting quality drift far from the best model.
BARGAIN decides which records a cheap model can safely handle. It does so with an adaptive sampling plan, meaning it checks a changing slice of outputs to judge quality on the fly. The method carries firm statistical promises about accuracy, and variants promise precision or recall instead. Across eight real datasets it saved far more money than earlier cascade methods while keeping its guarantees.
Catalog record
| CREATOR | Sepanta Zeighami, Shreya Shankar, Aditya Parameswaran |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 134 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02896
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Sepanta Zeighami, Shreya Shankar, Aditya Parameswaran Abstract . Large Language Models (LLMs) are being increasingly used as a building block in data systems to process large text datasets. To do so, LLM model providers offer multiple LLMs with different sizes, spanning various cost-quality trade-offs when processing text at scale. Top-of-the-line LLMs (e.g., GPT-4o, Claude Sonnet) operate with high accuracy but are prohibitively expensive when processing many records. To avoid high costs, more affordable but lower quality LLMs (e.g., GPT-4o-mini, Claude Haiku) can be used to process records, but we need to ensure that the overall accuracy does not deviate substantially from that of the top-of-the-line LLMs. The model cascade framework provides a blueprint to manage this trade-off, by using the confidence of LLMs in their output (e.g., log-probabilities) to decide on which records to use the affordable LLM. However, existing solutions following this framework provide only marginal cost savings and weak theoretical guarantees because of poor estimation of the quality of the affordable LLM’s outputs. We present BARGAIN, a method that judiciously uses affordable LLMs in data processing to significantly reduce cost while providing strong theoretical guarantees on the solution quality. BARGAIN employs a novel adaptive sampling strategy and statistical estimation procedure that uses data and task characteristics and builds on recent statistical tools to make accurate estimations with tight theoretical guarantees. Variants of BARGAIN can support guarantees on accuracy, precision, or recall of the output. Experimental results across 8 real-world datasets show that BARGAIN reduces cost, on average, by up to 86% more than state-of-the-art, while providing stronger theoretical guarantees on accuracy of output, with similar gains when guaranteeing a desired level of precision or recall. 1. Introduction LLMs are being increasingly used as a building block in data systems that process large text datasets for tasks such as extraction, filtering, summarization, and question answering ( Shankar et al. 2024 ; Patel et al. 2024 ; Liu et al. 2024 ; Anderson et al. 2024 ; Urban and Binnig 2024b ; Arora et al. 2023 ; Narayan et al. 2022 ) . For example, given a set of legal contracts, a lawyer might want to find those that relate to a specific law. To do so, a system can iterate over the contracts and, for each contract, ask an LLM to decide if it relates to the specific law. To obtain the most accurate results possible, users want to use top-of-the-line LLMs (e.g., GPT-4o, Claude Sonnet). However, such LLMs are prohibitively expensive at scale. Even a single scan of a few thousand-page long documents by an LLM can cost hundreds of dollars—and gets more expensive with more documents or when multiple scans are needed to address complex or iterative information needs. To support users with budgetary constraints, LLM companies often also provide affordable LLMs (e.g., GPT-4o-mini, Claude Haiku) that are much cheaper but can be less accurate; for instance, GPT-4o-mini is more than 15 \times cheaper than GPT-4o ( openai 2025 ) . However, using the affordable model can reduce answer quality relative to the top-of-the-line model. In such cases, users can often tolerate some marginal quality degradation as long as we can reduce cost substantially, e.g., if the system is guaranteed to match the top-of-the-line LLM’s output 90% of the time but at half the cost. The system then needs to decide when to use which LLM to minimize cost while guaranteeing this desired answer quality. Figure 1. Overview of Model Cascade A common paradigm to decide when to use the affordable model and when the more expensive model is model cascades ( Kang et al. 2017 ; Anderson et al. 2019 ; Kang et al. 2022 ; Russo et al. 2023 ; Kang et al. 2021 ) . In this paradigm, the more expensive LLM is called the oracle while the affordable LLM is called the proxy . In model cascades, we additionally have access to proxy scores which quantify, for each data record, how confident the proxy is in its output 1 1 1 Proxy scores are provided as part of prediction, e.g., log-probability of LLM outputs . Proxy scores help decide whether to use the proxy or the oracle to process a record based on a cascade threshold , \rho (see Fig. 1 , top): the proxy model answer is used for records whose proxy score is more than \rho , and the remaining records are processed with the oracle. The smaller the cascade threshold is, the more frequently the proxy is used to process the records. This means cheaper data processing at potentially worse quality, leading to a classic cost/quality trade off. A central problem in model cascades is setting the cascade threshold based on users’ desired answer quality. In the general case, users provide an accuracy target , e.g., outputs should match the oracle 90% of the time, while minimizing the number of oracle invocations. We refer to this problem as accuracy target (or AT) queries , where the number of oracle invocations avoided is the utility of the approach. Alternatively, in filtering tasks, users may be interested in a desired recall target while maximizing precision, or a precision target while maximizing recall, both settings introduced by Kang et al. 2020 , and respectively referred to as RT and PT Queries. We refer to the achieved precision for RT queries and recall for PT queries as the utility for these two queries. Moreover, we collectively refer to the user-specified accuracy, precision, and recall targets in AT, PT, and RT queries as quality targets . Similar to ( Kang et al. 2020 ) , the user provides a failure probability, \delta , and algorithms must meet quality targets with probability at least 1-\delta while maximizing utility. We focus on using LLMs for classification, specifically multiclass classification for AT queries and binary classification for PT and RT queries. To solve any of AT, PT and RT queries, a common approach ( Kang et al. 2020 ; Kang et al. 2017 ; Lu et al. 2018 ; Patel et al. 2024 ; Kang et al. [n.d.]a ) is to (1) sample and label a subset of records using the oracle, (2) estimate if various cascade thresholds meet the desired quality target and (3) select a cascade threshold among the ones estimated to meet the quality target, as illustrated in Fig. 1 (bottom). However, most existing methods ( Kang et al. 2017 ; Lu et al. 2018 ; Kang et al. [n.d.]a ) do not provide any guarantees on meeting the quality target. Such methods miss the quality target frequently; e.g., ( Kang et al. 2017 ; Lu et al. 2018 ) frequently achieve precision below 65% when given a target of 90% ( Kang et al. 2020 ) . To provide some guarantees, SUPG ( Kang et al. 2020 ) (also used by ( Patel et al. 2024 ) ), employs Central Limit Theorem (CLT) to estimate if different thresholds meet the quality target from labeled samples. Due to the use of CLT, SUPG