Cost-of-Pass: An Economic Framework for Evaluating Language Models
Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yüksekgönül, James Zou · 2025
Read Cost-of-Pass: An Economic Framework for Evaluating Language Models by Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yüksekgönül, James Zou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13359.txt.
Introduction
Mehmet Erol and co-authors proposed in 2025 an economic way to judge language models by what a correct answer costs. Wide use of AI depends on value beating the cost of running the model. A score that counts only accuracy misses that trade-off.
Their measure, cost-of-pass, is the expected money cost of getting one right solution. Looking across models, light models won for basic number tasks and large models for fact-heavy ones. Reasoning models won for hard number tasks despite higher per-word cost. Over a year, the best cost fell fast, roughly halving every few months for hard tasks. Tricks that add small gains rarely paid for themselves.
Catalog record
| CREATOR | Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yüksekgönül, James Zou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13359
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cost-of-Pass: An Economic Framework for Evaluating Language Models Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yüksekgönül, James Zou Abstract The widespread adoption of AI systems in the economy hinges on their ability to generate economic value that outweighs their inference costs. Evaluating this tradeoff requires metrics that account for both performance and costs. Building on Farrell’s theory of productive efficiency, we develop an economically grounded framework for evaluating language models’ productivity by combining accuracy and inference cost. We formalize cost-of-pass , the expected monetary cost of generating a correct solution. We then define the frontier cost-of-pass as the minimum cost-of-pass achievable across available models or the human-expert(s), using the approximate cost of hiring an expert. Our analysis reveals distinct economic insights. First , lightweight models are most cost-effective for basic quantitative tasks, large models for knowledge-intensive ones, and reasoning models for complex quantitative problems, despite higher per-token costs. Second , tracking this frontier cost-of-pass over the past year reveals significant progress, particularly for complex quantitative tasks where the cost has roughly halved every few months. Third , to trace key innovations driving this progress, we examine counterfactual frontiers—estimates of cost-efficiency without specific model classes. We find that innovations in lightweight, large, and reasoning models have been essential for pushing the frontier in basic quantitative, knowledge-intensive, and complex quantitative tasks, respectively. Finally , we assess the cost-reductions from common inference-time techniques (majority voting and self-refinement), and a budget-aware technique (TALE-EP). We find that performance-oriented methods with marginal performance gains rarely justify the costs, while TALE-EP shows some promise. Overall, our findings underscore that complementary model-level innovations are the primary drivers of cost-efficiency, and our economic framework provides a principled tool for measuring this progress and guiding deployment. * * * https://github.com/mhamzaerol/Cost-of-Pass . 1 Introduction The recent progress in generative AI, particularly language models (LMs), has sparked significant interest in their potential to transform industries, automate cognitive tasks, and reshape economic productivity ( Brynjolfsson et al., 2025 ; Eloundou et al., 2024 ; Acemoglu, 2024 ) . The widespread adoption of these AI systems in the economy hinges on whether the economic benefits generated by the tasks they can perform outweigh the associated inference costs, and whether those inference costs are lower than the cost of equivalent human labor. Consequently, two priorities have emerged at the forefront of LM research: advancing capabilities and reducing costs. These goals, however, often involve trade-offs with more powerful models or test-time techniques that offer higher accuracy at the expense of greater computational and monetary cost ( Chen et al., 2024 ; Parashar et al., 2025 ; Madaan et al., 2023 ; Wang et al., 2023 ; Kapoor et al., 2024 ) . While standard metrics capture accuracy or other system capabilities, they fail to account for cost, leading to an incomplete picture of progress. Ultimately, what matters to the users is not just raw capability, but the value delivered relative to cost and the standard has been to interpret and report these separately. As the ecosystem of models grows, it is essential to assess new models not in isolation, but in the context of a broader ecosystem, where marginal improvements may or may not justify higher costs, and do so in an easy-to-interpret manner. Figure 1: Highlights of the cost-of-pass framework and empirical analyses. Core concepts (left) set foundations for: (A) Comparing the Human Expert Baseline to the frontier achieved by the single most effective LM per task category. (B) Tracking the reduction in frontier cost-of-pass over time, indicating progress driven by new model releases (color-coded by family). (C) Quantifying the essential contribution of each model family: lightweight (less than $1 per million tokens), large, and reasoning; to the current cost-efficiency frontier, measured by the percentage of each family’s contribution. (D) Assessing the economic benefit (relative cost reduction) achieved by applying common inference-time techniques over the baseline model frontier (which rarely results in meaningful gains). Prior attempts to incorporate cost into evaluation have typically relied on fixed inference budgets ( Wang et al., 2024 ) , heuristic performance scores ( McDonald et al., 2024 ) , or non-monetary proxies such as conciseness ( Nayab et al., 2024 ) . These choices tie conclusions to specific constraints or heuristics, limiting the generality and economic interpretability of such evaluations. To systematically investigate the trade-off between cost and performance and analyze the LM ecosystem as a whole, we draw insights from a well-established and foundational framework from economics: production frontiers. Economists have long studied these frontiers, which map a set of inputs to the maximum output attainable under a given technology ( Farrell, 1957 ) . In Farrell’s original formulation, a producer is technically efficient if no input can be reduced without lowering output, and price efficient if the input mix minimizes cost given input prices. Together, these conditions yield the lowest possible cost per unit of output. Extending this framework, Aigner et al. (1977) introduced stochastic frontier production functions, in which the relationship between inputs and output is modeled as stochastic rather than deterministic, practically accounting for potential defective outputs that do not pass evaluation criteria due to factors beyond the producer’s control. These economic concepts are highly relevant to modern LMs, which inherently function as stochastic producers: for a given input, they yield a desired output (e.g., a correct solution) stochastically ( Brown et al., 2024 ) . Common practices such as employing scaffolds or more computationally intensive inference techniques ( Snell et al., 2024 ; Madaan et al., 2023 ; Wang et al., 2023 ) represent efforts to manipulate this production process. These strategies seek to increase the probability of success but typically do so at the expense of higher computational cost, directly mirroring the economic trade-offs inherent in production efficiency. Motivated by these parallels and the economic goal of minimizing cost per successful output under uncertainty, we adapt the ideas from economically grounded production theory to develop a quantitative evaluation framework tailored to LMs. We summarize our contributions as follows. Concepts. We introduce cost-of-pass (§ 2.2 ), which quantifies the expected monetary cost to achieve a successful output for a given problem. Building on this concept and incorporating a human-expert cost baseline, we define the frontier cost-of-pass (§ 2.4 ) as the minimum achievable cost-of-pass across all available options (LMs and human-expert) for that problem. We show these reveal distinct economic niches for model families (e.g., lightweight vs. reasoning models) on different tasks, which accuracy comparisons alone obscure (§ 3.2 ). Tracking progress with frontier cost-of-pass. Using the cost-of-pass and frontier cost-of-pass, we analyze economic improvements across three task categories from May 2024 to February 2025. We observe an exponential decrease in frontier cost-of-pass across all tasks, though the trends vary. Notably, we observe that, over the past year, the expected cost of generating a correct solution to complex quantitative problems has been cut in half every few months. We find that the frontier cost-of-pass is driven primarily by lightweight models and reasoning models (§ 3.3 ). Counterfactual frontier in the absence of model families. We show that our analysis reveals the complementary roles of different model types in driving recent progress. Innovations in lightweight models have been instrumental in reducing costs on basic