PromptWise: Online Learning for Cost-Aware Prompt Assignment in Generative Models
Xiaoyan Hu, Lauren Pick, Ho-fung Leung, Farzan Farnia · 2025
Read PromptWise: Online Learning for Cost-Aware Prompt Assignment in Generative Models by Xiaoyan Hu, Lauren Pick, Ho-fung Leung, Farzan Farnia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18901.txt.
Introduction
Xiaoyan Hu, Lauren Pick, Ho-fung Leung, and Farzan Farnia introduced PromptWise in 2025, a tool that picks which AI model should answer each prompt. Users now have many models to choose from, and the best model is not always worth its price. Past selection methods looked at quality alone and ignored cost.
PromptWise learns online which model fits each prompt well enough at the lowest cost. Unlike one-shot methods, it can try models in sequence for one prompt to keep total cost down. In tests on code writing and translation, it matched the quality of rival methods while costing much less. The code is shared openly for others to use.
Catalog record
| CREATOR | Xiaoyan Hu, Lauren Pick, Ho-fung Leung, Farzan Farnia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18901
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PromptWise: Online Learning for Cost-Aware Prompt Assignment in Generative Models Xiaoyan Hu, Lauren Pick, Ho-fung Leung, Farzan Farnia Abstract The rapid advancement of generative AI has provided users with a wide range of well-trained models to address diverse prompts. When selecting a model for a given prompt, users should weigh not only its performance but also its service cost. However, existing model-selection methods typically emphasize performance while overlooking cost differences. In this paper, we introduce PromptWise, an online learning framework that assigns prompts to generative models in a cost-aware manner. PromptWise estimates prompt–model compatibility to select the least expensive model expected to deliver satisfactory outputs. Unlike standard contextual bandits that make a one-shot decision per prompt, PromptWise employs a cost-aware bandit structure that allows sequential model assignments per prompt to reduce total service cost. Through numerical experiments on tasks such as code generation and translation, we demonstrate that PromptWise can achieve performance comparable to baseline selection methods while incurring substantially lower costs. The code is available at: github.com/yannxiaoyanhu/PromptWise . 1 Introduction Recent advances in large language models (LLMs) and text-guided generative AI have expanded access to a broad array of services ( Rombach et al., 2022 ; Touvron et al., 2023 ; OpenAI et al., 2024 ; DeepSeek-AI et al., 2025 ; Gemini-Team et al., 2025 ) . Users can now select among many well-trained models to address diverse prompts. This abundance raises a fundamental question: How should one assign prompts to models in a principled and accurate manner? A straightforward strategy is to choose the model with the highest averaged performance score and apply it universally to all prompts. However, the recent literature highlights that performance may vary significantly across prompt categories. A model may excel on one class of prompts yet perform suboptimally on others ( Qin et al., 2024 ; Frick et al., 2025 ; Hu et al., 2025 ) . Such discrepancies arise from differences in training data and architectural choices. To exploit this heterogeneity, prior studies propose either offline selector models ( Qin et al., 2024 ; Frick et al., 2025 ) or online bandit algorithms ( Hu et al., 2025 ) that adaptively allocate prompts to specialized models. Figure 1 : Illustration of cost-aware prompt assignment in Sudoku solving, where a single prompt may be routed to multiple models. PromptWise, via online learning, would route easier puzzles to the inexpensive GPT-4o-mini , while for harder instances it may escalate to stronger and more expensive available models GPT-4o and o1 . While these approaches improve prompt-specific performance, they all assume each prompt is assigned to exactly one model , overlooking the role of monetary and computational costs in users’ decisions on how to assign prompts to available models. In practice, users are not constrained to single-shot decisions. They may query multiple models for the same prompt, escalating to more expensive options only if cheaper attempts fail, as long as the cumulative expense is justified. For instance, consider the Sudoku example in Figure 1 . Suppose a user can choose among OpenAI’s o1 ($60 per million tokens (PMT)), GPT-4o ($10 PMT), and GPT-4o-mini ($0.6 PMT) 1 1 1 https://platform.openai.com/docs/pricing . A seemingly cost-efficient strategy would be to first attempt the least expensive GPT-4o-mini , escalate to GPT-4o if unsuccessful, and resort to o1 only for the hardest puzzles. Although the cheapest model may fail on complex tasks with high probability, the 100 times price difference could justify probing it first. Such multi-assignment strategies are natural in real-world deployments, yet absent from existing online model-selection frameworks. In this work, we introduce PromptWise , a framework for cost-aware online prompt-to-model assignment. At its core, PromptWise is a cost-aware contextual multi-armed bandit (CA-CMAB) formulation that explicitly incorporates both pricing heterogeneity and the possibility of multiple assignments per prompt. Unlike standard contextual bandits ( Langford and Zhang, 2007 ; Li et al., 2010 ; Chu et al., 2011 ) , which commit to a single decision per round, CA-CMAB tracks both the best achieved performance per prompt and the cumulative cost of all attempts (Figure 2 ). Specifically, PromptWise extends the classical Upper Confidence Bound (UCB) framework ( Auer, 2000 ; Bubeck and Cesa-Bianchi, 2012 ) with a new sum-min regret formulation. For each prompt, only the best outcome across attempted models contributes to the performance regret, while costs are fully accumulated. This structure formalizes the trade-off between predictive reward and monetary expenditure, bridging the gap between theoretical online learning formulations and the realities of commercial generative AI services. We evaluate PromptWise through both controlled simulations, where ground-truth optimal strategies are known, and real-world generative AI tasks, including Sudoku and chess puzzles as well as code generation and translation benchmarks (HumanEval ( Chen et al., 2021 ) , HumanEval-X ( Zheng et al., 2023 ) , BigCodeBench ( Zhuo et al., 2024 ) ). Across settings, PromptWise rapidly adapts to prompt distributions, achieves accuracy comparable to cost-unaware baselines, and reduces expenditures by large margins. In summary, this work’s contributions are threefold: • We formalize the cost-aware multi-assignment prompt-selection problem and highlight its practical importance for generative AI services. • We propose PromptWise, a CA-CMAB algorithm that balances performance and cost via a novel sum-min regret metric. • We demonstrate through simulations and real-world benchmarks that PromptWise achieves substantial cost reductions without sacrificing accuracy. Figure 2 : Interaction protocol of standard contextual bandits (top) versus our cost-aware contextual multi-armed bandit (CA-CMAB) framework in PromptWise (bottom). Unlike the standard setting, CA-CMAB enables multiple model assignments per prompt and explicitly balances accuracy (reward) with cumulative service cost, better reflecting practical usage scenarios. 2 Related Work Generalized linear contextual bandits. Contextual bandit (CB) is a widely used framework for sequential decision-making with side information, which has been extensively used in recommendation systems and advertising. A relevant assumption in the CB setup is that the expected reward conditioned to an incoming context admits a generalized linear model (GLM) ( Filippi et al., 2010 ; Li et al., 2017 ) . Particularly, the logistic bandit framework ( Faury et al., 2020 ; Dong et al., 2019 ) has demonstrated stronger performance in applications where rewards are binary. Bandit problems with cost constraints. A line of study considers multi-armed bandits (MAB) in a cost-aware setup, where each arm is associated with a cost distribution. The learner aims to not only optimize the cumulative return but also minimize the cost of pulling the arms. Ding et al. (2013) study cost-aware bandits with a constraint on the total paying cost. In Multi-fidelity MAB ( Kandasamy et al., 2016 ; Wang et al., 2023 ) , the learner may choose to observe (pseudo-)rewards of an arm at some fidelity level and incur the corresponding cost. Recently, Kanarios et al. (2024) study cost-aware best-arm identification, where the goal is to identify the largest reward arm using the minimum expected cost. To the best of our knowledge, cost-aware bandit algorithms in the literature consider optimizing a linear utility, e.g., the cumulative return minus (a regularized) cumulative cost. Online learning methods for training and selecting generative models. A line of research incorporates online learning and reinforcement learning in the training and selecting generative models. Grnarova et al. (2017) formulate the problem of training