Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models

Saaduddin Mahmud, Mason Nakamura, Kyle H. Wray, Shlomo Zilberstein · 2025

Read Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models by Saaduddin Mahmud, Mason Nakamura, Kyle H. Wray, Shlomo Zilberstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-10030.txt.

Introduction

Saaduddin Mahmud and co-authors argue in this 2025 paper that tuning a prompt without minding how the model will answer is a missed trick. Prompt optimization aligns closed language models, while tricks like best-of-N sampling or majority voting squeeze more from extra compute. The two levers interact, yet older methods tune prompts blind to the answering strategy.

Their framework, IAPO, tunes prompt and inference scale together under a stated budget and user trade-offs. Its training algorithm trims candidates in sequence and carries a finite-budget error guarantee. Across six tasks, from multi-goal writing to reasoning, inference-aware tuning beat blind tuning, making the case that the pair must be chosen as one.

Catalog record

CREATORSaaduddin Mahmud, Mason Nakamura, Kyle H. Wray, Shlomo Zilberstein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-10030

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models Saaduddin Mahmud, Mason Nakamura, Kyle H. Wray, Shlomo Zilberstein Abstract Prompt optimization methods have demonstrated significant effectiveness in aligning black-box large language models (LLMs). In parallel, inference scaling strategies such as Best-of-N Sampling and Majority Voting have also proven to enhance alignment and performance by trading off computation. However, existing prompt optimization approaches are inference strategy agnostic; that is, they optimize prompts without regard to the inference strategy employed during deployment. This constitutes a significant methodological gap, as our empirical and theoretical analysis reveals a strong interdependence between these two paradigms. Moreover, we find that user preferences regarding trade-offs among multiple objectives and inference budgets substantially influence the choice of prompt and inference configuration. To address this gap, we introduce a unified novel framework named Iapo (Inference-Aware Prompt Optimization) that jointly optimizes the prompt and inference scale, while being aware of the inference budget and different task objectives. We then develop a fixed-budget training algorithm for Iapo , which we call Psst (Prompt Scaling via Sequential Trimming), and analyze finite-budget guarantees on error probability. Finally, we evaluate the effectiveness of Psst on six different tasks, including multi-objective text generation and reasoning, and demonstrate the critical role of incorporating inference-awareness when aligning black-box LLMs through prompt optimization. Introduction In recent years, most state-of-the-art large language models (LLMs) are accessible only through black-box APIs. Traditional alignment methods that require access to model weights or logits are therefore infeasible. To address this issue, prompt optimization-based alignment methods have garnered interest ( Chang et al. 2024 ) . These methods typically enhance input prompts by rewording or appending additional instructions to better align the models’ outputs with a task’s objectives. Another broadly applicable alignment strategy for black-box models is scaling inference computations using strategies such as Best-of-N Sampling or Majority Voting . These inference scaling methods generate multiple candidate responses for the same query and select the final response via ranking or voting mechanisms ( Wang et al. 2022 ; Krishna et al. 2022 ; Gui, Gârbacea, and Veitch 2024 ; Yue et al. 2025 ) . Although existing prompt optimization techniques have achieved substantial success, they are typically agnostic to how model outputs are aggregated or sampled, overlooking the impact of such inference methods. Our initial empirical investigation reveals that the performance of optimized prompts is highly sensitive to the choice of inference-scaling approach. Furthermore, our theoretical analysis reveals that decoupling prompt optimization from inference can lead to misalignment. Finally, we observe that optimal alignment requires careful consideration of user-specific preferences regarding the trade-offs among multiple objectives, as well as the computational resources they are willing to expend. These findings expose a critical gap in current methods: the absence of a unified framework that simultaneously accounts for prompt optimization, inference-scaling strategies, user preferences, and computational resource constraints. To bridge this gap, we introduce Iapo (Inference-Aware Prompt Optimization), a novel prompt optimization framework designed explicitly to produce aligned responses from inference-scaled black-box LLMs. Iapo simultaneously optimizes prompt design and inference scaling strategies while considering different task objectives and computational budgets. We formulate the task of identifying an optimal policy for the Iapo framework as a contextual best-arm identification (BAI) problem. To efficiently solve this, we propose a fixed-budget training algorithm named Psst (Prompt Scaling via Sequential Trimming). Additionally, we introduce a warm-up heuristic that further improves performance within the training budget. We begin our analysis by deriving theoretical finite-budget guarantees on the error probability of Psst . Next, we empirically demonstrate the effectiveness of Psst for learning Iapo policies across six diverse tasks, including multi-objective text generation, mathematical reasoning, and commonsense reasoning benchmarks. Additionally, our analysis shows that ignoring inference scaling during prompt optimization can lead to substantial misalignment, highlighting the critical role of inference-awareness in aligning black-box LLMs. Related Work Over the years, considerable effort has been devoted to aligning large language models (LLMs) with human expectations in downstream tasks ( Minaee et al. 2024 ) . Many widely adopted alignment approaches—such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Reinforcement Learning with Verifiable Rewards (RLVR) ( Lambert 2025 ) —require access to model weights. This limitation has motivated increasing interest in black-box alignment methods such as prompt optimization , which can align black-box models through input manipulation alone ( Zhou et al. 2022 ; Ouyang et al. 2022 ; Chang et al. 2024 ) . Prompt optimization has demonstrated strong performance in both single-objective ( Cheng et al. 2023 ; Trivedi et al. 2025 ) and multi-objective ( Jafari et al. 2024 ; Zhao et al. 2025 ) settings. However, these methods are agnostic of the inference strategy used during deployment, potentially leading to suboptimal performance. In contrast, our approach explicitly addresses the interdependence of inference-time strategy and prompt optimization. Recently, Shi et al. 2024 framed prompt optimization as a fixed-budget best-arm identification (BAI) problem. While effective under limited evaluation budgets, the method remains inference agnostic and was only explored in single-objective settings. Our work builds on this foundation in two key ways: (1) we introduce a contextual formulation that models user preferences over multiple objectives and associated computational costs; and (2) we incorporate inference-awareness to ensure alignment with the actual inference strategy. To learn an optimal policy, we introduce a fixed-budget contextual BAI algorithm, Psst , inspired by Sequential Halving (SH) ( Karnin, Koren, and Somekh 2013 ) . While SH was originally developed for the pure bandit setting, the Iapo framework features both inter-context full-information feedback and intra-context semi-bandit feedback. Psst leverages these structural properties to achieve more efficient optimization, extending beyond what standard SH can accommodate. Another relevant line of work focuses on inference-time alignment , where model outputs are improved during inference without modifying model parameters. Some of these methods, such as GenARM and DEAL ( Xu et al. 2024 ; Huang et al. 2024 ) , require access to model logits, limiting their applicability in black-box settings. In contrast, Best-of-N sampling (BoN) and Majority Voting (MV) methods operate purely on model outputs and have shown strong empirical gains by generating multiple candidates and selecting the best one ( OpenAI 2024 ; Yue et al. 2025 ; Wang et al. 2022 ; Krishna et al. 2022 ) . However, these approaches introduce a non-trivial computational cost, and to our knowledge, none of them explicitly optimize the trade-off between computational budget and output quality. Further, our preliminary experiments show that such inference-scaling strategies interact non-trivially with prompt design: prompts optimized for single-shot decoding may yield suboptimal performance under BoN or MV, and vice versa. This necessitates an inference-aware prompt optimization framework. Finally, some white-box methods have recently integrated inference-awareness into the training process. For example, Chow et al. 2025 proposed an inference-aware fine-tuning procedure that explicitly optimizes for exploration–exploitation trade-offs under BoN. Similarly, BOND ( Sessa et al. 2024 ) and BonBon ( Gui, Gârbacea, and Veitch 2024 ) aim to distill BoN policies into a single-pass generation process through supervised fine-tuning. While these approaches avoid the cost of sampling at inference time, they require full access to model parameters and do not generalize beyond BoN-style strategies. In contrast, our method is complementary to inference-aware fine-tuning designed

Read the complete text