p 1 p1 : Better Prompt Optimization with Fewer Prompts
Zhaolin Gao, Yu (Sid) Wang, Bo Liu, Thorsten Joachims, Kianté Brantley, Wen Sun · 2026
Read p 1 p1 : Better Prompt Optimization with Fewer Prompts by Zhaolin Gao, Yu (Sid) Wang, Bo Liu, Thorsten Joachims, Kianté Brantley, Wen Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08801.txt.
Introduction
Gao and colleagues study what makes a task suited to prompt optimization, decomposing reward variance across system prompts into response variance and system prompt variance. Optimization succeeds when system prompt variance is large, but fails when response variance dominates. Adding more user prompts can hurt by shrinking system prompt variance, especially on mixed datasets. Their p1 method filters to a small subset of high variance user prompts, substantially improving optimization. Just two AIME 24 prompts yield a system prompt that generalizes across reasoning benchmarks.
Catalog record
| CREATOR | Zhaolin Gao, Yu (Sid) Wang, Bo Liu, Thorsten Joachims, Kianté Brantley, Wen Sun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91524 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
p 1 p1 : Better Prompt Optimization with Fewer Prompts Zhaolin Gao, Yu (Sid) Wang, Bo Liu, Thorsten Joachims, Kianté Brantley, Wen Sun Abstract Prompt optimization improves language models without updating their weights by searching for a better system prompt, but its effectiveness varies widely across tasks. We study what makes a task amenable to prompt optimization. We show that the reward variance across different system prompts can be decomposed into two components: variance among responses , which captures generation stochasticity, and variance among system prompts , which captures differences in system prompt quality. Prompt optimization succeeds when variance among system prompts is sufficiently large, but fails when variance among responses dominates the variance of the system prompts. Surprisingly, we further show that scaling to more user prompts can hurt optimization by reducing variance among system prompts, especially on heterogeneous datasets where different user prompts favor different system prompts. Motivated by this insight, we propose p 1 p1 , a simple user prompt filtering method that selects a small subset of user prompts with high variance across candidate system prompts. This subset of user prompts allows one to distinguish a good system prompt from a bad one, making system optimization easier. Experiments on reasoning benchmarks show that p 1 p1 substantially improves prompt optimization over training on the full dataset and outperforms strong baselines such as GEPA. Notably, training on only two prompts from AIME 24 yields a system prompt that generalizes well to other reasoning benchmarks. Figure 1: Comparison of p1 against the base model and baseline methods. For all methods, the system prompt is optimized based on AIME 24 and Qwen3-4B-Instruct-2507, and directly applied to these benchmarks and to Qwen3-30B-A3B-Instruct-2507. The results are averaged over 64 generations per user prompt. 1 Introduction System prompts have become a central interface for steering large language models (LLMs). A well-designed system prompt can substantially improve task performance by shaping the model’s reasoning style, formatting behavior, and adherence to instructions, all without modifying model weights ( 36 ; 29 ) . This makes prompt optimization an appealing alternative to full model training. Recent work has therefore explored automatic methods for optimizing prompts, including evolutionary search ( 9 ; 11 ; 1 ; 17 ) , and reinforcement learning (RL) ( 8 ; 16 ; 28 ) . In this paper, we focus on the RL setting where a prompt-generation policy proposes candidate system prompts and the reward is the accuracy of a frozen LLM under each candidate system prompt. Despite the simplicity of prompt optimization, the performance is strikingly inconsistent. On some tasks, optimized system prompts yield clear gains; on others, optimization barely improves even with substantial compute. We investigate the underlying mechanisms that govern this inconsistency. We demonstrate that the reward variance across different system prompts can be decomposed into two distinct components: variance among responses (e.g, math solutions), which captures the inherent generation stochasticity under a fixed system prompt, and variance among system prompts , which captures the true expected reward differences between system prompts. We show that prompt optimization is effective only when the latter component is sufficiently large. For example, on instruction-following benchmarks (e.g., IFBench ( 21 ) ), rewards are highly sensitive to the system prompt, creating a clean optimization signal. Conversely, on complex reasoning benchmarks (e.g., AIME ( 3 ) ), the optimization signal is heavily obscured because the variance among responses dominates the true differences between system prompts. Moreover, our analyses reveal a counterintuitive phenomenon that increasing the size of the dataset reduces the variance among system prompts . Since different user prompts (e.g, math questions) can favor different system prompts, averaging over a larger dataset causes these preferences to cancel out. Candidate system prompts begin to appear statistically identical in terms of their expected reward, diluting the signal for automatic prompt optimization. This effect is especially severe on heterogeneous tasks such as mathematical reasoning, where system prompts that help one example may harm another. By contrast, on more homogeneous tasks such as instruction following, a good system prompt tends to help many examples consistently, making prompt optimization feasible over a large dataset. Motivated by this insight, we propose p1 , a simple and effective user-prompt filtering method that improves prompt optimization while using fewer training examples . Instead of optimizing over the full training set, p1 selects a small subset of user prompts that exhibit high reward variance across candidate system prompts. Training on this filtered subset strengthens the optimization signal by focusing on prompts that best distinguish strong system prompts from weak ones. We evaluate p1 on both instruction-following and reasoning benchmarks. Our results show that standard prompt optimization can fail on heterogeneous reasoning datasets even with substantial compute, whereas filtering yields substantial improvements. Notably, training on just two prompts from AIME 24 produces a system prompt that generalizes well to other reasoning benchmarks . The learned prompts also transfer beyond the training setting. As shown in Fig. 1 , system prompts optimized for Qwen3-4B-Instruct-2507 also improve the larger Qwen3-30B-A3B-Instruct-2507 and boost performance on additional reasoning benchmarks not seen during training. 2 Problem Setup Let x^{\prime} denote a system prompt and x denote a user prompt (e.g., a math question). Given the tuple (x^{\prime},x) , we autoregressively sample a response y from a language model policy \pi , y\sim\pi(\cdot\mid x^{\prime},x) . We define a binary reward function r(x,y)\in\{0,1\} , where r(x,y)=1 indicates a correct response and 0 otherwise. The objective of prompt optimization is to find a system prompt x^{\prime} that maximizes the expected reward across a given dataset \mathcal{D} : \displaystyle\max_{x^{\prime}}\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi(\cdot\mid x^{\prime},x)}\left[r(x,y)\right]. (1) Following prior work, we cast prompt optimization as a reinforcement learning (RL) problem ( 8 ; 16 ; 28 ) . Let s denote a meta-prompt, and let \pi^{\prime} denote a system prompt generation policy. Our objective is to improve \pi^{\prime} such that it produces increasingly effective system prompts: \displaystyle\max_{\pi^{\prime}}\mathbb{E}_{x^{\prime}\sim\pi^{\prime}(\cdot\mid s),\,x\sim\mathcal{D},\,y\sim\pi(\cdot\mid x^{\prime},x)}\left[r(x,y)\right]. (2) Equivalently, we can define r(x^{\prime})=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi(\cdot\mid x^{\prime},x)}[r(x,y)] as the expected reward of a system prompt x^{\prime} where \pi is fixed throughout training, simplifying our objective to maximizing \mathbb{E}_{x^{\prime}\sim\pi^{\prime}(\cdot\mid s)}[r(x^{\prime})] . While any RL algorithm could be used, we adopt a purely on-policy variant of GRPO ( 23 ; 7 ) , without KL regularization to a reference policy \pi_{\mathrm{ref}} ( 30 ) and without standard-deviation-based advantage normalization ( 19 ) . At step t , we maximize the following objective: \displaystyle\mathbb{E}_{x^{\prime}\sim\pi^{\prime}_{t}(\cdot\mid s)}\left[\frac{1}{|x^{\prime}|}\sum_{l=1}^{|x^{\prime}|}\frac{\pi^{\prime}(x^{\prime}_{l}\mid s,x^{\prime}_{<l})}{\pi^{\prime}_{t}(x^{\prime}_{l}\mid s,x^{\prime}_{<l})}\bigl(r(x^{\prime})-V^{\pi^{\prime}_{t}}(s)\bigr)\right], (3) where x^{\prime}_{l} denotes the l -th token of the generated system prompt x^{\prime} , and V^{\pi^{\prime}_{t}}(s):=\mathbb{E}_{x^{\prime}\sim\pi^{\prime}_{t}(\cdot\mid s)}[r(x^{\prime})] serves as the value baseline. We select this formulation because it provides a clean RL objective that is equivalent to RLOO ( 2 ) where the gradient matches with standard policy gradient ( 10 ) . 3 Prompt Learnability In this section, we present a set of preliminary experiments to study the learnability of user prompts, i.e., which classes of user prompts can be optimized effectively with better system prompts and which appear substantially more difficult to improve. To optimize the objective in Eq. 3 , for each step, we sample N system prompts x^{\prime} from \pi^{\prime}_{t}(\cdot\mid s) . For each sampled system prompt and each user prompt x in the dataset