Supplement Generation Training for Enhancing Agentic Task Performance
Young-Min Cho, Daniele Bonadiman, Divya Bhargavi, [0.3em] Tamer Alkhouli Salvatore Romeo Dongwei Jiang Khushbu Pahwa, [0.3em] Yubin Ge Etsuko Ishii Monica Sunkara Yi Zhang, [0.8em] University of Pennsylvania, AWS Agentic AI Labs[0.3em] [email protected], {dbonadim, yizhngn}@amazon.com · 2026
Read Supplement Generation Training for Enhancing Agentic Task Performance by Young-Min Cho, Daniele Bonadiman, Divya Bhargavi, [0.3em] Tamer Alkhouli Salvatore Romeo Dongwei Jiang Khushbu Pahwa, [0.3em] Yubin Ge Etsuko Ishii Monica Sunkara Yi Zhang, [0.8em] University of Pennsylvania, AWS Agentic AI Labs[0.3em] [email protected], {dbonadim, yizhngn}@amazon.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20727.txt.
Introduction
Cho and ten colleagues propose Supplement Generation Training for agentic tasks, avoiding post-training massive models for new tasks. Training large models for agentic tasks grows impractical from costs, long cycles, and obsolescence as new models appear. Instead, Supplement Generation Training trains a smaller model to generate added text that, appended to the input, helps the larger model solve tasks better. Those models adapt supplements to task needs, improving performance without modifying the large models. The approach decouples task optimization from large models and enables deployment of agents in applications.
Catalog record
| CREATOR | Young-Min Cho, Daniele Bonadiman, Divya Bhargavi, [0.3em] Tamer Alkhouli Salvatore Romeo Dongwei Jiang Khushbu Pahwa, [0.3em] Yubin Ge Etsuko Ishii Monica Sunkara Yi Zhang, [0.8em] University of Pennsylvania, AWS Agentic AI Labs[0.3em] [email protected], {dbonadim, yizhngn}@amazon.com |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44409 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supplement Generation Training for Enhancing Agentic Task Performance Young-Min Cho, Daniele Bonadiman, Divya Bhargavi, [0.3em] Tamer Alkhouli Salvatore Romeo Dongwei Jiang Khushbu Pahwa, [0.3em] Yubin Ge Etsuko Ishii Monica Sunkara Yi Zhang, [0.8em] University of Pennsylvania, AWS Agentic AI Labs[0.3em] [email protected], {dbonadim, yizhngn}@amazon.com Abstract Training large foundation models for agentic tasks is increasingly impractical due to the high computational costs, long iteration cycles, and rapid obsolescence as new models are continuously released. Instead of post-training massive models for every new task or domain, we propose Supplement Generation Training (SGT), a more efficient and sustainable strategy. SGT trains a smaller LLM to generate useful supplemental text that, when appended to the original input, helps the larger LLM solve the task more effectively. These lightweight models can dynamically adapt supplements to task requirements, improving performance without modifying the underlying large models. This approach decouples task-specific optimization from large foundation models and enables more flexible, cost-effective deployment of LLM-powered agents in real-world applications. 1 Introduction Figure 1: Overview of Supplement Generation Training. Instead of giving the input task directly to the actor model, it is first routed through the supplement generator, which produces supplemental information to aid in task completion. This supplement is concatenated with the original input and passed to the actor model, which solves the task with the additional guidance. The quality of the actor’s solution then serves as a proxy reward for training the supplement generator, indirectly signaling the effectiveness of the generated supplement. The most capable language models today are increasingly deployed as closed-source, API-only systems, where gradient access is unavailable. Even when fine-tuning remains an option, the computational overhead is significant, and newly released foundation models routinely render task-specific fine-tunes obsolete before they can be fully amortized. For these reasons, improving model behavior without modifying weights is often the only viable path, and optimization pressure naturally shifts from the model to the input ( 5 ; 32 ) . Prompt optimization is the natural response to this constraint, but most current methods primarily select from or rearrange existing artifacts rather than generating new, input-specific content. Global methods such as DSPy and TextGrad optimize instruction templates or textual variables across an entire dataset ( 13 ; 30 ) . Recent query-dependent approaches such as Local Prompt Optimization and Prompt-OIRL tailor prompts to individual inputs ( 10 ; 24 ) , yet their adaptation largely operates within a fixed pool of templates. Whether the search is global or per-instance, these methods generally do not learn to synthesize new reasoning structures that may be necessary to solve the task. The dynamic of training and prompt optimization are similar to the relationship between an executive and their assistant. The executive is highly capable but expensive and fully occupied; they cannot be retrained for every new task. The assistant’s role is not merely to relay instructions verbatim but to prepare the right context, provide relevant background, and frame each problem so that the executive can apply their expertise most effectively. A good assistant learns, over time, which framings lead to better decisions, what supplementary information proves useful, and how to anticipate the executive’s reasoning process. In much the same way, we argue that prompt optimization should not be limited to selecting or rearranging existing templates but should learn to synthesize new supporting content tailored to each input, preparing the context a frozen model needs to perform at its best. Inspired by this relationship, we propose Supplement Generation Training (SGT) . Trained with proxy rewards from the task outcomes, SGT teaches a small, open-source model to generate short, instance-specific textual supplements that steer a frozen actor at inference time. Given a query, the supplement generator produces an instance-specific supplement such as a summary, a set of likely mistakes, or a reasoning scaffold, which is appended to the input before the actor is called. With the additional context provided by the supplement generator, the actor model can solve the task more effectively, without any modification to its weights. SGT offers several compelling advantages. First, it is highly cost-efficient: rather than relying on hundred-billion-parameter models, SGT primarily trains a 1.7B model that is lightweight and fast to train, dramatically reducing computational overhead. Second, SGT dynamically selects the supplement type based on the specific task, input instance, and target actor model, enabling targeted assistance rather than one-size-fits-all augmentation. Third, SGT is broadly generalizable, demonstrating consistent improvements across a wide range of benchmarks, actor LLMs, and supplement generator LLMs. In our experiments, we show that SGT outperforms baseline models without supplements by 21%, showing consistent improvements across a wide range of benchmarks and LLMs from different model families. Through ablation studies, we further demonstrate the critical role of iterative preference optimization in gradually achieving these gains. In the qualitative analysis, we further examine the distribution of supplement types across different training stages and reveal a search-and-focus strategy emerging throughout the supplement sampling process. Our work makes the following key contributions: • Supplement Generation Training (SGT). We introduce a new framework that trains a small supplement generator model to produce per-input textual supplements that enhance the performance of large LLMs without modifying their parameters. • Efficient training pipeline with SFT+DPO. We develop a scalable training procedure combining warm-start SFT and iterative DPO, leveraging outcome-based rewards from a black-box actor model. • Comprehensive empirical validation. We evaluate SGT across 5 benchmarks, achieving 21% improvement over baselines. Further analysis reveals strong generalization across model families and tasks. Figure 2: Illustration of our proposed training pipeline: Supplement Generation Training. To teach model how to generate, how to select, and how to improve supplements, we structure the pipeline into a Warm-Start SFT phase followed by an iterative DPO phase. 2 Related Works Automatic prompt optimization (APO) aims at replacing manual prompt tuning with algorithms that search instructions or exemplars. Global/template methods include LLM-driven instruction induction and reinforcement learning over discrete prompt tokens ( 32 ; 6 ) . Black-Box Prompt Optimization (BPO) adapts this to API-only models by iteratively editing prompts from task-level or preference feedback ( 5 ) . Other effective global/template methods that emerged in recent years utilize textual reflection-based feedback to optimize the prompts ( 30 ; 1 ) . DSPy introduces a programming abstraction for LLM pipelines, compiling declarative modules into optimized prompts by automatically searching over instruction and demonstration configurations ( 13 ) . TextGrad treats LLM-generated feedback as textual gradients and applies gradient-descent-like updates to iteratively refine prompt variables ( 30 ) . While both methods effectively optimize global prompt templates, they do not generate new, instance-conditioned content tailored to individual inputs. Beyond global template search, recent work formalizes local and query-dependent automatic prompt optimization, where prompts are tailored per input rather than fixed across a dataset. Local Prompt Optimization (LPO) constrains editing to salient “optimization tokens,” reporting faster convergence and stronger gains than global methods on reasoning benchmarks ( 10 ) . In parallel, Prompt-OIRL frames query-dependent prompt selection via an offline reward model learned from historical prompt–outcome logs, enabling per-input prompt choice without calling the target LLM at inference time ( 24 ) . These methods optimize or select the prompt itself, often by editing an instruction template, scoring candidate prompts with an offline evaluator,