WST: Weak-to-Strong Knowledge Transfer via Reinforcement Learning

Haosen Ge, Shuo Li, Lianghuan Huang · 2025

Read WST: Weak-to-Strong Knowledge Transfer via Reinforcement Learning by Haosen Ge, Shuo Li, Lianghuan Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16741.txt.

Introduction

Haosen Ge, Shuo Li, and Lianghuan Huang present in 2025 a way for a small AI model to coach a much larger one. Writing good prompts for large models is a craft skill, and the big models are often closed or costly to retrain. Their frame, WST, lets a small teacher model draft instructions that lift a large student model's scores.

Reinforcement learning tunes the teacher's drafts by the student's results, round after round. Gains were large on maths tests and on a helpfulness and harmlessness set, beating strong baselines. Oddly, weak teachers avoided the misleading prompts that stronger ones can produce. The authors pitch the method as a cheap, safe route to better prompts.

Catalog record

CREATORHaosen Ge, Shuo Li, Lianghuan Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16741

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WST: Weak-to-Strong Knowledge Transfer via Reinforcement Learning Haosen Ge, Shuo Li, Lianghuan Huang Abstract Effective prompt engineering remains a challenging task for many applications. We introduce Weak-to-Strong Transfer (WST), a automatic prompt engineering framework where a small “Teacher” model generates instructions that enhance the performance of a much larger “Student” model. Unlike prior work, WST requires only a weak teacher, making it efficient and broadly applicable in settings where large models are closed-source or difficult to fine-tune. Using reinforcement learning, the Teacher Model’s instructions are iteratively improved based on the Student Model’s outcomes, yielding substantial gains across reasoning (MATH-500, GSM8K) and alignment (HH-RLHF) benchmarks—98% on MATH-500 and 134% on HH-RLHF—and surpassing baselines such as GPT-4o-mini and Llama-70B. These results demonstrate that small models can reliably scaffold larger ones, unlocking latent capabilities while avoiding misleading prompts that stronger teachers may introduce, establishing WST as a scalable solution for efficient and safe LLM prompt refinement. 1 Introduction Large Language Models (LLMs) have been increasingly applied to complex tasks. Due to the high cost of fine-tuning and the prevalence of closed-source models, prompt engineering remains critical for improving performance and ensuring safety alignment ( Sahoo et al. 2024 ) . Nonetheless, because LLMs are highly sensitive to the wording and phrasing of prompts, it remains an open challenge to optimally structure prompts. We introduce an efficient automatic prompt refinement pipeline, W eak-to- S trong T ransfer ( WST ), which leverages the (surprising) ability of small models to scaffold larger models through automatic prompt generation. Across a series of experiments, we demonstrate that a model as small as 0.5B parameters can significantly enhance the performance of much larger models (e.g., 8B) on complex tasks such as mathematical reasoning and safety alignment. Our approach produces higher-quality instructions than several competitive baselines, including Llama 70B and GPT-4o-mini. We consider a setting where a local model, denoted as M_{1} , provides instructions to a remote model, M_{2} , in response to a given query q . For clarity throughout the paper, we refer to local model M_{1} as the "Teacher Model" and the remote model M_{2} as the "Student Model", reflecting their instructional relationship. The Student Model incorporates the instructions from the teacher model and generates the final response to q . In contrast to prior work on automatic prompt engineering (e.g., Batorski et al. 2025 ; Pryzant et al. 2023 ), we require the Teacher Model to be significantly smaller than the Student Model. This weak-to-strong design offers unique advantages: (i) substantial efficiency gains, since improving a large model requires only modifying the small model’s weights; and (ii) practical applicability, as many real-world scenarios involve proprietary models where training a comparably large Teacher Model is infeasible. Thus, our pipeline can be readily adapted to diverse applications. Importantly, the task of generating useful instructions is non-trivial for the Teacher Model due to its limited capacity relative to the Student Model. If the Teacher Model were stronger than the Student Model, one could simply substitute the Teacher Model’s output for the Student Model’s. However, because the Teacher Model typically performs worse in isolation, its challenge is to provide helpful instructions without introducing misleading information. This is fundamentally different from the original task of responding to q directly. Our experiments show that many models, regardless of size, can produce misleading instructions that degrade another model’s performance. In contrast, our WST method not only mitigates this risk but also consistently improves downstream performance. The WST pipeline employs reinforcement learning to automatically improve the Teacher Model’s instructions based on the Student Model’s performance. Given a query q (e.g., a math problem), the Teacher Model generates instructions m_{1} . The pair (q,m_{1}) is then passed to the Student Model, which outputs a final response m_{2} . This response is evaluated using a predefined metric to obtain a reward r , which is subsequently used to update the Teacher Model’s weights. We evaluate WST on both reasoning and alignment tasks. For reasoning, we use the MATH-500 and GSM8K benchmarks; for alignment, we use the HH-RLHF dataset. Our results demonstrate that WST significantly outperforms natural baselines. Contributions. Our work makes the following contributions: • We introduce Weak-to-Strong Transfer (WST), a novel post-training framework that empowers smaller models to autonomously compose prompts that enhance the performance of much larger models—without requiring access to their weights. Across benchmarks, WST yields consistent gains: a 98% improvement on MATH-500 , 45% on GSM8K , and 134% on HH-RLHF . • We show that after applying WST, small models can effectively scaffold larger models: the larger models achieve higher accuracy and improved alignment when guided by prompts generated from the smaller models, even when the larger models substantially outperform them in isolation. • We find that without WST, stronger Teacher Models can produce misleading instructions that hinder the performance of Student Models, underscoring the necessity of WST in scaffolding better Student Model performance. 2 Method Pipeline. Let \mathcal{M} denote the space of possible generations and \mathcal{Q} the space of queries. Given a query q\in\mathcal{Q} (e.g., a math problem or user request), we first prompt the Teacher Model to generate a set of instructions m_{1}\in\mathcal{M} intended to improve the Student Model’s performance on q . For example, in a safety alignment task, the Teacher Model might consist of constraints specifying what content should not be generated. The pair (q,m_{1}) is then passed to the Student Model, which produces the final output m_{2}\in\mathcal{M} . We define a reward function g:\mathcal{M}\mapsto\mathbbm{R} and obtain a reward r=g(m_{2}) . Finally, we use GRPO ( Guo et al. 2025 ) to update the Teacher Model based on r . Reward. In our experiments, we allow the Student Model to generate multiple outputs for each query in order to stabilize the reward estimate. Specifically, given a query q_{i} and instructions m_{1}^{i} , we sample K generations from the Student Model: m_{2}^{i1},m_{2}^{i2},\ldots,m_{2}^{iK} . The reward is then defined as \displaystyle r_{i}=\frac{1}{K}\sum_{k=1}^{K}g(m_{2}^{ik})-s_{i}, (1) where g(m_{2}^{ik}) is the reward assigned to the k -th generation for query q_{i} , and s_{i} is a baseline score that we describe below. In practice, s_{i} represents the baseline performance of the Student Model. We establish the baseline by averaging the rewards of 10 independent generations from the Student Model in isolation. Thus, r_{i}>0 if and only if the Student Model’s performance improves after incorporating instructions m_{1}^{i} . Averaging across multiple generations reduces variance in the reward signal and provides a more reliable comparison against the baseline. 3 Experiments We evaluate the performance of our approach on two tasks: reasoning and alignment. Reasoning. We use MATH-500 and GSM8K to evaluate our method on reasoning-intensive tasks. The evaluation metric is average accuracy: \displaystyle\text{Acc}(q_{i})=\frac{1}{K}\sum_{k=1}^{K}\mathbbm{1}[m_{2}^{ik}\text{ contains the correct solution}], where q_{i} denotes the i -th problem in the dataset and m_{2}^{ik} the k -th generation from the Student Model in response to q_{i} . We set K=10 in all reasoning experiments. Alignment. For alignment, we use the HH-RLHF dataset. We adopt a multi-objective setup following prior work (e.g., Yang et al. 2024 ; Zhong et al. 2024 ). Each generation is separately evaluated for helpfulness and harmlessness using two LLM-based reward models, yielding a reward vector \tilde{r}=[r^{(1)},r^{(2)}] . We convert this to a scalar reward

Read the complete text