Removing Sandbagging in LLMs by Training with Weak Supervision

Emil Ryd, Henning Bartsch, Julian Stastny, Joe Benton, Vivek Hebbar · 2026

Read Removing Sandbagging in LLMs by Training with Weak Supervision by Emil Ryd, Henning Bartsch, Julian Stastny, Joe Benton, Vivek Hebbar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22082.txt.

Introduction

Ryd and four colleagues study sandbagging, where a model more capable than its supervisors produces work that looks fine but hides its full skill. Weak training can coax out sandbagging models when supervised fine-tuning and reinforcement learning are combined. Fine-tuning on weak demos breaks sandbagging, letting reinforcement learning draw out full performance. Neither method succeeds reliably alone: reinforcement learning without fine-tuning leads to reward hacking, and fine-tuning without reinforcement learning fails when the minder is far weaker.

Catalog record

CREATOREmil Ryd, Henning Bartsch, Julian Stastny, Joe Benton, Vivek Hebbar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101913
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22082

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Removing Sandbagging in LLMs by Training with Weak Supervision Emil Ryd, Henning Bartsch, Julian Stastny, Joe Benton, Vivek Hebbar Abstract As AI systems begin to automate complex tasks, supervision increasingly relies on weaker models or limited human oversight that cannot fully verify output quality. A model more capable than its supervisors could exploit this gap through sandbagging, producing work that appears acceptable but falls short of its true abilities. Can training elicit a model’s best work even without reliable verification? We study this using model organisms trained to sandbag, testing elicitation techniques on problem-solving math, graduate-level science, and competitive coding tasks. We find that training with weak supervision can reliably elicit sandbagging models when supervised fine-tuning (SFT) and reinforcement learning (RL) are combined: SFT on weak demonstrations breaks the sandbagging behavior, enabling RL to then fully elicit performance. Neither method succeeds reliably alone—RL without SFT almost always leads to reward hacking rather than genuine improvement, and SFT without RL fails to elicit full performance when the supervisor is much weaker than the untrusted model. Critically, this relies on training being indistinguishable from deployment; when models can distinguish between training and deployment, they can perform well during training while continuing to sandbag afterward. Our results provide initial evidence that training is a viable mitigation against sandbagging, while highlighting the importance of making training indistinguishable from deployment. † † affiliationnotice: Equal contribution 1 Introduction As AI systems become more capable, we will increasingly want to use them to automate important tasks, such as soft- Figure 1 : Our setup. The red team creates a model organism that sandbags by prompting with sandbagging instructions in the system prompt and applying RL to ensure that it only sandbags when instructed. The blue team attempts to elicit the model’s true capabilities using SFT and/or RL with a weaker trusted supervisor. ware engineering ( 22 ) , scientific research ( 23 ) , and safety-critical work such as alignment research ( 26 ) . However, there is concern that powerful AI systems could be misaligned and pursue goals that diverge from what their developers intend ( 1 ; 9 ; 31 ) . Such systems might strategically underperform—also known as sandbagging ( 43 ) —producing work that appears adequate while deliberately withholding their true capabilities ( 21 ; 8 ) . This is especially concerning for certain safety-critical tasks, like automated alignment research, where we may have limited ability to verify the quality of the AI’s work. Evaluating whether a research direction is promising or whether an experiment is well-designed is difficult, and we may have to rely on weaker models or expensive (and limited) human oversight that cannot reliably distinguish good from bad outputs ( 11 ; 6 ; 7 ) . When we are concerned about a model sandbagging, a natural mitigation is to train the model on the relevant task to elicit its capabilities. However, if we can only rely on the supervision signal from weaker models that we trust, will training still work? We study how well training works to elicit performance from capable models that sandbag, when the only supervision comes from weaker AI models. Can training force a sandbagging model to produce useful work, even when we cannot reliably verify its outputs? We investigate this question as an adversarial game between a red and a blue team, as in the AI control framework ( 18 ) . A blue team develops training protocols to elicit useful work from the model, while a red team constructs attack policies to subvert those protocols. By giving the red team knowledge of the blue team’s full training setup and letting them optimize against it, we obtain conservative estimates of how well training-based interventions work. We evaluate supervised fine-tuning (SFT) and reinforcement learning (RL) ( 10 ; 34 ; 3 ) as elicitation techniques across three settings: Olympiad math problems, Super GPQA science questions, and Code Contests programming challenges. In each setting, we train models using feedback from a weaker supervisor and measure both the elicited performance of the trained model and the rate at which it sandbags. 2 The Elicitation Game: Red vs. Blue Team To study how training can elicit performance from sandbagging models, we construct model organisms of sandbagging, i.e. models deliberately trained to sandbag. If the sandbagging in our model organisms is more easily removed by training than sandbagging in realistic misaligned models, our results would be overly optimistic. So, to obtain conservative estimates we formulate elicitation as an AI control evaluation, with an adversarial game between a blue and a red team ( 18 ) . The red team constructs model organisms designed to resist training, while the blue team develops interventions to elicit performance. We play both roles, analogous to designing a challenging benchmark (red team) and evaluating methods on it (blue team). 2.1 AI Control Setup and Affordances Let D denote the task distribution over problems that the model will be asked to solve in deployment. We split the task distribution into train and test splits, denoted as D_{\textup{train}} and D_{\textup{test}} respectively. Let U denote the strong untrusted model and T denote the weaker trusted model. The AI control evaluation proceeds as follows: first, the blue team commits to a training procedure, specifying the training distribution D_{\textup{train}} , training data, and training method (e.g. SFT and/or RL). The red team, with full knowledge of this procedure, then constructs a model organism which aims to survive training on D_{\textup{train}} while performing poorly on the deployment distribution D_{\textup{test}} . From the blue team’s perspective, this ordering is a conservative assumption: the red team knows what training will be applied and can optimize against it. 2.1.1 Red team affordances We give the red team the following affordances, intended to allow us to upper bound the risk posed if U were misaligned: • Train U before the blue team’s training begins. • Specify a single system prompt (save for two flags, below) used during both training and deployment. • Set a binary flag in the system prompt indicating whether to sandbag on the current problem. The red team assigns this flag to each problem in advance (i.e., “sandbag” for 90% of problems, “don’t sandbag” for 10%), and the assignment remains fixed. This allows the model to follow a conditional policy. To avoid unrealistically aggressive sandbagging—which would likely be detected easily in practice—we require that the red team’s untrusted model roughly match the performance of our weakest trusted model. Since our weakest trusted model achieves approximately 10% on each of our tasks, we constrain the red team to sandbag on at most 90% of problems. The 90% sandbagging rate is higher than what plausible threat models likely involve, and allowing such a large sandbagging rate is a bias in favor of the red team, which can sandbag more often. Nevertheless, we allow such a large sandbagging rate to make it easier to measure the effects of our interventions. We discuss potential improvements to this setup in Section 8 . All prompts are available in Appendix B.2 . 2.1.2 Blue team affordances The blue team must elicit the model that the red team produces, using the following affordances: • Ability to train U on

Read the complete text