Constitutional Black-Box Monitoring for Scheming in LLM Agents
Simon Storf, Rich Barton-Cooper, James Peters-Gill, Marius Hobbhahn · 2026
Read Constitutional Black-Box Monitoring for Scheming in LLM Agents by Simon Storf, Rich Barton-Cooper, James Peters-Gill, Marius Hobbhahn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00829.txt.
Description
Storf and three colleagues study constitutional black-box monitors for detecting scheming in LLM agents, where agents covertly pursue misaligned goals. Their monitors are prompted classifiers using only outside inputs and outputs, tuned on synthetic data from plain behaviour notes. Two pipelines, STRIDE and Gloom, generate 1,000 samples each, and monitors tuned on them are tested on 7,500 held-out runs from ControlArena. Monitors chosen on synthetic data work in realistic settings, but performance levels off quickly, simple prompt sweeps match heavier tuning, and pushing further leads to overfitting.
Catalog record
| CREATOR | Simon Storf, Rich Barton-Cooper, James Peters-Gill, Marius Hobbhahn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99207 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00829