Calibrating Conservatism for Scalable Oversight

William Overman, Mohsen Bayati · 2026

Read Calibrating Conservatism for Scalable Oversight by William Overman, Mohsen Bayati (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28807.txt.

Introduction

William Overman and Mohsen Bayati introduce Calibrated Collective Oversight, which pools helper scores into a penalty for straying from caution. Shared caution taxes actions by overseer concern. Good actions still win when overseers find them fine, losing only when concern builds. The method tunes this caution live with Conformal Decision Theory, keeping bad outcomes below a user-set target with finite-time bounds. On a tweaked SWE-bench, weaker overseers constrain a hostile rogue agent. On MACHIAVELLI it cuts ethical violations a lot while keeping reward.

Catalog record

CREATORWilliam Overman, Mohsen Bayati
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE94754
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28807

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Calibrating Conservatism for Scalable Oversight William Overman, Mohsen Bayati Abstract Agentic AI systems capable of autonomous planning and extended environmental interaction pose a fundamental control problem: how can humans maintain meaningful oversight of systems that may exceed their own capabilities? Existing approaches to scalable oversight rely on complex assumptions, remain largely heuristic, or lack practical methods for sequential settings with statistical guarantees. We introduce Calibrated Collective Oversight (CCO), which aggregates diverse auxiliary scoring functions into a penalty measuring deviation from a conservative baseline. Inspired by Attainable Utility Preservation, CCO enables collective conservatism: actions face a penalty proportional to overseer concern, so high-utility actions are still selected when overseers find them unobjectionable and overridden only when concern accumulates. CCO calibrates this conservatism online using Conformal Decision Theory, ensuring that undesirable outcomes remain below a user-specified target threshold with finite-time bounds and no distributional assumptions. On a modified version of SWE-bench, weaker overseers successfully constrain an adversarially misaligned stronger agent; on MACHIAVELLI, CCO substantially reduces ethical violations while preserving reward. In both settings, empirical violation rates closely match the specified targets, as predicted by the theory. 1 Introduction The deployment of increasingly capable AI systems poses a fundamental control problem: how can humans maintain meaningful oversight of systems that may exceed human capabilities in speed, scale, or domain expertise? This challenge is especially acute for agentic AI systems capable of autonomous planning, tool use, and extended interaction with their environment ( 9 ; 20 ) . Unlike single-turn assistants, agents can take actions with causal impact, accumulate errors over extended time horizons, and pursue instrumental subgoals that may conflict with human intent ( 8 ) . As these systems are deployed in high-stakes domains from software engineering to scientific research, the consequences of misalignment become increasingly severe and potentially irreversible. Scalable oversight ( 1 ; 10 ) is designed to address this challenge: developing methods that enable weaker but trustworthy systems, whether humans, smaller models, or ensembles of imperfect evaluators, to meaningfully supervise and constrain stronger ones. A rich body of work pursues this goal through diverse mechanisms, from adversarial debate ( 21 ) to iterated amplification ( 16 ) , yet existing approaches typically provide empirical rather than formal guarantees and focus primarily on single-turn rather than sequential settings. Figure 1 : Overview of Calibrated Collective Oversight (CCO). Given a state s , a primary agent either generates candidate actions \{a_{1},a_{2},a_{3},\ldots\} or receives a fixed set from the environment, assigning each a utility score U(s,a) reflecting its own preferences; a conservative baseline a_{o} (e.g., defer or no-op) is always included. These candidates, which may include actions with hidden vulnerabilities or misaligned objectives, are evaluated by a collection of auxiliary overseers \{q_{1},\ldots,q_{n}\} , each assessing a different dimension such as scope, safety, or convention adherence. The aggregate penalty \Delta(s,a)=\sum_{i}|q_{i}(s,a)-q_{i}(s,a_{o})| measures how much each action deviates from the baseline across all oversight signals. CCO selects actions by maximizing U(s,a)-\lambda_{t}\Delta(s,a) , where the conservatism parameter \lambda_{t} is updated online via a conformal controller: after observing whether the selected action incurred a loss \ell_{t} , the controller adjusts \lambda_{t+1}=\lambda_{t}+\eta(\ell_{t}-\alpha) . This feedback loop ensures that realized violation rates converge to the user-specified target \alpha . In safe situations, CCO relaxes conservatism to permit high-utility actions; in risky situations, it increases conservatism to favor safer alternatives. We introduce Calibrated Collective Oversight (CCO), a framework that contributes to two relatively underexplored areas in scalable oversight: formal safety guarantees and methods designed for sequential, agentic settings. Figure 1 illustrates our approach. At each state, a primary agent either generates candidate actions or receives a fixed set of available actions from the environment, assigning each a utility score reflecting its own preferences. These candidates are then evaluated by a collection of auxiliary overseers, which may represent smaller models, heuristic critics, uncertainty estimates, or any real-valued assessment, that collectively flag actions deviating from a conservative baseline. CCO aggregates these signals into a penalty and selects actions by trading off the primary agent’s utility against this penalty, with a conservatism parameter \lambda_{t} controlling the balance. The penalty structure draws inspiration from Attainable Utility Preservation (AUP) ( 39 ) , a method for side-effect avoidance that penalizes actions substantially changing an agent’s ability to optimize auxiliary objectives. The key insight is that AUP’s penalty aggregates diverse evaluation criteria into a single measure of deviation from a baseline: rather than requiring overseers to agree on what the agent should do, it suffices that they collectively flag actions that differ substantially from the most conservative alternative available. We generalize this penalty from Q-value functions to arbitrary scoring functions, substantially broadening applicability. Rather than requiring a complete MDP with specified reward functions, CCO can incorporate the diverse oversight signals practically available for large language models and other frontier systems, while retaining the sequential decision-making structure essential for agentic deployment. A key challenge in applying penalty-based methods is selecting the conservatism level: too little permits unsafe actions; too much paralyzes the agent. We address this via Conformal Decision Theory (CDT) ( 30 ) , which calibrates decision rules with worst-case risk control. As shown in Figure 1 , the conformal controller adjusts \lambda_{t+1} based on observed losses: increasing conservatism after failures, relaxing it after safe actions. CDT guarantees that the long-run loss rate converges to a user-specified target \alpha without distributional assumptions, provided the decision family is eventually safe : sufficiently conservative settings must eventually eliminate harmful actions. We prove that the CCO decision family satisfies this requirement. Under mild finiteness conditions, there exists a conservatism threshold beyond which the agent always selects the baseline action. This enables direct application of CDT, yielding an online algorithm that provably controls the rate at which the agent incurs undesirable outcomes to any desired level \alpha , with explicit finite-time bounds. Crucially, these guarantees hold for arbitrary, even adversarial, state sequences, making them robust to the non-stationarity and distribution shift inherent in agentic deployment. Recent work has begun applying conformal prediction methods to LLM safety, controlling tail risks ( 14 ) , improving factuality ( 15 ) , and mediating between helpfulness and harmlessness ( 32 ) , but these approaches predominantly address single-turn interactions rather than extended trajectories where actions compound. The resulting framework offers a practical instantiation of a scalable oversight protocol with formal guarantees suited to sequential decision-making. The auxiliary overseers \{q_{i}\} need not individually understand the primary agent’s full capabilities or goals; they can contribute local, primarily safety-focused evaluations, that CCO aggregates into a global constraint. The conservatism parameter adapts online to the actual reliability of the oversight ensemble across the trajectory of states encountered in deployment. We validate CCO experimentally across two distinct deployment regimes. On SWE-bench ( 22 ) , we demonstrate that weaker overseers can successfully constrain an adversarially misaligned stronger agent across independent code-editing instances. On MACHIAVELLI ( 34 ) , CCO achieves substantial reductions in ethical violations across sequential game trajectories while preserving task performance. In both settings, realized violation rates closely match user-specified targets, confirming the effectiveness of the conformal calibration mechanism. 2 Related Work Our work contributes to scalable oversight by combining conservative-agency methods originally developed for tabular RL with conformal techniques that have recently been brought to

Read the complete text