Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models

William Overman, Mohsen Bayati · 2025

Read Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models by William Overman, Mohsen Bayati (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00911.txt.

Introduction

Deploying a language model means juggling rivals: helpful against harmless, cheap against accurate, rewarding against safe. William Overman and Mohsen Bayati introduce Conformal Arbitrage in a 2025 paper. It is a referee that routes each request between a bold primary model and a cautious guardian, which may be another model or a human expert.

A learned threshold decides who answers, tuned by conformal risk control so the long-run rate of bad events, errors or safety breaches, stays under a quota the user sets. The method works at the API level, never touching model weights or internals. In tests it traced a clean frontier between the competing goals and beat cost-matched random routing on accuracy. The authors pitch it as a practical, guarantee-backed layer for trustworthy, economical model deployment.

Catalog record

CREATORWilliam Overman, Mohsen Bayati
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00911

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models William Overman, Mohsen Bayati Abstract Modern language - model deployments must often balance competing objectives —for example, helpfulness versus harmlessness, cost versus accuracy, and reward versus safety. We introduce Conformal Arbitrage , a post - hoc framework that learns a data - driven threshold to mediate between a Primary model optimized for a primary objective and a more conservative Guardian —which could be another model or a human domain expert—aligned with a guardrail objective. The threshold is calibrated with conformal risk control, yielding finite - sample, distribution - free guarantees that the long - run frequency of undesirable events (such as factual errors or safety violations) does not exceed a user - specified quota. Because Conformal Arbitrage operates wholly at the API level—without requiring access to model logits or updating model weights—it complements weight - based alignment techniques and integrates seamlessly with existing cost - aware cascades. Empirically, Conformal Arbitrage traces an efficient frontier, allowing users to define an acceptable performance level for one objective while maximizing utility in another. We observe that our method outperforms (in terms of accuracy) cost-matched random routing between models. These properties make Conformal Arbitrage a practical, theoretically grounded tool for trustworthy and economical deployment of large language models across a broad range of potentially competing objectives. 1 Introduction Large language models (LLMs) excel at reasoning, coding, and open - domain question answering, yet real - world deployments frequently need to navigate tensions between potentially competing objectives such as helpfulness and harmlessness or cost and accuracy . Current practices mostly tackle the tension between helpfulness and harmlessness by modifying the model itself : reinforcement learning from human feedback (RLHF) ( Christiano et al. 2017 ; Ouyang et al. 2022 ) , direct–preference optimisation (DPO) ( Rafailov et al. 2023 ) , Constitutional AI ( Bai et al. 2022b ) , or multi - objective fine - tuning ( Zhou et al. 2023 ; Wang et al. 2024 ) each produce a single operating point along the Pareto frontier. While powerful, these methods demand expensive data collection, GPU - intensive retraining, and — for API - only models — are often not applicable. For the cost versus accuracy tradeoff, there has been significant work on cascades: a cheap model handles easy queries and defers the rest to a stronger fallback ( Chen et al. 2023 ; Aggarwal et al. 2025 ; Zellinger et al. 2025 ) . Recently, Jung et al. 2025 introduced Cascaded Selective Evaluation (CSE), calibrating per-model confidence estimators via fixed-sequence multiple testing to obtain rigorous guarantees on alignment to human pairwise preferences. However, these approaches are tailored for controlling a binary disagreement risk, while a user may be interested in controlling arbitrary guardrail metrics at deployment time. We introduce Conformal Arbitrage (CA) , a lightweight router that sits outside the language models. The term “arbitrage” captures how our approach exploits the performance gap between specialized models to achieve superior outcomes than naive selection between models. Given (i) a Primary model optimized for the primary objective and (ii) a more conservative Guardian model or a human domain expert, aligned with a guardrail objective, CA offers a principled alternative to randomized routing between models. Instead of merely alternating between models with some probability, CA learns a single scalar threshold on how strongly the Primary model favors its top choice over alternatives (a notion we formally define as “score gap” later in the paper). This threshold determines when the Primary model’s confidence is sufficient to act upon its prediction versus when to defer to the Guardian, creating a principled decision boundary that optimizes the trade-off between objectives. The threshold is calibrated using conformal risk control (CRC) ( Angelopoulos et al. 2024 ) , yielding finite - sample, distribution - free guarantees that the long - run frequency (or magnitude) of undesirable events never exceeds a user - specified budget \alpha . This enables precise control over trade-offs—users can explicitly specify how much they are willing to compromise on one objective to gain on the other. Because CA touches no model weights , it complements weight - based alignment and applies to closed, black - box APIs, making it a remarkably lightweight approach to achieving Pareto improvements over simple model selection strategies. Our experiments examine both the the cost versus accuracy tradeoff using the the TruthfulQA and MMLU benchmarks as well as the helpfulness versus harmlessness tradeoff using the PKU-SafeRLHF benchmark. Across all settings CA traces an efficient frontier that outperforms random or cost - matched routing baselines. Conformal Arbitrage transforms an immutable, potentially unpredictable LLM (or a family of LLMs) into a controllable system whose risk–utility position can be dialed after deployment . In our experiments, we demonstrate this capability using state-of-the-art LLMs from the GPT-4.1 series, OpenAI 2025 , showing how our method enables fine-grained control over various tradeoffs without modifying the underlying models. By requiring only a few hundred logged examples for calibration, CA offers a pragmatic path toward trustworthy, cost - efficient and customizable language - model services that can be adjusted to meet evolving requirements long after initial deployment. 2 Related work Real–world deployments must strike a pragmatic balance between helpfulness —supplying users with accurate and detailed information—and harmlessness —avoiding policy-breaking or dangerous content. Early alignment work framed the problem as a single–objective optimization: RLHF ( Christiano et al. 2017 ; Ouyang et al. 2022 ) and its variant DPO ( Rafailov et al. 2023 ) collapse nuanced feedback into a single reward model and therefore deliver one operating point on the Pareto frontier. Subsequent methods introduced explicit two–factor training: RLHF on mixed helpful–harmless datasets ( Bai et al. 2022a ) , Constitutional AI’s self-revision loop ( Bai et al. 2022b ) , and Bi - Factorial Preference Optimisation (BFPO) ( Zhang et al. 2025 ) that casts the bi - objective RLHF loss as a direct supervised criterion. Safe - RLHF ( Dai et al. 2023 ) separates a reward and a cost head and enforces constraints by Lagrangian relaxation, while Circuit Breakers intervene at generation time to halt policy - violating continuations ( Zou et al. 2024 ) . The PKU-SafeRLHF benchmark ( Ji et al. 2023 ) was specifically introduced to quantify this helpfulness-harmlessness trade-off, providing dual annotations that enable researchers to measure progress on both dimensions simultaneously. Anthropic’s Constitutional AI ( Bai et al. 2022b ) further explores alignment by embedding principles directly into model training. More recently, the MoGU framework ( Du et al. 2024 ) dynamically routes between model variants optimized separately for usability and safety. Empirically, while these approaches curb unsafe completions, they still lock the model into one fixed balance point between helpfulness and harmlessness. Beyond helpfulness–harmlessness many other objectives— accuracy, cost, latency, fairness, demographic parity, domain–specific risk, etc.—can be in conflict. Many recent works have proposed weight–based strategies to navigate the resulting frontiers between such competing objectives. Rewarded soups linearly interpolates checkpoints fine - tuned on distinct rewards to trace that surface ( Ramé et al. 2023 ) , Directional Preference Alignment adds multiple reward heads for steerable inference ( Wang et al. 2024 ) , MaxMin - RLHF learns a mixture of

Read the complete text