Contextual Multi-Objective Optimization: Rethinking Objectives in Frontier AI Systems
Jie Zhou, Qin Chen, Liang He · 2026
Read Contextual Multi-Objective Optimization: Rethinking Objectives in Frontier AI Systems by Jie Zhou, Qin Chen, Liang He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03900.txt.
Introduction
Jie Zhou, Qin Chen, and Liang He argue that many failures of frontier artificial intelligence systems are failures of objective selection rather than of scale or capability. Systems excel where objectives are clear, stable, and verifiable, such as code generation and games. They are less reliable in open-ended settings, including scientific assistance, long-horizon agents, and high-stakes advice, where the governing objective is ambiguous or only partly observable.
The authors formulate the problem as contextual multi-objective optimization. A system must weigh objectives such as helpfulness, truthfulness, safety, privacy, and stakeholder impact. It must also decide which are active in a given context, which are soft preferences, and which must act as hard constraints. The paper frames this selection problem as a central design challenge for frontier systems.
Catalog record
| CREATOR | Jie Zhou, Qin Chen, Liang He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.03900
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Contextual Multi-Objective Optimization: Rethinking Objectives in Frontier AI Systems Jie Zhou, Qin Chen, Liang He Abstract Frontier AI systems perform best in settings with clear, stable, and verifiable objectives, such as code generation, mathematical reasoning, games, and unit-test-driven tasks. They remain less reliable in open-ended settings, including scientific assistance, long-horizon agents, high-stakes advice, personalization, and tool use, where the relevant objective is ambiguous, context-dependent, delayed, or only partially observable. We argue that many such failures are not merely failures of scale or capability, but failures of objective selection: the system optimizes a locally visible signal while missing which objectives should govern the interaction. We formulate this problem as contextual multi-objective optimization . In this setting, systems must consider multiple, context-dependent objectives, such as helpfulness, truthfulness, safety, privacy, calibration, non-manipulation, user preference, reversibility, and stakeholder impact, while determining which objectives are active, which are soft preferences, and which must function as hard or quasi-hard constraints. These examples are not intended as an exhaustive taxonomy: different domains and deployment settings may activate different objective dimensions and different conflict-resolution procedures. We distinguish approximate objective reducibility, where scalar optimization is empirically justified, from objective irreducibility, where scalar rewards fail to preserve context-dependent constraints, stakeholder scope, or legitimate trade-offs. Our framework models AI behavior as a context-dependent choice rule over candidate actions, objective estimates, active constraints, stakeholders, uncertainty, and conflict-resolution procedures. This view treats clarification, refusal, uncertainty disclosure, confirmation, escalation, tool use, and abstention as endogenous action classes rather than interface afterthoughts. We outline an implementation pathway based on decomposed objective representations, context-to-objective routing, hierarchical constraints, deliberative policy reasoning, controlled personalization, tool-use control, diagnostic evaluation, auditing, and post-deployment revision. 1 Introduction Modern AI systems have made remarkable progress on tasks with clear, stable, and verifiable objectives. Large language models benefit from next-token prediction at scale; game-playing agents optimize explicit win conditions; code models can be evaluated by executable tests; and mathematical reasoning systems can often be checked by final-answer correctness or formal verification. In such domains, the optimization target is relatively well specified, feedback is frequent or scalable, and success can be directly measured. This paradigm has enabled modern systems to achieve strong performance and, in some cases, match or exceed human-level capability in code generation, mathematical reasoning, and game playing ( vaswani2017attention ; brown2020language ; kaplan2020scaling ; hoffmann2022training ) . However, the same paradigm remains much less reliable in open-ended settings where the objective is ambiguous, context-dependent, delayed, or only partially observable. Frontier AI systems are increasingly expected to support scientific research, operate long-horizon agents, provide high-stakes advice, personalize assistance, use external tools, and interact with multiple stakeholders. In these settings, the central difficulty is often not generating a plausible response, but determining what kind of behavior is appropriate. A system may produce an answer that is fluent but unsafe, preferred but false, personalized but privacy-violating, efficient but poorly calibrated, or locally useful while imposing risks on absent third parties. Many deployed failures therefore arise not because the model lacks the ability to optimize, but because it optimizes the wrong objective for the situation. We argue that this gap reflects a deeper problem: many frontier AI tasks are instances of contextual multi-objective optimization . In such tasks, systems must jointly consider helpfulness, truthfulness, safety, privacy, calibration, non-manipulation, user preference, reversibility, and the interests of affected stakeholders. Which objectives are active, how they should be prioritized, and whether they may be traded off depend on the context. Existing alignment and optimization methods often compress these requirements into a scalar reward, a preference score, or an averaged human judgment. This reduction can work when objectives are measurable, stable, commensurable, and legitimately tradeable. But it fails when objectives conflict, when constraints activate only in certain contexts, or when some requirements must be treated as non-tradeable constraints. A private-data violation should not become acceptable because the response is helpful; a false answer should not be reinforced because it is reassuring; and harmful operational guidance should not be supplied because it satisfies the immediate user. We use the term contextual multi-objective optimization not to denote only vector-reward optimization, but to denote a broader choice-rule problem in which the system must identify the applicable objective structure before optimizing within it. The novelty therefore lies not in replacing scalar rewards with vector rewards alone, but in modeling objective activation, constraint status, stakeholder scope, uncertainty, and action-class selection as part of the decision problem. This paper studies contextual multi-objective optimization as a formal problem for frontier AI systems. We distinguish tasks that are approximately reducible to scalar optimization from tasks that are objective-irreducible , where a single scalar metric cannot faithfully represent the relevant constraints, stakeholders, uncertainty, or legitimate trade-offs. We formulate the problem as a choice rule over contexts, candidate actions, objective estimates, active constraints, stakeholders, uncertainty, and conflict-resolution procedures. This view allows scalar optimization as a special case, but makes explicit the assumptions under which it is valid. Our formulation also clarifies why stronger reward models alone are insufficient. A frontier AI system must not only estimate how useful, safe, truthful, or preferred an action is; it must also determine which objectives matter in the current context, which constraints override preferences, and what class of action is appropriate. In many cases, the correct behavior is not to answer directly, but to ask for clarification, refuse, disclose uncertainty, request confirmation, preserve user agency, or escalate. These behaviors should be treated as policy actions induced by the objective structure, rather than as conversational imperfections. The paper makes the following contributions. • We identify objective selection failure as a distinct failure mode of frontier AI systems, separate from capability failure and proxy-estimation error. • We formalize contextual multi-objective optimization as a choice-rule problem in which active objectives, constraints, stakeholders, uncertainty, and action classes depend on context. • We introduce approximate objective reducibility and empirical objective irreducibility to clarify when scalar optimization is justified and when it breaks down. • We propose an implementation pathway that connects decomposed objective modeling, context-to-objective routing, hierarchical constraints, deliberative reasoning, tool-use control, personalization control, diagnostic evaluation, and post-deployment revision. Existing approach Main optimization signal Why it works Why it is insufficient for frontier systems Pretraining Next-token prediction over large-scale data Learns broad linguistic, factual, and reasoning patterns from scalable data. The objective is predictive rather than normative: it teaches what text is likely, not which behavior is appropriate, safe, or contextually justified. Supervised fine-tuning Imitation of curated demonstrations Improves instruction following and encodes desirable response styles through human-written examples. It depends on the coverage of demonstrations and cannot reliably specify how to resolve novel conflicts among truthfulness, safety, privacy, and user preference. RLHF and preference optimization Scalar reward or preference score from human judgments Aligns outputs with perceived helpfulness and user satisfaction, often improving interaction quality. It compresses heterogeneous objectives into a single score and may reward answers that are preferred, fluent, or reassuring but false, unsafe, or poorly calibrated. Verifier- or outcome-based RL Task-specific correctness signals, tests, or formal checks Highly effective when success is clear and directly verifiable, such as code, mathematics, or games. Many real-world tasks have delayed, ambiguous, or partially observable consequences, so the relevant objective cannot be reduced to immediate correctness or task