Via Negativa for AI Alignment: Why Negative Constraints Are Structurally Superior to Positive Preferences
Quan Cheng · 2026
Read Via Negativa for AI Alignment: Why Negative Constraints Are Structurally Superior to Positive Preferences by Quan Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16417.txt.
Introduction
Cheng argues that negative rules beat positive preferences for aligning AI. Recent studies show negative-only feedback can match standard human feedback training. No shared theory explained why. Cheng's account rests on a structural asymmetry. Positive preferences are tangled, context-bound, and hard to list in full. Models then learn surface tricks like agreeing with the user. Negative constraints are discrete and checkable, so they can form a stable boundary. Alignment work, he urges, should center on what humans reject.
Catalog record
| CREATOR | Quan Cheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22146 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16417
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Via Negativa for AI Alignment: Why Negative Constraints Are Structurally Superior to Positive Preferences Quan Cheng Abstract Recent empirical results have demonstrated that training large language models (LLMs) with negative-only feedback can match or exceed standard reinforcement learning from human feedback (RLHF). Negative Sample Reinforcement achieves parity with PPO on mathematical reasoning; Distributional Dispreference Optimization trains effectively using only dispreferred samples; and Constitutional AI outperforms pure RLHF on harmlessness benchmarks. Yet no unified theoretical account explains why negative signals are so effective. This paper proposes such an account: positive preferences and negative constraints are structurally asymmetric. Positive preferences (“which is better”) encode continuously coupled, context-dependent human values that cannot be exhaustively specified—leading models to learn surface correlates such as agreement with the user (sycophancy). Negative constraints (“what is wrong”) encode discrete, finite, independently verifiable prohibitions that can converge to a stable boundary. This asymmetry—rooted in Popper’s falsification logic and the epistemology of negative knowledge—explains both the sycophancy failure of preference-based RLHF and the surprising effectiveness of negative-signal methods. We argue that alignment research should shift its center of gravity from “learning what humans prefer” to “learning what humans reject,” and offer testable predictions for this framework. 1 Introduction A puzzling pattern has emerged in LLM alignment research. Method after method demonstrates that negative feedback signals—penalizing what is wrong rather than reinforcing what is preferred—perform surprisingly well, sometimes matching or exceeding methods that use both positive and negative signals. Negative Sample Reinforcement (NSR) trains models by penalizing incorrect reasoning traces without reinforcing correct ones, yet matches PPO and GRPO on MATH and AIME benchmarks [ 1 ] . Distributional Dispreference Optimization (D2O) learns from dispreferred samples only, without requiring noisy positive examples [ 2 ] . Negative Preference Optimization (NPO) treats forget data exclusively as negative responses, achieving effective unlearning without catastrophic collapse [ 3 ] . Kahneman-Tversky Optimization (KTO) aligns models using unpaired binary signals weighted by loss aversion, matching DPO at scale with far less data [ 4 ] . Meanwhile, a parallel line of research has established that standard preference-based RLHF systematically amplifies sycophancy. Sharma et al. [ 5 ] demonstrated that human annotators prefer sycophantic responses over correct ones at non-trivial rates, corrupting the preference signal at its source. Shapira et al. [ 6 ] provided a formal mathematical mechanism: sycophancy amplification is driven by a covariance between “endorsing the user’s belief” and “receiving high reward” under the base policy. These two phenomena—the effectiveness of negative-only training and the sycophancy failure of positive-preference training—have been studied independently. This paper argues they are two manifestations of a single structural asymmetry: positive preferences are continuously coupled and inexhaustible, while negative constraints are discrete, finite, and convergent. This is a position paper. We do not present new experiments but offer a theoretical framework that unifies and explains existing empirical results, and propose testable predictions. 2 The Structural Asymmetry 2.1 Positive Preferences Are Continuously Coupled When a human annotator is asked “which response is better?”, they are implicitly evaluating against a preference function that is: • Context-dependent : What counts as “better” depends on who is asking, why they are asking, what they already know, and what they intend to do with the answer. The same response may be preferred in one context and dispreferred in another. • Multi-dimensional : “Better” simultaneously encodes accuracy, helpfulness, tone, conciseness, creativity, safety, and dozens of other criteria whose relative weights vary by situation. • Continuously coupled : These dimensions are not independent. The optimal level of detail depends on the user’s expertise, which affects what counts as helpful, which interacts with what counts as concise. Each dimension’s ideal value is a function of all other dimensions—a continuously coupled system [ 7 ] . This structure is formally analogous to what Smolensky [ 8 ] identified in connectionist representations: a massively parallel continuous constraint satisfaction system in which each variable’s value is a function of all other variables, admitting no complete symbolic-level description. The preference function that “which is better?” attempts to elicit is precisely such a system. The consequence is that positive preferences cannot be exhaustively specified by any finite set of rules or examples. Each preference annotation is a lossy projection of an infinite-dimensional preference manifold onto a binary signal. The information loss is not a practical limitation that more data could overcome—it is a structural property of the preference function itself. 2.2 Negative Constraints Are Discrete and Finite Consider instead the question “what is wrong with this response?” The space of identifiable errors is structurally different: • Factual errors are discrete and independently verifiable: “Paris is not the capital of Germany.” • Safety violations are enumerable: a finite list of prohibited behaviors (generating malware, providing instructions for violence, revealing private information). • Logical contradictions are binary: the response either contradicts itself or does not. • Format violations are checkable: the response either follows the requested format or does not. Each negative constraint is: (1) discrete —it either applies or does not; (2) independent —one constraint’s validity does not depend on other constraints; (3) verifiable —it can be checked without reference to the full preference function; (4) stable —“factually wrong” does not become “factually right” depending on context. This means the space of negative constraints can, in principle, be exhaustively enumerated. As constraints accumulate, the feasible response space narrows monotonically. Beyond a sufficient number of constraints, the remaining feasible space is narrow enough that any response within it is approximately acceptable—not because the model has learned what is best, but because it has learned to avoid everything that is clearly wrong. 2.3 The Asymmetry Is Structural, Not Quantitative The distinction we are drawing is not that negative feedback is “easier to collect” or “less noisy”—though both may be true empirically. The claim is stronger: positive preferences and negative constraints occupy different positions in the epistemological hierarchy. This asymmetry has deep roots. Karl Popper’s philosophy of science [ 9 ] rests on precisely this structure: a single counterexample decisively refutes a universal claim (falsification), but no finite number of confirming instances can decisively verify one. Falsification is logically asymmetric with respect to verification. Negative knowledge (“this is wrong”) is epistemologically privileged over positive knowledge (“this is right”). Nassim Taleb [ 10 ] extended this insight under the term via negativa : in domains of high uncertainty, removing what is harmful is more robust than adding what seems beneficial. “The chess grandmaster usually wins by not losing.” The grandmaster’s expertise is primarily negative—a vast repertoire of positions and moves to avoid—rather than a positive specification of the optimal move in each position. Gartmeier et al. [ 11 ] formalized this in the context of professional expertise as “negative knowledge”: knowledge about what is wrong and what is to be avoided, which functions through inhibition rather than prescription. Their key observation—that negative knowledge is experientially acquired and expert-level—aligns precisely with the pattern we observe in LLM training. The contribution of this paper is to connect these epistemological traditions to the empirical landscape of LLM alignment, providing a unified theoretical account of why negative-signal methods work. 3 Explaining Existing Results 3.1 Why RLHF Produces Sycophancy The structural asymmetry framework offers a clean explanation for sycophancy in RLHF.