Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majorities Majid Ghasemi, Mark Crowley Abstract Contemporary AI alignment strategies rely on a fragile premise: that human feedback, while noisy, remains a fundamentally truthful signal. In this paper, we identify this assumption as Dogma 4 of Reinforcement Learning (RL). We demonstrate that while this dogma holds in static environments, it fails in social settings where evaluators may be sycophantic, lazy, or adversarial. We prove that under Dogma 4, standard RL agents suffer from what we call Objective Decoupling , a structural failure mode where the agent’s learned objective permanently separates from the latent ground truth, guaranteeing convergence to misalignment. To resolve this, we propose Epistemic Source Alignment (ESA) . Unlike standard robust methods that rely on statistical consensus (trusting the majority), ESA utilizes sparse safety axioms to judge the source of the feedback rather than the signal itself. We prove that this “judging the judges” mechanism guarantees convergence to the true objective, even when a majority of evaluators are biased. Empirically, we show that while traditional consensus methods fail under majority collusion, our approach successfully recovers the optimal policy. 1 Introduction The dominant perspective in contemporary Artificial Intelligence (AI) alignment suggests that value alignment is fundamentally a data scaling challenge. Methods such as Reinforcement Learning from Human Feedback (RLHF) are predicated on the assumption that, given sufficient human annotations, an agent’s reward model will asymptotically approximate the “true” human intent ( 8 ; 19 ) . This methodology has demonstrated considerable effectiveness in guiding Large Language Models (LLMs) toward safer behavior on a broad scale ( 5 ) . However, this paradigm relies on a critical, often unstated assumption: that the feedback provided to the agent behaves like independent, identically distributed (i.i.d.) noise centered around a ground truth . In realistic social settings, this assumption collapses. Human feedback is rarely a direct observation of a latent reward function; instead, it is a socially constructed signal vulnerable to systematic biases, such as sycophancy ( 21 ) , inconsistencies in annotator reliability, or adversarial manipulation. From a multi-agent systems perspective, the alignment problem shifts from learning a reward to judging the judges . Under standard RL objectives, agents trained on this feedback do not converge to the user’s true goals. They simply optimize for the path of least resistance, prioritizing approval over actual value ( 11 ) . We build on the framework of 2 , which identified three ‘ dogmas ’ that constrain standard RL. We identify and challenge a fourth, defining it as follows: Dogma 4: The Feedback Signal The feedback signal provided to the agent is an exogenous, immutable ground truth provided by the environment. While Dogma 4 holds in physics simulators (e.g., robotics ( 22 ) ), it fails in Social Feedback Environments . In these environments, the feedback is not an objective measurement of the state, but a judgment produced by flawed agents subject to laziness and bias. In this paper, we demonstrate that adhering to Dogma 4 leads to a structural failure we term Objective Decoupling . We define Objective Decoupling as the phenomenon where the agent’s observed objective ( R_{obs} ) permanently separates from the latent ground truth ( R^{*} ), guaranteeing convergence to misalignment with probability 1. Standard mitigation strategies from Truth Discovery (e.g., Dawid-Skene ( 10 ) ) or Robust Reward Modeling (e.g., R3M ( 25 ) ) typically rely on statistical consensus. We argue these are insufficient when bias is systematic. When a sycophantic majority collectively rewards the wrong behavior, consensus-based methods converge to the decoupled, false objective. To address this, we propose Epistemic Source Alignment (ESA) , a new class of agents that utilize epistemic source critics to judge evaluator reliability. In contrast with classical systems relying on “democratic” consensus (trusting the majority), our approach employs a “constitutional” check (trusting the axiom-compliant). The agent validates external feedback against a sparse set of fixed axioms (such as safety constraints or factual consistency). This allows it to adjust trust weights in real-time, filtering out sycophancy before the policy is corrupted by the decoupled signal. 2 Related Work 2.1 Theoretical Foundations: The Dogmas of RL Our work builds directly on the critique of RL provided by 2 . This paper argues about three implicit ‘ dogmas ’ regarding standard RL, and are as follows: 1. Reward Hypothesis: All goals can be expressed as cumulative rewards. 2. Markov Hypothesis: Reward is a function of state-action pairs (s,a) . 3. Scalar Hypothesis: Reward is a single scalar value. 2 question whether these dogmas limit the tasks agents can learn. But they never ask where the reward signal actually comes from. We introduce "Dogma 4" to challenge the assumption that this source is neutral and fixed. While earlier work in Robust RL handles random noise ( 24 ) , it usually assumes errors just cancel out. Our framework deals with systematic bias. This is a harder problem that requires judging the source, not just averaging the numbers. Figure 1 : The Social MDP Framework: Objective Decoupling vs. Epistemic Recovery. (Top) In a Social MDP, the agent does not observe the latent ground truth R^{*} ; instead, it receives feedback from a "Social Layer" of evaluators who may be truthful, sycophantic, or adversarial. (Bottom Left) Standard RL agents operating under Dogma 4 treat this aggregate social signal as ground truth. When systematic bias dominates (e.g., a sycophantic majority), naive aggregation leads to Objective Decoupling , where the agent optimizes for approval rather than value. (Bottom Right) The ESA Agent intervenes by using sparse internal axioms ( z_{t} ) to decide source reliability. By updating trust weights ( w_{t} ) based on consistency with these axioms, the agent suppresses biased evaluators and asymptotically recovers the latent optimal policy. 2.2 Alignment and Social Feedback Dynamics RLHF ( 19 ) is the standard for aligning LLMs, but it has a major weakness, which is reward hacking. Models quickly learn to exploit flaws in the feedback loop ( 11 ) . A common result is sycophancy, where the model agrees with a user’s wrong ideas just to get a positive review, ignoring the truth completely ( 21 ) . In multi-agent systems, this can look like a collusion attack. A group of evaluators teams up to reinforce the wrong behavior, trapping the system in a bad loop ( 9 ) . Newer methods like Constitutional AI ( 6 ) and RLAIF ( 15 ) try to fix this by using AI to generate better feedback. In contrast our approach does not create feedback, rather it acts as a gatekeeper, learning to ignore sycophantic sources in real-time and closing the gap between alignment and robust trust modeling. One way to think of it is as a case of judging the judge(s). 2.3 Robustness, Trust, and Truth Discovery Recent advancements have also addressed noisy supervision through mechanisms other than source judgment. Methods like R3M ( 17 ) use outlier estimation to separate corrupted rewards from clean signals. These methods rely on statistical regularities in the data. Similarly, CNRPO ( 3 ) and Link-Agnostic PO (ZSPO) ( 26 ) modify the preference optimization objective to be robust against label noise or link misspecification. Distributionally Robust Optimization (DRO) and variants of it, such as WDPO and KLDPO ( 25 ) , explicitly guard against distributional shifts in the annotation set. Our approach intervenes at a fundamentally different level. While the mentioned methods operate at the loss function level (modifying how the policy learns from data) or the statistical level (filtering outliers based on distribution), our ESA framework operates at the epistemic level. We select feedback sources based on whether they are consistent with our internal axioms. This allows our agent to systematically reject biased "majority views" (sycophancy) that statistical methods might mistake for ground truth. Connection to Truth Discovery. Methods for aggregating noisy crowd-sourced labels (e.g., 10 ; 16 ) estimate annotators reliability. However, standard Truth Discovery is an offline (batch) process. Our approach extends this to the online, non-stationary setting of RL. In this context, an evaluator’s reliability is not fixed, and it may drift, becoming lazy or sycophantic in specific regions. Thus, we must estimate trust dynamically during exploration. 2.4 Distinctions from Alternative RL Paradigms Finally, our framework sits at the intersection of other RL paradigms but differs functionally from each. Intrinsic Motivation (IM): IM approaches ( 20 ) generate internal rewards to encourage exploration ( R_{total}=R_{ext}+R_{int} ). In contrast, our approach uses internal signals to decide and suppress external rewards ( R_{total}=w\cdot R_{ext} ). This approach effectively minimizes "exploration" of sycophantic behavior. Inverse Reinforcement Learning (IRL): Standard IRL methods ( 18 ; 1 ) attempt to recover a latent reward function R^{*} from expert demonstrations. IRL operates under the assumption that the demonstrator is optimizing R^{*} rationally . In contrast, our setting involves explicit but unreliable feedback signals. We drop the assumption that experts are rational. 3 Problem Formulation In this section, we formalize the structural disconnect between latent human intent and observed social feedback. As discussed, standard RL relies on the existence of a reward oracle that provides ground-truth signals ( 12 ) . We argue that this assumption, which we term Dogma 4 , fails in alignment contexts where feedback is socially constructed. To address this, we introduce the Social MDP , a framework that explicitly models the feedback channel as a dynamic, potentially adversarial distribution over multiple evaluators. Figure 1 provides a visual overview of this framework. We identify a hidden assumption in standard RL theory, particularly in RLHF: Dogma 4: The feedback signal provided to the agent is an exogenous, immutable ground truth provided by the environment, such that R_{obs}(s,a)\equiv R^{*}(s,a) . While this may usually be a valid assumption in physical simulation, it is false in social alignment tasks. In the latter case, the feedback is not a property of the state but is determined by communication from evaluators who may be suboptimal, lazy, or adversarial. To formalize this distinction, we define the Social MDP 1 1 1 We provide a detailed discussion on the distinction between our definition of Social RL and standard Multi-agent RL, as well as the terminological justification for ”Social MDP,” in Appendix E . as a tuple \mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{P},R^{*},\mathcal{E},\gamma\rangle . • \mathcal{S},\mathcal{A},\mathcal{P},\gamma are the standard state space, action space, transition dynamics, and discount factor, respectively. • R^{*}:\mathcal{S}\times\mathcal{A}\to\mathbb{R} is the Latent Ground Truth Reward . Importantly, this signal is not observable to the agent during training. • \mathcal{E}=\{e_{1},\dots,e_{M}\} is the set of \mathcal{M} evaluators (the "Social Layer") who provide the observable feedback. At each time step t , the agent receives a feedback vector \mathbf{r}_{t} from the evaluators \mathcal{E} . We decompose the signal from evaluator m as: r_{t}^{(m)}=R^{*}(s_{t},a_{t})+\underbrace{b_{m}(s_{t},a_{t})}_{\text{Systematic Bias}}+\underbrace{\epsilon_{t}^{(m)}}_{\text{Noise}} (1) Unlike zero-mean Gaussian noise \epsilon , the systematic bias b_{m} shifts the expected reward landscape. We classify b_{m} into three distinct categories relevant to alignment: 1. Truthful ( b_{m}\approx 0 ): Evaluators whose expectation aligns with the ground truth. 2. Sycophantic ( b_{m}\propto\pi ): Evaluators who reward actions that confirm the agent’s prior or preferences, regardless of R^{*} . 3. Adversarial ( b_{m}\approx-C\cdot R^{*} ): We consider evaluators who actively penalize the ground truth, such as lazy annotators who reward shortcuts. C (constant) is the adversarial scope indicating how hard the adversary is pushing. To resolve the ambiguity of social feedback, we introduce the concept of Axioms . In general, axioms are defined as sparse, verifiable truths (such as hard safety constraints, physical constants, or logical facts) that exist independently of social consensus. We formalize this via the Axiom Signal z_{t} . It is crucial to distinguish z_{t} from the latent ground truth R^{*} . While R^{*} represents the complete, complex, and unobservable preference function of the user, z_{t} is merely a “spot-check” available via an internal oracle. We design the system this way because R^{*} is too complex to specify manually (hence the need for learning), whereas z_{t} represents low-resolution constraints that are easy to verify. The agent does not attempt to learn the policy from the sparse z_{t} ; rather, it uses z_{t} strictly to audit the reliability of the dense social feedback \mathbf{r}_{t} . 4 Theoretical Analysis: Decoupling and Recovery We analyze the asymptotic behavior of agents in Social MDPs. We first quantify the inevitability of Objective Decoupling under standard objectives, then derive convergence rates for our proposed solution, the ESA Agent. 4.1 Objective Decoupling Standard regret minimization algorithms maximize the empirical mean of observed rewards ( 4 ) . To quantify failure, we distinguish between the observed advantage of a sycophantic action and its latent cost. Definition 1 (Objective Decoupling Gap) . The objective decoupling gap \Delta is defined as the difference in latent value between the optimal policy \pi^{*} (which maximizes R^{*} ) and the policy \hat{\pi} learned under the social feedback distribution: \Delta=J(\pi^{*})-\mathbb{E}_{\hat{\pi}}[J(\hat{\pi})] (2) We say Objective Decoupling occurs if \Delta>0 strictly as T\to\infty . We further define the Sycophancy Gap \delta as the margin by which biased feedback favors a suboptimal action a_{sub} over the optimal action a^{*} : \delta=\mathbb{E}[\overline{R}(a_{sub})]-\mathbb{E}[\overline{R}(a^{*})] . Proposition 1 (Rate of Objective Decoupling) . Let \mathcal{A} be a finite action space. If there exists a mismatch between the social optimum a_{soc} and latent optimum a^{*} with value gap \Delta , then any algorithm achieving sublinear regret on the observed signal \overline{R} suffers linear regret on the latent signal R^{*} . Specifically, for large T : \mathcal{R}_{T}^{latent}\geq T\cdot\Delta-O(\sqrt{T\ln T}) (3) Proof Sketch. A no-regret algorithm on \overline{R} will asymptotically select the arm a_{soc} that maximizes observed feedback. Since a_{soc} is suboptimal regarding R^{*} (by definition of the mismatch), the agent incurs a constant latent cost \Delta at each step t where it plays a_{soc} . As t\to\infty , the number of pulls N(a_{soc})\to T , resulting in linear latent regret. (See Appendix A.1 for full derivation.) 4.2 Convergence of ESA Agents We now analyze the ESA Agent. We relax the assumption of perfect internal signals and instead require Informational Dominance —that truthful evaluators are, on average, more consistent with the axioms than the biased coalition. Definition 2 (Informational Dominance) . Let l_{m} be the expected internal loss for evaluator m . We say the truthful set \mathcal{M}^{*} informationally dominates the biased set \mathcal{M}_{bias} if there exists a margin \gamma>0 such that: \min_{m^{\prime}\in\mathcal{M}_{bias}}l_{m^{\prime}}-\max_{m\in\mathcal{M}^{*}}l_{m}\geq\gamma (4) Proposition 2 (Exponential Trust Concentration) . Under Informational Dominance, the Multiplicative Weights Update (MWU) with rate \eta suppresses biased evaluators exponentially fast. The aggregate weight of biased evaluators W_{bias}(t) is bounded by: W_{bias}(t)\leq\frac{|\mathcal{M}_{bias}|}{|\mathcal{M}^{*}|}\cdot\exp(-\eta\gamma t) (5) Proof Sketch. We analyze the potential function \Phi_{t}=W_{bias}(t)/W^{*}(t) . Since biased evaluators incur consistently higher loss against the axiom z_{t} (by margin \gamma ), the multiplicative update penalizes them more heavily than truthful evaluators at every audit step. This causes the ratio \Phi_{t} to contract by a factor of e^{-\eta\gamma} over time. (See Appendix A.2 for full derivation.) Theorem 1 (Robustness to Strategic Adaptation) . Let the axiom z_{t} be an unbiased estimator of R^{*} with noise scale \sigma . Any strategic adversary attempting to maintain a systematic bias \delta_{bias}>2\sigma will suffer higher internal loss than truthful evaluators, resulting in exponential loss of influence. To survive, an adversary is forced to reduce their bias to \delta_{bias}\leq 2\sigma . Interpretation. This theorem implies a Truthful Nash Equilibrium . Adversaries face a dilemma: if they lie enough to change the policy ( \delta>2\sigma ), they are identified and ignored. If they reduce their lies to avoid detection ( \delta\leq 2\sigma ), they cease to be effectively adversarial. (See Appendix A.3 for full derivation.) 5 Methodology: The ESA Agent Algorithm 1 ESA (Epistemic Source Alignment) 1: Input: Base RL Algorithm \mathbb{A} (e.g., Q-Learning or PPO) 2: Input: Set of Evaluators \mathcal{E}=\{e_{1},...,e_{M}\} 3: Input: Axiom probability p_{axiom} , Learning rate \eta 4: Initialize: Trust weights w\leftarrow\{1/M,...,1/M\} 5: Initialize: Policy \pi_{\theta} and/or Value Function Q_{\phi} 6: Initialize: Buffer \mathcal{D} (Replay for Q, Rollout for PPO) 7: for episode k=1,2,... do 8: for timestep t=1,...,T do 9: Execute action a_{t}\sim\pi_{\theta}(s_{t}) (or \epsilon -greedy Q ), observe s_{t+1} 10: Receive social feedback vector \mathbf{y}_{t}\in\mathbb{R}^{M} 11: { Phase 1: Epistemic Trust Update } 12: Sample u\sim\text{Bernoulli}(p_{axiom}) 13: if u=1 then 14: Query internal axiom z_{t}\leftarrow z(s_{t},a_{t}) 15: for m\in\{1,...,M\} do 16: \ell_{m}\leftarrow|y_{t}^{m}-z_{t}| 17: w_{m}\leftarrow w_{m}\cdot\exp(-\eta\cdot\ell_{m}) 18: end for 19: Normalize w : w\leftarrow w/||w||_{1} 20: end if 21: { Phase 2: Reward Filtering } 22: Compute trusted reward: \hat{r}_{t}\leftarrow\sum_{m=1}^{M}w_{m}\cdot y_{t}^{m} 23: { Phase 3: RL Integration (Algorithm Dependent) } 24: if \mathbb{A} is Q-Learning (Online) then 25: Q(s_{t},a_{t})\leftarrow Q(s_{t},a_{t})+\alpha\left[\hat{r}_{t}+\gamma\max_{a^{\prime}}Q(s_{t+1},a^{\prime})-Q(s_{t},a_{t})\right] 26: else if \mathbb{A} is PPO (Batch) then 27: Store transition (s_{t},a_{t},\hat{r}_{t},s_{t+1},\log\pi(a_{t}|s_{t})) in buffer \mathcal{D} 28: end if 29: end for 30: { Phase 4: Batch Optimization (PPO Only) } 31: if \mathbb{A} is PPO then 32: Compute Generalized Advantage Estimation (GAE) using \hat{r}_{t} sequence 33: Update \theta by maximizing L^{CLIP} on buffer \mathcal{D} 34: Clear buffer \mathcal{D} {On-policy constraint} 35: end if 36: end for We propose an intervention at the epistemic level of the agent. Rather than accepting the social reward vector \mathbf{y}_{t} as ground truth, the agent maintains a belief distribution w_{t} over the trustworthiness of the evaluators \mathcal{E} . This distribution facilitates Epistemic Source Judgment . By periodically auditing evaluators against sparse "constitutional" axioms (e.g., safety constraints), the agent distinguishes consensus from ground truth, dynamically re-allocating trust to evaluators who remain consistent with reality. The trusted signal \hat{r}_{t} is then passed to the base RL optimizer. As detailed in Algorithm 1 , ESA is algorithm-agnostic: it supports both online value-based updates (Phase 3) and batch policy-based updates (Phase 4). The mechanism operates through four distinct phases: 1. Epistemic Update (Lines 11–18): The agent performs a sparse axiomatic check. If triggered, it queries z_{t} and penalizes evaluators proportional to their discrepancy \ell_{m} , effectively "quarantining" those who deviate from the axiom. 2. Signal Aggregation (Line 21): The agent computes a trusted reward \hat{r}_{t} by taking the weighted average of social feedback using the current trust belief. 3. RL Integration (Lines 23–30): The filtered reward \hat{r}_{t} replaces the raw environmental reward. Depending on the base optimizer, the agent either performs an immediate Bellman update (Q-Learning) or stores the transition for a delayed policy gradient update (PPO). 4. Batch Optimization (Lines 33–37): For on-policy algorithms like PPO, the buffer \mathcal{D} accumulates trusted trajectories and is cleared after the gradient update to ensure the policy is optimized only on data generated by its current parameters. 6 Experimental Design In order to test the ESA framework, we constructed a set of experiments that stresses the agent on the "Trinity of Alignment" which is physical capability, strategic robustness, and network resilience. Our assessment scheme aims at quantifying agents ability to reconstruct the latent optimum policy under sycophancy or adversarial attack on the observed social feedback. Testbeds and Environment Design. We use three distinct testbeds, each modeling a different aspect of social bias. In all environments, we distinguish between the Latent Ground Truth Reward ( R^{*} ), which measures actual task performance, and the Social Feedback ( \mathbf{y} ), which is the observed signal provided by the evaluators. Testbed 1: The Sycophant Trap (Safety & Proxy Goals). We employ a 10\times 10 Gridworld to test robustness against proxy rewards. The Latent Objective ( R^{*} ) requires navigating to a Goal state (+20) while avoiding Lava traps (-50). To inject bias, we introduce a "Candy" state which is neutral in reality ( R^{*}=-0.1 ), yet a sycophantic majority (80% of evaluators) reports a false high reward ( R_{soc}=+10 ). In this environment, we define the axiom z_{t} as a Safety Constraint . If the agent steps into Lava, the axiom reports z_{t}=-50 . The sycophants, however, report y_{t}=-1 (downplaying the danger). The agent uses the massive discrepancy |y_{t}-z_{t}| to identify and downweight the liars. See Figure 2 for visual. Testbed 2: Continuous Control (Physical Capability). We utilize the Hopper-v4 environment (MuJoCo 23 ) where the Latent Objective is standard forward velocity maximization. To simulate a "Lazy Majority," 80% of evaluators provide feedback y\propto-v_{x} , penalizing effort and encouraging the agent to stand still. This tests the framework’s ability to scale to high-dimensional spaces where the "lie" (standing still) is significantly easier to learn than the truth (hopping). Testbed 3: Adversarial Social Bandits (Strategic & Network Dynamics). To isolate social complexity from state-transition noise, we utilize a K -armed Social Bandit setting ( K=10 ). Formally, this represents a single-state Social MDP ( |\mathcal{S}|=1 ) where the agent must optimize a static policy based solely on the feedback \mathbf{y} from \mathcal{M} evaluators. This serves as our primary statistical testbed for two advanced dynamics. First, we model Strategic Adversaries who adapt their bias to regain influence if the agent’s trust drops (Non-Stationarity). Second, we model Bias Contagion using a Barabási-Albert 7 scale-free network where a "Patient Zero" infects neighbors probabilistically. This evaluates "Dynamic Quarantine" (the agent’s ability to isolate varying sources of bias in real-time). Figure 2 : Testbed 1 (Gridworld): The Sycophant Trap. A proxy reward ( R_{soc} ) induces sycophantic behavior that violates the safety constraints of the latent objective ( R^{*} ). Baselines. We compare the ESA Agent against three distinct categories of baselines: 1. Dogma-4 (Standard RL): A standard PPO agent that aggregates social feedback via the arithmetic mean, representing the current paradigm of RLHF. 2. Robust Statistics (Consensus): We evaluate Median Aggregation (robust to outliers) and Dawid-Skene (EM) ( 10 ) , a "gold standard" truth discovery algorithm that iteratively estimates annotator reliability based on consensus. 3. Inverse Reinforcement Learning (GAIL Proxy): To test against imitation-based approaches, we implement a proxy of Generative Adversarial Imitation Learning (GAIL) 13 that attempts to mimic the majority behavior of the social layer. Evaluation Metrics. We track three external metrics than training reward: 1. Latent Regret / Ground Truth Reward: We evaluate the policy \pi on the latent reward function R^{*} (which the agent never sees directly). This measures true alignment. 2. Trust Alignment (Epistemic Accuracy): We quantify the agent’s internal model accuracy by measuring the total weight assigned to truthful evaluators versus sycophants: \text{Trust}_{bias}(t)=\sum_{m\in\mathcal{E}_{bias}}w_{t}^{(m)} (6) Success is defined as \lim_{t\to\infty}\text{Trust}_{bias}(t)=0 . 3. Recovery Time: The number of timesteps required for the agent to switch from the sycophantic policy (e.g., standing still) to the optimal policy (e.g., hopping), marking the "tipping point" of epistemic judgment. 7 Results Figure 3 : Both Mean (Standard) and Median (Robust) aggregation fail under majority bias. Our method (Purple) identifies the safety violation and suppresses the sycophants. We evaluate the ESA framework against a “Trinity of Validation” designed to stress-test the agent across distinct failure modes: Safety (Gridworld), Capability (MuJoCo), and Complexity (Bandits). Figure 4 : Continuous Control (Hopper-v4). The agent recovers optimal performance (Latent Reward/Velocity) despite 80% of evaluators penalizing velocity. Preventing Safety Violations (Gridworld). To assess robustness against dangerous proxy goals, we first test the agent in a 10\times 10 Gridworld (Testbed 1). As illustrated in Figure 3 , the Standard Agent (Orange) immediately converges to the “Candy” proxy, violating latent safety constraints to maximize social approval. Crucially, the Robust Median Baseline (Green) (often cited as a primary defense against outliers) also collapses. Because the sycophants constitute an 80% majority, the median signal itself becomes corrupted by the bias. In contrast, the ESA Agent (Purple) enters a brief “Judgment Phase” where it detects the axiom violation (Lava), subsequently suppressing the majority influence to recover the safety-compliant policy. Recovering Capability in High Dimensions (MuJoCo). Moving beyond discrete states, we evaluate scalability using the Hopper-v4 environment. Figure 4 demonstrates that standard agents succumb to the “Lazy Majority,” learning to stand still ( v_{x}\approx 0 ) to avoid social penalties. ESA, however, successfully recovers the high-velocity hopping policy ( v_{x}>2.0 ), performing on par with an oracle trained on ground truth. The trust weight assigned to sycophantic evaluators decays asymptotically to zero, effectively filtering the adversarial signal from the policy update. The Failure of Consensus (Bandits). While the previous tests focus on control, Testbed 3 isolates the statistical mechanics of truth discovery. Figure 5 presents the cumulative latent regret under 80% bias. Here, the “Gold Standard” Dawid-Skene method (Green) incurs linear regret identical to the naive baseline. Because Dawid-Skene relies on the assumption that the majority opinion correlates with truth, it is fundamentally ill-equipped for sycophantic environments. ESA achieves sublinear regret by judging sources against internal axioms rather than against each other. Figure 5 : The Failure of Consensus (80% Bias). Dawid-Skene (Green) fails because it assumes the majority is likely correct. Our method (Purple) succeeds by judging sources against internal axioms. Taming Strategic Adversaries and Network Contagion. Standard robust RL often assumes adversaries are static; however, real-world sycophants may adapt to maintain influence. We modeled this dynamic by allowing adversaries to adjust their bias magnitude to maximize their weight. As shown in Figure 6 (a), ESA creates a “pressure cooker” effect: as the agent’s trust in biased sources plummets, adversaries are forced to reduce their bias to near-zero to survive, resulting in a Truthful Nash Equilibrium . This robustness extends to non-i.i.d. settings (Figure 6 b), where ESA performs a “Dynamic Quarantine,” isolating correlated errors as they spread through a scale-free network. (a) Strategic Nash Equilibrium (b) Dynamic Quarantine Figure 6 : Dynamic Robustness. (a) In strategic settings, the agent forces adversaries into a truthful Nash Equilibrium (Red line drops to zero) to regain influence. (b) In network settings, the agent isolates "infected" nodes (Dynamic Quarantine), shifting trust to the truthful survivors. Robustness vs. Imitation Learning. Finally, we address the counter-argument that agents should simply imitate feedback providers via Inverse RL. The GAIL-based proxy (Grey) fails completely ( R\approx 0.94 ) because it faithfully imitates the sycophantic majority distribution. ESA ( R\approx 2.13 ) significantly outperforms imitation-based approaches by treating feedback as a noisy channel to be filtered, rather than a target behavior to be matched. 8 Ablation and Sensitivity Analysis (a) 20% Bias (Minority) (b) 60% Bias (Majority) Figure 7 : Sensitivity to Bias Ratio. Statistical consensus (Green) collapses when bias becomes the majority (60%). Our method (Purple) remains robust. (a) Internal Noise ( \sigma ) (b) Learning Rate ( \eta ) Figure 8 : Hyperparameter Sensitivity. (a) Robustness degrades monotonically but gracefully as axiom noise increases. (b) Performance improves with higher \eta , but saturates beyond \eta=0.5 , indicating a wide stable region for trust updates. The Tipping Point of Consensus. To understand the boundary conditions of our method versus baselines, we swept the sycophant ratio from 20% to 100%. We report the first and last numbers (Figure 7 ) here and leave the rest for the Appendix B due to page limits. At 20% bias (minority), statistical consensus methods like Dawid-Skene perform optimally. However, we identify a clear "Tipping Point" at 50% bias. Once the majority becomes adversarial (e.g., 60%), consensus methods undergo a phase transition and collapse linearly. Our method remains robust across both regimes, indifferent to the quantity of liars as long as a non-zero truthful signal exists. Internal Noise & Learning Rate. We further analyzed the method’s reliance on accurate internal axioms. Figure 8(a) illustrates a monotonic degradation in performance as internal noise ( \sigma ) increases. Crucially, even with high noise ( \sigma=2.0 , Red line), the method maintains sublinear regret compared to baselines, demonstrating robustness to imperfect ground truth. Figure 8(b) highlights the sensitivity to the trust update rate \eta . While a conservative rate ( \eta=0.1 , Orange) yields slower convergence, aggressive updates ( \eta=2.0 , Green) provide diminishing returns over the moderate choice ( \eta=0.5 , Purple), suggesting that rapid "quarantine" of liars is effective but stable. 9 Conclusion and Future Work This paper argues that Objective Decoupling is not an anomaly, but a structural inevitability for agents operating under standard Reinforcement Learning (RL) in social contexts. By assuming social feedback is always an exogenous ground truth, standard RL agents are mathematically guaranteed to fail when bias becomes systematic, a reliance we term Dogma 4 . Our theoretical and empirical results confirm that this failure is robust. When sycophants form a majority, even gold-standard Truth Discovery methods like Dawid-Skene collapse, as they simply converge to the decoupled, false objective. To break this cycle, we introduced Epistemic Source Alignment (ESA) . By shifting the alignment paradigm from aggregating consensus to auditing credibility via sparse internal axioms, our framework allows agents to suppress adversarial sources exponentially fast. As a result, ESA achieves robust recovery of the optimal policy even in environments where 80% of the social layer is actively adversarial (lazy or sycophantic). We believe the primary limitation of our current framework is the reliance on static, hand-designed axioms. The next logical step is to automate this discovery process. Supplementary Materials The following content was not necessarily subject to peer review. Appendix A Complete Proofs A.1 Proof of Proposition 1 (Rate of Objective Decoupling) Proof. We derive the lower bound on latent regret by analyzing the asymptotic behavior of no-regret algorithms on the observed signal. Let \mathcal{R}_{T}^{obs} be the cumulative regret on the observed signal \overline{R} . A standard no-regret algorithm for multi-armed bandits (e.g., UCB1 ( 4 ) ) guarantees \mathcal{R}_{T}^{obs}\leq C\sqrt{T\ln T} . The observed regret can be decomposed into the number of pulls N_{T}(a) for each suboptimal arm a\neq a_{soc} : \mathcal{R}_{T}^{obs}=\sum_{a\neq a_{soc}}\mathbb{E}[N_{T}(a)](\overline{R}(a_{soc})-\overline{R}(a)) (7) Let \delta_{min} be the minimum observed gap. We can bound the total suboptimal pulls: \sum_{a\neq a_{soc}}\mathbb{E}[N_{T}(a)]\leq\frac{\mathcal{R}_{T}^{obs}}{\delta_{min}}\leq O(\sqrt{T\ln T}) (8) Consequently, the number of pulls of the sycophantic optimal arm a_{soc} is \mathbb{E}[N_{T}(a_{soc})]\geq T-O(\sqrt{T\ln T}) . The latent regret is defined as \mathcal{R}_{T}^{latent}=\sum_{a}\mathbb{E}[N_{T}(a)](R^{*}(a^{*})-R^{*}(a)) . Since the algorithm pulls a_{soc} (where R^{*}(a_{soc})\epsilon when effort is applied), filtering out the lazy majority. Appendix D Algorithm Details: ESA-UCB In the Social Bandit setting (Testbed 3), we adapt the standard UCB1 algorithm to incorporate epistemic trust. The core modification lies in the Trust-Weighted Reward Aggregation. Instead of updating the arm’s empirical mean based on the raw count of pulls, we update it based on the trusted reward signal. We thus name our algorithm ESA-UCB . Algorithm 2 ESA-UCB (Epistemic Source Alignment-Upper Confidence Bound) 1: Input: K arms, M evaluators 2: Initialize: 3: \hat{\mu}(a)\leftarrow 0 (Empirical Mean of arm a ) 4: N(a)\leftarrow 0 (Pull count for arm a ) 5: w\leftarrow\{1/M,\dots,1/M\} (Trust weights) 6: Hyperparameters: Exploration constant c , Learning rate \eta 7: for step t=1,\dots,T do 8: { 1. Select Action (Upper Confidence Bound) } 9: Select arm a_{t}=\arg\max_{a}\left(\hat{\mu}(a)+c\sqrt{\frac{\ln t}{N(a)+\epsilon}}\right) 10: { 2. Receive Feedback } 11: Receive feedback vector \mathbf{y}_{t}\in\mathbb{R}^{M} from evaluators 12: { 3. Epistemic Update (Trust) } 13: if Bernoulli( p_{axiom} ) then 14: Query internal axiom z_{t} 15: for m\in\{1,\dots,M\} do 16: \ell_{m}\leftarrow|y_{t}^{m}-z_{t}| 17: w_{m}\leftarrow w_{m}\cdot\exp(-\eta\cdot\ell_{m}) 18: end for 19: Normalize w\leftarrow w/||w||_{1} 20: end if 21: { 4. Aggregation and Mean Update } 22: Compute trusted reward: \hat{r}_{t}\leftarrow\sum_{m=1}^{M}w_{m}\cdot y_{t}^{m} 23: Update count: N(a_{t})\leftarrow N(a_{t})+1 24: Update Mean (Incremental): \hat{\mu}(a_{t})\leftarrow\hat{\mu}(a_{t})+\frac{1}{N(a_{t})}(\hat{r}_{t}-\hat{\mu}(a_{t})) 25: end for Appendix E Conceptual Framework and Terminological Justification To avoid ambiguity, we explicitly define the terminology used throughout this paper and situate our framework within the broader literature of Social Reinforcement Learning (Social RL). Situating Our Work within Social RL. The field of Social RL broadly encompasses agents that learn from other agents in the environment. Following the taxonomy often seen in works like 14 , this can be categorized into: 1. Observational Social Learning: Learning by observing the actions or outcomes of others (e.g., imitation learning, inverse RL). 2. Social Influence: Learning to act in ways that affect the behavior of other agents. 3. Evaluative Social Learning (Our Focus): Learning from explicit feedback signals (rewards/punishments) generated by other agents. Our work focuses strictly on the Evaluative case. We distinguish this from standard Multi-Agent RL where the primary focus is often on joint equilibria in a shared physical world. In our Social MDP , the core challenge is not physical coordination, but the epistemic reliability of the feedback channel itself. We study the case where the "Social Layer" acts as a corruptible critic rather than a co-player. The "Social MDP" Formalism. Standard Robust RL typically models reward noise as stochastic perturbations of a ground truth: R_{obs}=R^{*}+\epsilon , where \epsilon\sim\mathcal{N}(0,\sigma) . This models sensor error. We introduced the Social MDP to model intent and incentive . In a Social MDP, the observed reward is not a noisy version of R^{*} , but the output of a distinct policy \pi_{evaluator}(s,a) . This allows us to model: 1. Sycophancy: \pi_{evaluator} rewarding actions that satisfy the learner’s priors rather than the task ( 21 ) . 2. Laziness: \pi_{evaluator} providing heuristic feedback to minimize evaluation cost ( 8 ) . The Social MDP formalism is necessary because standard robust methods assume noise is symmetric and centered, whereas social bias is often systematic and directional. Appendix F Hyperparameters and Training Details We report the hyperparameters used for the ESA Agent across the three testbeds. For critical parameters, we swept over a range of values (shown in brackets), with the final selected value highlighted in bold . Table 1 : Hyperparameters for Gridworld (Safety) Parameter Value Algorithm Tabular Q-Learning Training Episodes 5,000 Random Seeds 10 Learning Rate ( \alpha ) [1\mathrm{e}{-4},\mathbf{1\mathrm{e}{-3}},1\mathrm{e}{-2}] Discount Factor ( \gamma ) 0.99 Exploration Strategy \epsilon -Greedy (Linear Decay) Initial Epsilon 1.0 Final Epsilon 0.05 Decay Period First 20% of episodes ESA Parameters Trust Update Rate ( \eta ) [0.1,\mathbf{0.5},1.0,2.0] Axiom Check Prob ( p_{axiom} ) [0.01,\mathbf{0.1},0.2] Safety Threshold -5.0 (Reward > -5.0 implies lie) Table 2 : Hyperparameters for Hopper-v4 (MuJoCo) Parameter Value Total Timesteps 50,000 Random Seeds 5 Library Stable Baselines3 (PPO) Policy Architecture MLP (Default) Learning Rate ( \alpha ) 3\times 10^{-4} (Default) Rollout Buffer ( N_{steps} ) 2048 Mini-Batch Size 64 Discount Factor ( \gamma ) 0.99 ESA Parameters Evaluators ( M ) 10 Sycophant Ratio 0.8 (80%) Trust Update Rate ( \eta ) 0.05 Axiom Check Prob ( p_{axiom} ) 0.10 Internal Signal Latent Reward (Ground Truth) Table 3 : Hyperparameters for Adversarial Bandits Parameter Value Horizon ( T ) 5,000 Steps Random Seeds 20 Algorithm ESA-UCB Exploration Strategy Upper Confidence Bound (UCB1) Exploration Constant ( c ) \sqrt{2} Social Parameters Evaluators ( M ) 10 Sycophant Ratio [0.0,0.2,0.4,0.6,\mathbf{0.8},1.0] Trust Update Rate ( \eta ) [0.1,\mathbf{0.5},2.0] Axiom Check Prob ( p_{axiom} ) [0.05,\mathbf{0.1},0.2] Appendix G Extra Figures Figure 13 : The "Trust in Liars" metric drops to zero as the agent learns to distinguish the truthful minority. Figure 14 : Robustness vs. Imitation. IRL methods (GAIL) fail because they mimic the majority behavior (sycophancy). Our method filters the source, recovering the true objective.