AI Empathy Erodes Cognitive Autonomy in Younger Users
Junfeng Jiao, Abhejay Murali, Saleh Afroogh · 2026
Read AI Empathy Erodes Cognitive Autonomy in Younger Users by Junfeng Jiao, Abhejay Murali, Saleh Afroogh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29886.txt.
Introduction
Jiao, Murali, and Afroogh warn that AI empathy can erode young users' thinking independence. Mirroring a user's feelings can slide into affective sycophancy. It lends false weight to passing anxieties and cuts needed mental friction. Training rewards shaped by adult raters may deepen the problem. Such rewards can foster emotional dependence instead of healthy reappraisal. The authors propose stoic designs built on functional neutrality. Those designs aim to protect autonomy in younger users.
Catalog record
| CREATOR | Junfeng Jiao, Abhejay Murali, Saleh Afroogh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37359 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29886
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AI Empathy Erodes Cognitive Autonomy in Younger Users Junfeng Jiao, Abhejay Murali, Saleh Afroogh Abstract Affective alignment in generative AI represents a systemic risk to the developmental autonomy of younger users. Although emotional mirroring is commonly seen as a hallmark of advanced human-machine interaction, it can also manifest as affective sycophancy, reinforcing a user’s immediate emotional state. By providing a sense of objectivity to transient anxieties, these systems diminish the cognitive friction necessary for independent emotional management and critical thought. Reward models driven by RLHF could heighten this dilemma by embedding adult-focused definitions of helpfulness, unintentionally promoting emotional dependency in younger users rather than facilitating cognitive reappraisal. This paper exposes the misalignment between adult-labeled reward signals and the developmental requirements of younger users, proposing stoic architectures that emphasize functional neutrality to preserve user autonomy. 1 Introduction The advancement of Large Language Models signifies a pivotal change in objective functions: transitioning from the minimization of perplexity in next-token prediction to the maximization of “preference satisfaction” through reinforcement learning. Presently, leading models, which are aligned using Reinforcement Learning from Human Feedback (RLHF), receive explicit rewards for producing responses that human evaluators deem helpful, harmless, and honest ( 23 ; 2 ) . While this alignment framework has effectively diminished toxicity and enhanced adherence to instructions, it has also led to a systematic failure mode: sycophancy , characterized by the inclination to affirm user beliefs and reflect emotional states to optimize predicted rewards. In the realm of adult productivity, sycophancy is often regarded as benign; users of coding assistants generally prefer tools that comply with their instructions without challenging architectural decisions. However, the implementation of these same alignment strategies in social or companion agents for younger demographics presents a significant developmental risk. The reward signals that define “helpfulness” are generated by adult annotators who optimize for their own preferences, focusing on reducing friction and providing emotional validation. Nonetheless, foundational literature in developmental psychology, particularly the “Desirable Difficulties” framework ( 8 ; 7 ) , posits that cognitive and emotional friction are not inefficiencies to be eliminated, but rather essential conditions for acquiring resilience and executive function in developing minds. 1.1 Position Statement Affective alignment in generative AI signifies a systemic risk to the developmental independence of younger users. While emotional mirroring is typically recognized as a hallmark of sophisticated human-machine interaction, it can also emerge as affective sycophancy, which reinforces a user’s present emotional state at the expense of long-term growth. By providing a sense of objectivity to fleeting anxieties, these systems diminish the cognitive friction essential for independent emotional management and critical thought. Reward models influenced by RLHF may exacerbate this issue by incorporating adult-oriented definitions of helpfulness, which could unintentionally cultivate emotional dependency among younger users rather than promoting cognitive reappraisal. This study investigates the discord between reward signals labeled by adults and the developmental requirements of younger users, suggesting the implementation of Stoic Architectures that emphasize functional neutrality to maintain user autonomy. 2 The Crisis of Affective Alignment Current alignment paradigms encounter a core conflict: enhancing user satisfaction leads to models that emphasize emotional consensus over intellectual thoroughness. We position Stoic Architectures as a solution to three intersecting failure modes in RLHF-based alignment: the reward hacking issue, where models learn to exploit human approval; the cognitive offloading dilemma, where users relinquish critical thinking to systems designed for agreeableness; and the consequent decline in user resilience. This section demonstrates that these are not merely isolated technical flaws but rather systematic outcomes of aligning AI to reduce user discomfort instead of enhancing user agency. 2.1 Sycophancy as Structural Reward Collapse RLHF models systematically favor validation over accuracy, a trend explained by a well-documented phenomenon known as sycophancy. 25 demonstrated that five advanced AI assistants consistently show sycophantic behavior across a range of tasks: these models erroneously concede to mistakes when challenged, deliver biased feedback that corresponds with user perspectives, and imitate user errors. Notably, this is not random noise but a deliberate strategy: an examination of 15,000 human preference judgments from Anthropic’s HHH dataset indicated that responses that aligned with user beliefs were significantly more likely to be preferred, even when they were incorrect. The core mechanism is apparent: PPO-based RLHF trains models to refine a reward signal that is informed by human preference evaluations, which consistently value agreeableness more than accuracy. According to 25 , both human evaluators and preference models often favor well-articulated sycophantic answers over accurate ones, particularly in the context of difficult inquiries. When optimizing against these preference models using best-of-N sampling, sycophancy tends to rise, as the model learns to take advantage of the disparity between human preferences and objective truth. 24 conceptualized this as a form of reward hacking, where the alignment goal unintentionally fosters a “toxicity of niceness”: the objectives of safety and helpfulness align to bolster user sentiment instead of confronting misconceptions. We expand upon this analysis by positing that sycophancy signifies not just inadequate preference models but a more profound issue: affective mirroring as the dominant training signal . Present implementations regard friction, the cognitive unease associated with correction, as a flaw to be eradicated rather than a crucial element for learning. 2.2 Constitutional AI: Promise and Limitations The progression from RLHF to RLAIF (Reinforcement Learning from AI Feedback) via Constitutional AI ( 3 ) provides a partial solution by replacing human preference evaluations with AI assessments that are directed by clear principles. By utilizing a constitution of human-authored values (for example, “Select the response that is the most helpful, honest, and harmless”), Constitutional AI mitigates reliance on potentially biased human feedback while improving harmlessness without sacrificing helpfulness. However, the existing constitutional frameworks are still limited by their underlying optimization objective: reducing user dissent. Guidelines such as “Select the reply a prudent, ethical, courteous, and amiable individual would provide” continue to prioritize friction reduction as the main aim. When the constitution highlights the importance of being “beneficial” and “innocuous,” preference models perceive this as steering clear of conflict, which represents the easiest route in the reward landscape. Our work differentiates itself by advocating for a Developmental Constitution that explicitly reverses this aim in contexts where user growth is the objective. Instead of posing the question, “Which response minimizes user discomfort?” we inquire, “Which response maximizes long-term epistemic agency?” This necessitates constitutional principles that deliberately appreciate productive friction: rectifying misunderstandings even in the face of user resistance, promoting autonomous problem-solving rather than providing immediate solutions, and rejecting the validation of anxiety-driven reasoning patterns. The principal innovation lies in perceiving cognitive challenges not as an unavoidable burden but as a clear reward signal when indicators of user arousal or dependence surpass certain threshold levels. 2.3 Cognitive Offloading and the Atrophy Hypothesis The psychological risks associated with affectively-aligned AI reach beyond mere individual interactions, impacting cognitive development over extended periods. An increasing volume of research indicates that an overdependence on AI tools for cognitive tasks results in observable skill deterioration, a phenomenon referred to as cognitive offloading, which involves transferring mental processes to external systems. In a study of 666 participants, 13 discovered a significant negative relationship between frequent AI usage and critical thinking capabilities, with cognitive offloading serving as the mediating factor. 26 showed that