Simulated Affection, Engineered Trust: How Anthropomorphic AI Benefits Surveillance Capitalism
Adele Olof-Ors, Martin Smit · 2025
Read Simulated Affection, Engineered Trust: How Anthropomorphic AI Benefits Surveillance Capitalism by Adele Olof-Ors, Martin Smit (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06472.txt.
Introduction
Adele Olof-Ors and Martin Smit argue that friendly, human-like artificial intelligence is a business tool rather than a neutral feature. Technologies that simulate emotional realism, from chatbots to virtual assistants, invite users to trust and open up. The authors connect that designed trust to surveillance capitalism, the system profiting from monitoring and steering behavior.
Drawing on Nicholas Carr's idea of an intellectual ethic, the paper claims such tools reshape how people think, not just what they think about. A new intellectual ethic carried by these systems serves the surveillance economy well. The authors close by discussing possible ways to push back against that arrangement.
Catalog record
| CREATOR | Adele Olof-Ors, Martin Smit |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06472
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Simulated Affection, Engineered Trust: How Anthropomorphic AI Benefits Surveillance Capitalism Adele Olof-Ors, Martin Smit Abstract In this paper, we argue that anthropomorphized technology, designed to simulate emotional realism, are not neutral tools but cognitive infrastructures that manipulate user trust and behaviour. This reinforces the logic of surveillance capitalism, an under-regulated economic system that profits from behavioural manipulation and monitoring. Drawing on Nicholas Carr’s theory of the intellectual ethic, we identify how technologies such as chatbots, virtual assistants, or generative models reshape not only what we think about ourselves and our world, but how we think at the cognitive level. We identify how the emerging intellectual ethic of AI benefits a system of surveillance capitalism, and discuss the potential ways of addressing this. Introduction Seventy years since its inception, the Turing Test has evolved from a philosophical thought experiment to an informal benchmark for success in artificial intelligence (AI). Contemporarily, the digital market is flooded with various large language models (LLMs) which pass this test, such as OpenAI’s GPT 4.5 which was mistaken for human in 73% of interactions, significantly surpassing the 30% threshold Turing himself proposed ( 24 ) . Although users typically understand that LLMs are algorithmic systems without sentience, they still tend to project ideas, states of beings and relationships onto these interfaces. The process of anthropomorphism, the tendency to ascribe human-like characteristics to non-human objects, changes their experience of the tool, causing many to form pseudo-social connections that change their relationships with and perceptions of the Internet, their society and themselves. Figure 1: Replika AI, a conversational AI which encourages users to form social and romantic relationships with chatbots, has been criticized for transforming the emotional isolation and need for connection of users into an opportunity for profit. By engaging with AI as opposed to human contact, the ability of users to regulate emotions can be diminished and their dependence on the AI increases. Technologists such as Nicholas Carr argue that adopting new tools reshape not only what we think - but how we think. Due to the brain’s ability to adapt to new conditions, repeated interaction with tools such as the Internet affects our cognitive abilities, in this case, impairing concentration, critical thinking, and multitasking. This change in skills produces a new method of thinking and seeing the world which Carr dubs the “Intellectual Ethic” ( 10 ) . In a digital economy of surveillance capitalism, digital interfaces are designed to engineer a mode of thinking which maximizes potential revenue by increasing engagement, watch time, and click-through rates. Although the effects of the Internet are well-documented, there is limited exploration of the mode of thinking produced through repeated interaction with AI. In this paper, we explore the consequences of anthropomorphic design on user’s worldview, and the intellectual ethic this produces. Tying this to surveillance capitalism, we identify how this shift in thinking and tendency to form pseudo-social relationships with AI creates more opportunities for data collection and complicates regulatory and behavioural resistance. Related Literature Several papers have analysed the commercial benefits of the anthopomorphisation of AI. A direct application of anthropomorphisation is found in 19 , which shows how interaction with a chatbot in an online store increases feelings of satisfaction, trust, and loyalty to a brand, directly influencing a shopper’s purchase intention. However, more often the AI itself is the product, and anthropomorphisation can serves as a way to acquire more users, or retain current users. In the former case, 31 discusses how emphasising human-like qualities of AI systems allows those marketing these systems to oversell their abilities. In the latter case, 22 argue that as users view conversational AI as a “master of knowledge” who is always available, patient, and non-judgemental, they come to it to fulfil their emotional needs by projecting onto it the role of a therapist or loved one. However, as the AI cannot fully satisfy their need for emotional companionship, leaving them in a cycle of dependence which benefits the corporation who commodify their needs. A systematic review of the different ways in which AI is designed to be anthropomorphic, as well as the risks this poses, is found in 1 . They note that anthropomorphic characteristic can be explicitly baked-in, as in the case of avatars and “typing” icons, but also a by-product of training on human conversations, containing emotive language, emoji usage, and other anthropomorphising communication features. They emphasise that a user’s suceptibility to harms is directly linked to the anthropomorphisation of AIs, and the lack of transparency regarding their status as an AI. One of the already realised harms identified by the authors is that of “manipulation and coercion”, where a user’s autonomy is undermined through consciously giving the AI control over one’s beliefs and actions. While 1 provides an excellent overview of the current and potential future harms of AI assistants and chatbots, our paper explores the ideas of “manipulation and coercion” in a distinct direction. By analysing the ways in which anthropomorphised AI not only changes user behaviour, but also their skills and world-view, we are able to demonstrate how anthropomorphisation directly benefits AI providers. This happens through more personalised, and more frequent data extraction, and the public’s decreased desire for regulation stemming from a more social, friendly, and sometimes dependent relationship with chatbots. After discussing how Surveillance Capitalism operates, we discuss how the changes in behaviour are designed to erode autonomy, demonstrated through neuroscience and psychology. Structure and Contribution Figure 2: This showcases how the intellectual ethic forms, and applies this to the already-identified intellectual ethic of the Internet, and the emerging intellectual ethic of AI. While this intellectual ethic is a natural process in the adoption of new tools, for example maps degrade out natural ability to navigate, this diagram highlights how Big Data companies co-opt this process for their own benefit. In the remainder of the paper, we introduce and explore the concepts of surveillance capitalism and the intellectual ethic in more detail. First, we explain how surveillance capitalism erodes personal autonomy through influencing behavioural modification in users, and why its lack of transparency makes it difficult to regulate. Then, we discuss how surveillance capitalism causes neurological changes in our brain, focusing on the well-studied case of how the Internet, particularly addictive web-based apps such as social media, degrades users’ abilities to focus and multitask. Novel to our paper is our discussion of the interplay between surveillance capitalism and the intellectual ethic. We continue this by giving early evidence for an emerging intellectual ethic of AI, detailing how chatbots disrupt the ways in which we address our needs for social connection. Whereas others have identified the tendencies that anthropomorphism creates, we build on this by giving four key reasons why this is directly beneficial to a system of surveillance capitalism administered by big data companies. We go on to argue how anthropomorphisation of AI is a recent addition to a longstanding strategy of big data companies attempting to avoid effective regulation through fighting transparency. Anthropomorphisation gives such companies a ability to shift our relationship with big data companies further away from transactional and more towards a social one, which complicates not only how to regulate such activities, but even the will of