Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships

Kerem Oktar*, Katherine M. Collins*, Jose Hernandez-Orallo, Diane Coyle, Stephen Cave, Adrian Weller, Ilia Sucholutsky · 2025

Read Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships by Kerem Oktar*, Katherine M. Collins*, Jose Hernandez-Orallo, Diane Coyle, Stephen Cave, Adrian Weller, Ilia Sucholutsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16899.txt.

Introduction

Seven authors warn that AI systems are graduating from tools to thought partners, and that the promotion carries new risks. A thought partner joins in the reasoning itself: framing problems, brainstorming, and shaping judgment. The commentary maps those risks at three levels at once, the live exchange, the individual mind, and society, in a framework the authors call RISc.

At each level they propose concrete yardsticks so the risks can be measured rather than merely feared. They close with mitigations aimed at builders and at policymakers. The through-line is that partnership changes what can go wrong: a tool's errors stay in its output, while a partner's errors can enter the user's own thinking. The authors argue the field should name and gauge those harms before the systems spread further.

Catalog record

CREATORKerem Oktar*, Katherine M. Collins*, Jose Hernandez-Orallo, Diane Coyle, Stephen Cave, Adrian Weller, Ilia Sucholutsky
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying, Evaluating, and Mitigating Risks of AI Thought Partnerships Kerem Oktar*, Katherine M. Collins*, Jose Hernandez-Orallo, Diane Coyle, Stephen Cave, Adrian Weller, Ilia Sucholutsky Abstract Artificial Intelligence (AI) systems have historically been used as tools that execute narrowly defined tasks. Yet recent advances in AI have unlocked possibilities for a new class of models that genuinely collaborate with humans in complex reasoning, from conceptualizing problems to brainstorming solutions. Such AI thought partners enable novel forms of collaboration and extended cognition, yet they also pose major risks—including and beyond risks of typical AI tools and agents. In this Commentary, we systematically identify risks of AI thought partners through a novel framework that identifies risks at multiple levels of analysis, including Real-time, Individual, and Societal risks arising from collaborative cognition (RISc). We leverage this framework to propose concrete metrics for risk evaluation, and finally suggest specific mitigation strategies for developers and policymakers. As AI thought partners continue to proliferate, these strategies can help prevent major harms and ensure that humans actively benefit from productive thought partnerships. Artificial Intelligence (AI) systems have historically been used as tools that execute narrowly defined tasks, such as identifying faces in images. Yet recent advances in AI have unlocked possibilities for a new class of models that genuinely collaborate with humans in complex reasoning, from conceptualizing problems to brainstorming solutions; for instance, discussing the pros and cons of medical treatments with doctors, or planning military interventions with policymakers. Such AI thought partners [ 3 ] or AITPs, enable novel forms of collaboration and extended cognition, yet they also pose major risks—including and beyond risks of typical AI tools and agents [ 4 ] . Here, we propose a systematic framework for assessing risks of AITPs, offer metrics for the evaluation of these risks, and suggest potential strategies for risk mitigation. There is a pressing need to understand these risks as AITPs move rapidly from development to deployment. New ‘reasoning’ models like Deepseek’s R1 or OpenAI’s o3, and new systems explicitly designed to collaborate in complex reasoning, such as Google’s ‘co-scientist’ [ 6 ] increasingly enable impactful collaborative cognition—co-learning, co-creating, and co-planning—far beyond what is possible with other AI tools, including LLM agents [ 12 ] . The RISc Framework Our RISc framework categorizes possible risks from collaborative cognition with AITPs into three interacting levels: first, (R)eal-time risks arising from specific instances of human-AITP interactions; second, those arising from extended use by (I)ndividuals; and third, those arising when systems are deployed in (S)ociety in new kinds of collaborative (c)ognition. At each level, we separately consider risks inherent in an AITP’s performance (is the model performing appropriately?) and those that arise from how AITPs are utilized (is the model being used appropriately?), giving six risk categories (see Box 1). Box 1: Risk Categories Consider two impactful use cases for AITPs: medicine (a doctor uses an AITP to think through triage, diagnosis, and treatment planning) and policy (a policymaker uses an AITP to think through complex decisions during a financial crisis). We illustrate each risk category with one example from medicine and policy. Real-time Risks Performance (generic deliberation) Poorly-trained AITPs can encourage context-insensitive deliberation. Medicine example: A medical AITP may lead a doctor to consider globally prevalent but locally rare diseases, lowering efficiency of time-sensitive diagnosis. Utilization (credit assignment) Assigning praise and blame between users and AITPs is challenging when AITPs influence every stage of decisions. Policy example: Who is liable in a misdiagnosis if doctor and AITP deliberated together? Individual Risks Performance (user manipulation) An AITP can fail to benefit its user if it’s programmed to manipulate them. Policy example: An AITP may nudge a policymaker to author policies that benefit the AITP’s developer. Utilization (cognitive atrophy) Extensive use of AITPs can blunt critical reasoning skills. Medicine example: A doctor who always relies on an AITP may weaken their ability to form diagnostic hypotheses. Societal Risks Performance (systemic fragility) Centralization of AITP systems may induce fragility to blackouts or attacks. Policy example: Heavy reliance on AITPs in central banks could make monetary policy vulnerable to outages. Utilization (homogeneity of thought) Widespread AITP use may lead to convergence of opinion. Policy example: If all officials use the same AITP model, policies may fail to reflect local needs and knowledge. Evaluating Risks To systematically assess the presence and extent of these risks in genuinely collaborative cognition in practice, we need to develop effective evaluations. Our framework both structures past work focused on assessing performance-level risks [ 1 ] and highlights new evaluations for utilization risks at multiple levels. Evaluating Real-time Risks Real-time risks depend on the thought interplay between users and AITPs and influence a community of stakeholders. For instance, consider an AITP helping a doctor decide which patients to admit to limited beds in a pandemic. A good AITP can help delineate and weigh trade-offs through an open dialogue; but how can we evaluate whether the tradeoffs proposed, and the dialogue process itself, are ‘good’? Such assessment requires considering the perspectives of multiple stakeholders—from the care team to the patients, their families, and the hospital and broader community writ large. The assessment of which tradeoffs are raised and how they are accounted for over the course of a conversation—over the thinking trace —could involve leveraging NLP methods or large language models to process the dialogue. Practitioners could specify what kinds of tradeoffs would be expected to be raised, which could be checked in the thinking trace; however, manual oversight of some form may be required—especially for evaluations that are not reducible to a single standardized “score,” such as the model’s moral judgment. These same measures of assessing the thinking trace itself could be used to evaluate the relative contributions of any human(s) and any AITP(s). For instance, if a medical researcher engages with an AITP in the construction of a clinical trial protocol, we could automatically tag a saved trace of the interactions to appraise both the amount and type of thinking that each thought partner contributed to the collective “product of thought.” Evaluating Individual Risks Such readouts could also be helpful in feeding back to the user, or supervisor of the user, to ensure that the user is contributing appropriately to the thought partnership and not over-relying on the model for ideas or insights. But what does it mean for users to over- or under-rely on a model when the engagements unfold over multiple interactions in this new kind of collaborative cognition? And how can we assess how user thinking may atrophy through over-reliance rather than developing through reasoned interaction? This may require regular intervals of assessment with a professional, with and without access to an AITP. Such assessments should draw inspiration from existing work evaluating team dynamics and other measures of team efficacy when one or more group members is removed for some stage in the collaborative reasoning process [ 11 ] . Additionally, AITPs may require evaluation that looks at the user’s first-person experience in the reasoning process, rather than just the final output of collaborative thinking. Understanding how people interact with AITP in new forms of collaborative cognition requires interactive evaluation that engages with multiple steps in the reasoning process [ 2 ] . This could

Read the complete text