When not to help: planning for lasting human-AI collaboration

Mark Steyvers, Lukas Mayer · 2025

Read When not to help: planning for lasting human-AI collaboration by Mark Steyvers, Lukas Mayer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01837.txt.

Introduction

Mark Steyvers and Lukas Mayer present a 2025 paper on when an AI assistant should hold back its help. Systems that chat with users in real time face a dilemma: help too often, or with tips the user does not need, and people tune out or switch the system off.

The authors build a model that tracks a user's hidden mental state, such as how engaged they are. The model weighs how well the user would do alone against the boost from help and the risk of lost engagement. This look-ahead reasoning is the core of their framework. In simulations, the adaptive approach beat both always-help and never-help rules. The results favor help that protects long-term trust, not just short-term accuracy.

Catalog record

CREATORMark Steyvers, Lukas Mayer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01837

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When not to help: planning for lasting human-AI collaboration Mark Steyvers, Lukas Mayer Abstract AI systems and technologies that can interact with humans in real time face a communication dilemma: when to offer assistance and how frequently. Overly frequent or contextually redundant assistance can cause users to disengage, undermining the long-term benefits of AI assistance. We introduce a cognitive modeling framework based on Partially Observable Markov Decision Processes (POMDPs) that addresses this timing challenge by inferring a user’s latent cognitive state related to AI engagement over time. Additionally, our framework incorporates reasoning about the long-term effects of AI assistance, explicitly aiming to avoid actions that could lead the human user to disengage or deactivate the AI. A key component of our approach is counterfactual reasoning: at each time step, the AI considers how well the user would perform independently and weighs the potential boost in performance against the risk of diminishing engagement with the AI. Through simulations, we show that this adaptive strategy significantly outperforms baseline policies in which assistance is always provided or never provided. Our results highlight the importance of balancing short-term decision accuracy with sustained user engagement, showing how communication strategies can be optimized to avoid “alert fatigue” while preserving the user’s receptiveness to AI guidance. Keywords: AI assistance, POMDPs, sequential decision-making, advice-taking, Bayesian modeling, engagement 1 Introduction Modern AI systems and technologies are increasingly integrated into our daily lives, offering guidance, alerts, and assistance in a wide range of contexts. However, their success hinges on timely communication—namely, knowing when to convey important information and when to remain silent. A common and often frustrating issue arises when these systems provide excessive or redundant advice, failing to account for the user’s familiarity with a situation. For example, consider a driver pulling into their home garage—a task they have completed successfully countless times—only to be inundated with warnings about potential collisions from the car’s assistance system. Such alerts, while well-intentioned, may feel unnecessary and intrusive, leading the user to disregard them entirely. Over time, this can result in “alert fatigue,” where the user becomes desensitized to critical warnings due to frequent exposure to non-essential ones Ancker et al. (2017) . Similarly, AI systems designed to assist in decision-making often fail to adapt to a user’s level of expertise or familiarity with a task. For instance, an AI advisor providing investment recommendations might over-explain basic financial principles to a seasoned investor, reducing the perceived value of its assistance. Over time, the user may disengage, ignoring even high-quality advice in unfamiliar or complex scenarios where the AI’s guidance could have been beneficial. These examples highlight the dual challenge faced by AI systems: providing timely and useful assistance while avoiding over-advising that can diminish user engagement. As emphasized in general guidelines for human-AI interaction Amershi et al. (2019) , achieving this balance requires careful consideration of the timing and frequency of AI advice, ensuring that assistance is delivered only when it is contextually relevant and beneficial. Addressing this challenge requires an understanding of how users interact with AI systems and how their engagement evolves over time. Foundational work in AI assistance, such as the seminal frameworks proposed by Horvitz (1999) ; Horvitz et al. (2003) , introduced utility-based approaches to balance the benefits of AI intervention against its potential costs. These frameworks involve formulating the decision to provide assistance as an optimization problem, where the AI maximizes an expected utility function that accounts for factors such as task outcomes and the cost of AI intervention. While these approaches have been instrumental in advancing decision-theoretic AI assistance, they require an explicit formulation of the penalty of over-advising. In contrast, the computational framework proposed in this paper shifts the focus to a reasoning problem about the human’s cognitive state—specifically, their engagement and adherence to AI advice. This framework accounts for how adherence evolves over time as a function of the perceived usefulness of past advice. By incorporating a cognitive model, the AI can predict the likelihood of the human making the correct decision both with and without AI assistance. In this framework, the cost of over-advising arises naturally as the consequence of disengagement, leading to poor decisions and missed opportunities where AI assistance could be beneficial. This approach builds on insights from recent studies in human-AI interaction Chen et al. (2024) ; Noti & Chen (2023) ; Buçinca et al. (2024) , which emphasize the importance of understanding and adapting to user behavior but extends them by explicitly incorporating engagement dynamics as a key variable in the decision-making process. In this paper, we propose a cognitive modeling framework based on Partially Observable Markov Decision Processes (POMDPs) that addresses these challenges by adapting assistance to a user’s inferred cognitive state while explicitly reasoning about the long-term impact of its interventions to avoid user disengagement or system deactivation. We detail the theoretical underpinnings, provide simulation results, and discuss the potential for applying these methods. However, we note that we do not yet incorporate results from behavioral studies to test the assumptions of the cognitive modeling framework, leaving that extension for future research. 2 Cognitive Modeling Framework We present a computational framework that addresses the trade-offs in providing AI assistance by dynamically adapting to the user’s engagement state. Unlike traditional utility-based approaches, which optimize static reward functions, our framework integrates a cognitive model of engagement dynamics, enabling the AI to balance immediate decision accuracy with the long-term goal of maintaining user engagement. The core idea is to model the interaction between the AI and the human as a sequential decision-making problem. The AI must decide, at each time step, whether to provide assistance based on the human’s task context and the state of engagement, which determines the probability that the user adheres to the AI assistance. Our approach builds on the principles of Partially Observable Markov Decision Processes (POMDPs), which enable the AI to reason under uncertainty. The state space includes both observable variables, such as task context and human decisions, and latent variables, such as the human’s engagement state at each point in time. Through this framework, the AI learns to optimize its advising strategy, ensuring that assistance is provided when it is most needed while avoiding unnecessary intervention in familiar or simple contexts. The following subsections detail the key components of this framework, including the problem setup, the modeling of human decision-making with and without AI assistance, and the dynamics of engagement. Figure 1: Dynamic graphical model that shows the dependencies between states over time. The shaded symbols represent information observable to the AI assistant. The unshaded symbols represent latent cognitive states. The variable y_{t} is human decision with the AI assistance. The variable y^{\prime}_{t} represents a counterfactual decision had the AI assistance not occurred. 2.1 Problem Setup At each time step t , the human makes a binary decision y_{t}\in\{\text{correct},\text{incorrect}\} . The correctness of the decision is influenced by the task context c_{t} , which varies over time and determines the difficulty of making the correct choice. We simplify the contexts into two levels of familiarity: \{\text{low},\text{high}\} . The AI can either provide advice ( a_{t}=\text{on} ) or withhold it ( a_{t}=\text{off} ). The human’s decision-making behavior is modeled under two conditions: with and

Read the complete text