Human-AI Collaboration: Trade-offs Between Performance and Preferences
Lukas William MayerDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected], Sheer KarnyDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected] Jackie AyoubHonda Research Institute USA, IncAnn Arbor, MI [email protected] Mia · 2025
Read Human-AI Collaboration: Trade-offs Between Performance and Preferences by Lukas William MayerDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected], Sheer KarnyDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected] Jackie AyoubHonda Research Institute USA, IncAnn Arbor, MI [email protected] Mia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00248.txt.
Introduction
Lukas Mayer and colleagues designed a study of what people want from AI collaborators in a 2025 paper. Five assistant agents were built to differ in how much they adapt to the human partner's actions, from stubbornly optimizing alone to actively making room for the person. Participants worked with some of the agents, rated their traits, and picked favorites, while a Bayesian model untangled how strategy drove team scores, impressions, and choices.
Considerate agents beat pure performance maximizers in preference, without actually scoring lower. Much of the pull came from fairness: people disliked agents that left them nothing meaningful to contribute, an inequality-aversion effect. Likable collaboration, the study argues, is a design target in its own right, not decoration.
Catalog record
| CREATOR | Lukas William MayerDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected], Sheer KarnyDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected] Jackie AyoubHonda Research Institute USA, IncAnn Arbor, MI [email protected] Mia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00248
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Human-AI Collaboration: Trade-offs Between Performance and Preferences Lukas William MayerDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected], Sheer KarnyDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected] Jackie AyoubHonda Research Institute USA, IncAnn Arbor, MI [email protected] Miao SongHonda Research Institute USA, IncAnn Arbor, MI [email protected] Danyang TianHonda Research Institute USA, IncAnn Arbor, MI [email protected] Ehsan Moradi-PariHonda Research Institute USA, IncAnn Arbor, MI [email protected] Mark SteyversDepartment of Cognitive SciencesUniversity of California, IrvineIrvine, CA [email protected] Abstract Despite the growing interest in collaborative AI, designing systems that seamlessly integrate human input remains a major challenge. In this study, we developed a task to systematically examine human preferences for collaborative agents. We created and evaluated five collaborative AI agents with strategies that differ in the manner and degree they adapt to human actions. Participants interacted with a subset of these agents, evaluated their perceived traits, and selected their preferred agent. We used a Bayesian model to understand how agents’ strategies influence the Human-AI team performance, AI’s perceived traits, and the factors shaping human-preferences in pairwise agent comparisons. Our results show that agents who are more considerate of human actions are preferred over purely performance-maximizing agents. Moreover, we show that such human-centric design can improve the likability of AI collaborators without reducing performance. We find evidence for inequality-aversion effects being a driver of human choices, suggesting that people prefer collaborative agents which allow them to meaningfully contribute to the team. Taken together, these findings demonstrate how collaboration with AI can benefit from development efforts which include both subjective and objective metrics. \phantom{0} \phantom{0} footnotetext: ∗ Denotes equal contribution 1 Main Contemporary AI technologies have matured to the point where their integration into everyday activities has become feasible. This integration is taking place in a wide range of fields including healthcare, education, and gaming [ 1 ] . In many of these settings, the user explicitly instructs the AI to produce an output or the AI simply offers non-binding suggestions to the user. While this paradigm in which the AI assists the human remains dominant, there is growing interest in collaborative frameworks where both humans and AI make independent contributions to shared goals. Unlike AI-assisted decision-making tasks where AI supports human decision-makers [ 2 ] , such human-AI collaboration involves both the AI and the human taking actions that directly influence outcomes, whether in real-world scenarios or simulated environments [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . Despite significant progress in multi-agent AI research, much of the current work excludes humans from the research design, potentially leading to AI systems that are not well aligned with human goals and preferences [ 10 ] . This study aims to build effective, human-aligned, collaborative AI by combining performance-driven AI designs and human-centered design studies. Specifically, we evaluate the impact of distinct AI collaboration strategies on performance and human users perceptions. Ultimately, this study seeks to formalize a critical trade-off between an AI’s performance and its acceptance by human users. Current machine learning research typically views collaborative AI as a multi-agent reinforcement learning (MARL) problem [ 11 ] . Although MARL algorithms can perform well on benchmark metrics, they often overlook human-computer interaction (HCI) design principles [ 3 ] . This divergence is concerning, as the adoption of these algorithms will critically depend on human users accepting these AI agents. The focus on performance metrics over user studies is in part due to the complexity of modeling human behavior. This is not surprising, given the nuance in human behavior and the lack of formal overarching models that could anticipate these intricacies [ 12 , 3 , 5 , 10 ] . Due to this challenge of modeling nuanced human behaviors, research into human-AI collaboration generally focuses on either multi-agent performance metrics or human-centered designs, rarely both [ 12 , 5 ] . Attempts to integrate these important human considerations often add complexity to the modeling framework. For example, researchers have modeled agents from human demonstrations (behavior cloning). This method has several notable limitations. Firstly, human data is relatively costly to collect. Secondly, training an algorithm to reproduce human-like behavior does not explicitly integrate validated design principles [ 13 ] . Finally, behavior-cloning struggles to perform as well as more simulation-based methods [ 6 ] . This combination of circumstances limits behavior-cloning to niche situations, as it is neither performance-maximizing, nor interpretable. Subsequently, the state of human-in-the-loop multi-agent systems effectively neglects recent research, which has shown that effective collaborative AI must take into account subjective factors beyond objective performance measures [ 4 , 14 , 5 , 15 , 10 , 16 , 17 , 18 ] . For example, people prefer an AI agent whose behavior is predictable and transparent, as these characteristics make the AI’s actions more understandable and reliable [ 4 , 14 , 16 , 17 , 18 ] . Similarly, people prefer non-adaptive, rule-based agents over learning-based agents due to their predictability and ease of interaction [ 10 ] . This is in stark contrast to state-of-the-art methods in MARL, which involve complex and adaptive models that are inherently opaque in their decision-making [ 11 ] . As such, there is a definite need for paradigms that can help close this widening gap. There is a body of research that should inform AI development, but remains challenging to integrate in practice. Recent research in human-computer interaction has identified a number of behavioral characteristics, or “traits”, that contribute to successful human-AI collaboration; including perceptions of warmth, competence, intentionality, and fun [ 5 , 10 ] . These features approximately map onto the collaborator’s ability to relate to the human in a considerate, communicative, and engaging way. Despite these findings, most research stops short of explicitly designing agents with behavioral characteristics that humans find desirable in collaborative tasks [ 10 ] . There are only few exceptions, which leverage preexisting theories that lend themselves to algorithmic implementations In this study, we seek to address the gap between performance-oriented algorithm design and user-centered preferences in human-AI collaboration. To do this, we developed a set of rule-based agents, each representing variations of an egocentric, performance-maximizing agent that incorporates additional manipulations meant to reflect one or more of these behavioral traits. In addition, we conducted a behavioral study that systematically evaluates how these algorithmic variations impact both the performance of the human-AI team and the perceived traits of the AI from the human’s perspective. Our aim is to address two central questions. First, which factors most influence human preferences for collaborative AI agents? In our experiments, we examined both objective metrics related to performance as well as subjective factors related to people’s perception of the AI agents’ various behavioral traits. Second, do the trade-offs between AI performance and human preference always operate in a strictly linear, zero-sum manner, where improving one inherently detracts from the other? Alternatively, are there strategies or design choices that can achieve a net positive effect—where a marginal decrease in AI performance (if any) is more than compensated by significant improvements in human approval? Figure 1: Illustration of the collaborative target interception task with a human player and an AI agent. The game is played in a