Same Performance, Hidden Bias: Evaluating Hypothesis- and Recommendation-Driven AI CCS: Human-centered computing Laboratory experiments CCS: Human-centered computing Empirical studies in HCI
Michaela Benk, Tim Miller · 2026
Read Same Performance, Hidden Bias: Evaluating Hypothesis- and Recommendation-Driven AI CCS: Human-centered computing Laboratory experiments CCS: Human-centered computing Empirical studies in HCI by Michaela Benk, Tim Miller (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15824.txt.
Introduction
Benk and Miller show that equal scores can hide unequal decision habits. Support tools are usually judged by task scores or reliance rates. The pair compares recommendation-driven and hypothesis-driven designs in a web study. Some 290 people took part in the experiment. Signal Detection Theory frames the analysis of their choices. Recommendation designs lowered the evidence bar people set for action. Error patterns shifted even though total performance matched. Experts bent just as much as novices did. Process checks should join score checks, they urge.
Catalog record
| CREATOR | Michaela Benk, Tim Miller |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14624 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15824
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Same Performance, Hidden Bias: Evaluating Hypothesis- and Recommendation-Driven AI CCS: Human-centered computing Laboratory experiments CCS: Human-centered computing Empirical studies in HCI Michaela Benk, Tim Miller Abstract . The HCI community commonly evaluates decision support systems based on whether they improve task performance or promote appropriate user reliance. In this work, we look beyond decision outcomes to examine the process through which users develop decision-making strategies. Through a web-based experiment (N = 290) comparing recommendation-driven and hypothesis-driven interaction designs, and using Signal Detection Theory as a theoretical framework, we show that even when performance remains identical, recommendation-driven designs lower participants’ thresholds for sufficient evidence and introduce a “hidden bias” in their judgments, resulting in a shifted distribution of errors. Furthermore, we find that experts are just as susceptible to these systemic shifts as novices. We conclude by advocating for a shift in focus: prioritizing decision processes and the preservation of stable evidence standards over performance and reliance alone. 1. Introduction Consider two forensic analysts, Laura and Sergio, both using AI-based decision support systems to identify suspects in a large case involving thousands of digital files. Laura uses a recommendation-driven system: it suggests a match, highlights evidence, and provides a high confidence score. Laura reviews the system’s output and agrees with the recommendation; she trusts the output, as the highlighted segments seem convincing. Meanwhile, Sergio uses a hypothesis-driven system: it provides no recommendation, but offers tools to explore ambiguities and alternative interpretations of the data. After careful deliberation, he also identifies the same suspect. On the surface, the outcomes are identical. Both analysts reach the same conclusion. Both trust their respective system’s outputs. And both perform accurately on this case. However, the decision support paradigms of the two systems each foster qualitatively different decision-making strategies. The ‘hidden bias’ lies in how each system affects their cognitive process used to assess evidence, and the resulting error patterns. As noted by Miller 2023 , decision-makers in AI-assisted settings engage in two distinct but interrelated processes: (a) evaluating options and outcomes (“Is this a match?”) and (b) assessing the soundness of the DSS advice (“Should I trust this output?”). In hypothesis-driven settings, the analyst’s error patterns will primarily reflect their personal threshold used to evaluate evidence. In recommendation-driven settings, however, the evaluation of the recommendation conflates with the evaluation of the task: the AI’s “match” affects the threshold for what the analyst considers sufficient evidence and subsequently, the errors they make. While HCI evaluations frequently focus on performance (e.g., accuracy) and reliance (e.g., agreement with the AI’s output) ( Buçinca et al. 2021 ; Langer et al. 2024 ; Chen et al. 2023 ; Schemmer et al. 2022 ; Nourani et al. 2021 ; Vasconcelos et al. 2022 ) , they fail to reveal the hidden bias: whether and how the interaction design shifts the evidence threshold. This shift has profound implications for what Langer et al. 2024 term ’effective human oversight.’ They argue that humans must employ decision strategies that reliably detect errors to act as effective safeguards, a task that depends on how the users sets their threshold for accepting evidence before making a decision. By analyzing how design alters these thresholds, we can better understand whether an interface supports or undermines the human’s ability to balance errors. Signal Detection Theory as a Lens. Signal Detection Theory (SDT) provides a formal framework to decouple these processes by separating sensitivity ( d^{\prime} ), the ability to distinguish signal from noise, from the criterion ( c ), the evidence threshold that reflects the decision strategy ( Stanislaw and Todorov 1999 ) . In high-stakes domains, a “liberal” decision strategy (negative c values/lower threshold) increases the risk of false accusations, even if accuracy remains high. In our context, a liberal shift in c indicates that the analyst requires less evidence to confirm a suspect. Conversely, a “conservative” shift (positive values) might delay critical interventions. SDT has only recently been highlighted as an effective method to model human oversight in AI-assisted decision making ( Langer et al. 2024 ; Elder et al. 2024 ) . Contribution . Prior work has recognized the need to look at specific error patterns and highlighted how oversight effectiveness diminishes when users fail to distinguish AI errors from correct outputs ( Langer et al. 2024 ) . Our work extends this by demonstrating how specific interaction paradigms, namely recommendation-driven vs. hypothesis-driven designs, systematically alter evidence thresholds. Using SDT as a robust theoretical framework, we demonstrate that: • Hidden bias: While participants across conditions show similar overall accuracy, recommendation-driven designs shift users toward a lower threshold of evidence required. Conversely, hypothesis-driven approaches maintain similar thresholds as having no AI assistance. • Adaptation: We observe a divergence in how users adapt their strategies. Participants across conditions improve their performance over time, but only those in the recommendation-driven condition systematically adapt their evidence thresholds. • Expertise: Our findings reveal that domain expertise does not provide immunity against the ‘hidden bias’ that the interaction design can induce. We hope to foster a discussion in the research community about moving beyond traditional performance and reliance metrics and toward designing and evaluating AI systems in terms of the hidden bias they may elicit. 2. Method 2.1. Task and Materials To study decision strategies in a controlled, yet domain-general setting, we designed a decision-making task inspired by the forensic investigation vignette from Section 1 . The task preserves three key properties: (a) comparable difficulty for humans and AI, (b) binary decisions enabling decision-theoretic analysis, and (c) known ground truth. We adapted the OnlyConnectWall dataset ( Alavi Naeini et al. 2024 ) , which consists of word association puzzles containing misleading cues (“red herrings”). Participants were shown sets of 16 words and asked to judge whether a specific theme was present or not. Errors could occur as false positives (accepting an impostor theme) or false negatives (rejecting a true theme), allowing analysis of SDT metrics. AI assistance was simulated using pre-generated outputs from GPT-4, calibrated to achieve 70% accuracy, reflecting realistic but imperfect decision support behavior observed in prior work ( Bansal et al. 2021 ; Buçinca et al. 2021 ) . All stimuli and system outputs were fixed across conditions. 2.2. Study Design 2.2.1. Experimental Conditions Our overarching hypothesis posits that the interaction design influences people’s decision strategy. To test this hypothesis, participants were randomly assigned to one of five conditions: • Control : no AI assistance • Recommendation-driven (RD) : AI provides a binary recommendation. • Recommendation + explanation (RDX) : recommendation plus explanatory information and uncertainty cues. • Recommendation + hypothesis exploration (RDH) : recommendation with interactive exploration of alternative interpretations. • Hypothesis-driven (HD) : participants explored evidence and uncertainty without a recommendation. The conditions varied only in interaction design; task content was identical. 2.2.2. Participants and Procedure We recruited 320 UK-based participants from Prolific; after exclusions, 290 remained (mean age = 42, 49% female). Participants completed a training phase with feedback followed by a testing phase without feedback. In each trial, they judged whether a theme was present and identified supporting words if applicable. 2.2.3. Measures and Analysis Our main dependent variables were sensitivity ( d^{\prime} ) and decision criterion ( c ). There are diverse