“ Even explanations will not help in trusting [this] fundamentally biased system ”: A Predictive Policing Case-Study Conference: 33rd ACM Conference on User Modeling, Adaptation and Personalization; June 16–19, 2025; New York City, NY, USA 33rd ACM Conference on User Modeling, Adaptation and Persona
Siddharth Mehrotra, Ujwal Gadiraju, Eva Bittner, Folkert van Delden, Catholijn M. Jonker, Myrthe L. Tielman · 2025
Read “ Even explanations will not help in trusting [this] fundamentally biased system ”: A Predictive Policing Case-Study Conference: 33rd ACM Conference on User Modeling, Adaptation and Personalization; June 16–19, 2025; New York City, NY, USA 33rd ACM Conference on User Modeling, Adaptation and Persona by Siddharth Mehrotra, Ujwal Gadiraju, Eva Bittner, Folkert van Delden, Catholijn M. Jonker, Myrthe L. Tielman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11020.txt.
Introduction
Siddharth Mehrotra and five co-authors present a study from twenty twenty-five on trust in AI used for predictive policing. Such tools guess where crime may occur. Users can trust them too much or too little, and both skew acts. Past work said that giving reasons for a call can tune trust to the right level. The team tests text, visual, and mixed forms of reasons.
Retired police staff and lay users took part. Mixed reasons raised how much expert users said they trusted the tool. Yet no form led to better choices, and none built apt trust. The authors urge a rethink of reason giving in high risk tools. They close with policy hints for such systems.
Catalog record
| CREATOR | Siddharth Mehrotra, Ujwal Gadiraju, Eva Bittner, Folkert van Delden, Catholijn M. Jonker, Myrthe L. Tielman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-11020
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
“ Even explanations will not help in trusting [this] fundamentally biased system ”: A Predictive Policing Case-Study Conference: 33rd ACM Conference on User Modeling, Adaptation and Personalization; June 16–19, 2025; New York City, NY, USA 33rd ACM Conference on User Modeling, Adaptation and Personalization (UMAP ’25), June 16–19, 2025, New York City, NY, USA DOI: 10.1145/3699682.3728343 ISBN: 979-8-4007-1313-2/2025/06 CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Human computer interaction (HCI) Siddharth Mehrotra, Ujwal Gadiraju, Eva Bittner, Folkert van Delden, Catholijn M. Jonker, Myrthe L. Tielman Abstract . In today’s society, where Artificial Intelligence (AI) has gained a vital role, concerns regarding user’s trust have garnered significant attention. The use of AI systems in high-risk domains have often led users to either under-trust it, potentially causing inadequate reliance or over-trust it, resulting in over-compliance. Therefore, users must maintain an appropriate level of trust. Past research has indicated that explanations provided by AI systems can enhance user understanding of when to trust or not trust the system. However, the utility of presentation of different explanations forms still remains to be explored especially in high-risk domains. Therefore, this study explores the impact of different explanation types (text, visual, and hybrid) and user expertise (retired police officers and lay users) on establishing appropriate trust in AI-based predictive policing. While we observed that the hybrid form of explanations increased the subjective trust in AI for expert users, it did not led to better decision-making. Furthermore, no form of explanations helped build appropriate trust. The findings of our study emphasize the importance of re-evaluating the use of explanations to build [appropriate] trust in AI based systems especially when the system’s use is questionable. Finally, we synthesize potential challenges and policy recommendations based on our results to design for appropriate trust in high-risk based AI-based systems. 1. Introduction Artificial Intelligence (AI) is rapidly reshaping public organizations globally, mainly through machine learning approaches that automate routine administrative tasks and support decision-making ( Bullock 2019 ) . One of the key components to achieving effective decision-making is a user’s appropriate trust in AI systems. Despite recent efforts towards enhancing trust in algorithmic decision-making systems (e.g., adding explanations ( Dodge et al. 2019 ; Yang et al. 2020 ) , human oversight ( Tolmeijer et al. 2022 ; Ososky et al. 2013 ; Shively et al. 2018 ) , and confidence scores ( Zhang et al. 2020 ; Barbosa et al. 2022 ) ), comparatively little attention has been paid to building appropriate trust in them. Both under-trust and over-trust are deemed inappropriate ( Parasuraman and Riley 1997 ; Robinette et al. 2016 ; Lee and See 2004 ) , instead we require trust to be appropriate. Under-trust can lead to under-reliance, and over-trust can lead to over-compliance, which can negatively impact the task. Therefore, in this work we study whether we can improve the appropriateness of trust in an AI decision support system ( goal ). We propose to do this through explanations ( means ), and position our work in the context of AI-based predictive policing ( context ) as a high-risk domain. We choose AI-based predictive policing as our use case primarily because it represents a critical domain where appropriate trust is pivotal due to the high-stakes nature of decision-making. Additionally, the use of AI in predictive policing has been the subject of extensive debate for years, with numerous studies highlighting biases inherent in data collection practices and their impact on AI-driven decisions ( Marda and Narayan 2020 ; Meijer and Wessels 2019 ; Meijer et al. 2021 ; Ferguson 2018 ; Alikhademi et al. 2022 ; Leese 2024 ) . Despite these concerns, predictive policing systems continue to be used globally ( Mugari and Obioha 2021 ) . Therefore, in this study we explore whether providing explanations can help users critically evaluate AI decision-making in this use-case, encouraging introspection and ultimately NOT promoting trust but fostering appropriate trust. We draw inspiration from the ASPECT model by Jameson et al. ( Jameson et al. 2014 ) , as a useful means to identify the different facets of our complex use case. The ASPECT acronym refers to the first letters of six patterns: Attributes, Social influence, Policies, Experience, Consequences, and Trial and error. Each pattern may be perceived as an "aspect" of design choices, denoting a particular perspective or approach to addressing a problem or research question. Aligned with the ASPECT model, our approach with this use-case is based on attributes (explanations), social (socially relevant topic), policy (policy level implications), experience (user expertise), consequence (effect of explanation), and trial & error (human-AI agreement) based choice patterns of the ASPECT model. While AI systems can provide explanations in multiple formats, selecting the optimal design remains challenging. These formats include visual explanations like saliency maps ( Adadi and Berrada 2018 ) , textual explanations using words and phrases, and analytical explanations that enable users to explore both data and model ( Kim et al. 2018 ; He et al. 2022 ) . Although each method has garnered significant attention independently, comparative studies examining their relative effectiveness across different contexts and user groups remain limited ( Szymanski et al. 2021 ; Robbemond et al. 2022 ; Park et al. 2018 ; Nunes and Jannach 2017 ) . Notably, there is a particular gap in understanding how these different presentation methods impact appropriate trust in high-risk domains. Our research addresses this gap by examining user interaction and perception across textual, visual, and hybrid explanation formats to determine their effectiveness in fostering appropriate trust. According to Ribera & Lapedriza ( Ribera and Lapedriza García 2019 ) , the goal of XAI in any context depends not only on the presentation of explanations but also on the type of end-user that is on the receiving end of the explanations. For example, previous research has shown that the hybrid form of explanations significantly improves correct understanding for lay users compared to visual explanations ( Szymanski et al. 2021 ) . However, little work has been done so far to compare the utility of different explanation methods in building appropriate trust between expert users and lay users ( Larasati et al. 2023a ) . In the predictive policing use-case, this comparison is relevant as some police officers might have less professional experience to draw on than others, for example, police officers who have recently joined the department. Therefore, in this study, we compared the utility of different types of explanations with both expert and lay users ( moderation factor ) linked with the experience pattern of the ASPECT model. To better understand the moderating factor of user expertise in our study, we performed an application-grounded evaluation followed by a human-grounded evaluation. Human-grounded evaluation is appealing when orchestrating experiments with the target user groups is challenging ( Lakkaraju et al. 2016 ; Kim et al. 2015 ) . This is in line with Doshi-Velez & Kim ( Doshi-Velez and Kim 2017 ) who articulate and argue for the value of different layers of evaluation within XAI focusing on functionality, grounding, and presentation. Our sample of police officers corresponds to application-grounded evaluation which includes experts and real tasks. On the other hand, our sample of lay users corresponds