“Over-the-Hood” AI Inclusivity Bugs and How 3 AI Product Teams Found and Fixed Them DOI: 99.9999/9999999.9999999 CCS: Human-centered computing User studies CCS: Computing methodologies Intelligent agents
Andrew Anderson, Fatima A. Moussaoui, Jimena Noa Guevara, Md Montaser Hamid, Margaret Burnett · 2025
Read “Over-the-Hood” AI Inclusivity Bugs and How 3 AI Product Teams Found and Fixed Them DOI: 99.9999/9999999.9999999 CCS: Human-centered computing User studies CCS: Computing methodologies Intelligent agents by Andrew Anderson, Fatima A. Moussaoui, Jimena Noa Guevara, Md Montaser Hamid, Margaret Burnett (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19033.txt.
Introduction
Andrew Anderson and four co-authors report a field study of over-the-hood inclusivity bugs in user-facing artificial intelligence products. Beyond hidden bias in data or algorithms, the visible design of a product can shut out users whose problem-solving style differs from what designers assumed.
Three product teams applied an existing inclusive design method, not built for artificial intelligence, to hunt for such barriers. They found six kinds of inclusivity bugs unique to these products and fixed them. The study shows teams a practical way to spot who a product quietly excludes before release.
Catalog record
| CREATOR | Andrew Anderson, Fatima A. Moussaoui, Jimena Noa Guevara, Md Montaser Hamid, Margaret Burnett |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19033
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
“Over-the-Hood” AI Inclusivity Bugs and How 3 AI Product Teams Found and Fixed Them DOI: 99.9999/9999999.9999999 CCS: Human-centered computing User studies CCS: Computing methodologies Intelligent agents Andrew Anderson, Fatima A. Moussaoui, Jimena Noa Guevara, Md Montaser Hamid, Margaret Burnett Abstract . While much research has shown the presence of AI’s “under-the-hood” biases (e.g., algorithmic, training data, etc.), what about “over-the-hood” inclusivity biases: barriers in user-facing AI products that disproportionately exclude users with certain problem-solving approaches? Recent research has begun to report the existence of such biases—but what do they look like, how prevalent are they, and how can developers find and fix them? To find out, we conducted a field study with 3 AI product teams, to investigate what kinds of AI inclusivity bugs exist uniquely in user-facing AI products, and whether/how AI product teams might harness an existing (non-AI-oriented) inclusive design method to find and fix them. The teams’ work revealed 83 instances of 6 AI inclusivity bug types unique to user-facing AI products, their fixes covering 47 bug instances, and a new GenderMag inclusive design method variant, GenderMag-for-AI, that is especially effective at detecting certain kinds of AI inclusivity bugs. 1. Introduction Figure 1 . An AI-powered agricultural tool for predicting when crops are at risk of dying from cold temperatures (y-axis) over time (x-axis). The risky events are where the AI-predictions for this crop (yellow line) intersect with the forecast’s low temperature (blue line). Suppose an Artificial Intelligence (AI) product team creates a dashboard to help farmers and other agricultural stakeholders apply AI to help find when temperatures sink so low that 50% of their crop will die (Figure 1 ). Farmers can use the dashboard to decide whether or not to deploy frost-mitigation techniques to keep their crops above this temperature. However, the AI’s user-facing information in Figure 1 may not be useful to all its intended users. Farmers and other agricultural stakeholders have diverse backgrounds, economic resources, gender identities, ages, education levels, and agricultural experience levels. Suppose the farmer does not have much engineering/math background or has limited finances to deploy expensive frost-mitigation techniques ( Saxena et al. 2023 ) . This raises the question of whether a wide range of diverse farmers can decide, using this tool, what to actually do with their fields. If not, who would it fail to serve, and how should the tool change to include them? These are human-centered AI questions (HCAI). The HCI area of inclusive design aims to answer questions like these, so as to create user experiences that are “usable and understandable by as many people as possible, considering users’ diverse needs, backgrounds, and experiences” ( Interaction 2024 ) . Usability bugs that disproportionately exclude some groups from receiving a product’s intended benefits are sometimes termed “inclusivity bugs” ( Miller 2021 ) . In the field of AI, some inclusivity bugs are “under-the-hood,” such as algorithmic or training data biases. This paper instead considers “over-the-hood” inclusivity bugs in user-facing AI products. Recently, research has begun to report the presence of over-the-hood inclusivity bugs in AI products (e.g., ( Anderson et al. 2024 ; Hamid et al. 2025 ) ), but has not yet investigated AI inclusivity bugs that are unique to user-facing AI products , which we term “ AI inclusivity bugs .” We distinguish AI inclusivity bugs from other usability bugs by the following defining criteria: (1) AI inclusivity bugs exist only in user-facing AI information, making them AI usability bugs. (2) AI inclusivity bugs disproportionately disadvantage certain groups of AI product users, making them AI inclusivity bugs. In this paper, we report on a field study with three AI product teams to investigate AI inclusivity bugs: Team Game, Team Weather, and Team Farm. Team Game was working on explaining an AI-powered game that involved sequential decision making. Team Weather was working on AI-powered agriculture, whose prototype (Figure 1 ) was predicting whether and when to deploy frost-mitigation approaches. Finally, Team Farm was working on AI-powered irrigation scheduling. Through our field study with these AI product teams, we investigated the following research questions: RQ1 : What types of user-facing AI inclusivity bugs do AI product teams find? What do they look like? How common are they? RQ2 : How do AI product teams fix the AI inclusivity bug instances they find? RQ3 : Is an existing inclusive design method (GenderMag in this paper), which was not designed particularly for AI, “enough” for AI product teams to be effective at finding user-facing AI inclusivity bugs, or is something AI-specific needed? RQ4 : If something AI-specific is needed, how does the inclusive design method need to change? How effective are these changes? 2. Background: The Gender Inclusiveness Magnifier (GenderMag) Method The inclusive design method the AI product teams used was GenderMag (Gender Inclusiveness Magnifier) ( Burnett et al. 2016 ) . The GenderMag method is an inclusive design and evaluation method to assist evaluation teams in improving technology products’ inclusivity to diverse users. Multiple empirical studies have shown GenderMag’s efficacy at identifying inclusivity bugs and pointing toward fixes ( Burnett et al. 2016 ; Guizani et al. 2022 ; Murphy-Hill et al. 2024 ; Padala et al. 2020 ; Shekhar and Marsden 2018 ; Vorvoreanu et al. 2019 ) . At GenderMag’s core are five problem-solving styles , shown in Table 1 . Each style has a range of problem-solving style values , capturing diverse problem-solving approaches. These five problem-solving style types have repeatedly been shown in research to have strong ties to both problem-solving and gender ( Anderson et al. 2024 ; Burnett et al. 2016 ; Stumpf et al. 2020 ; Vorvoreanu et al. 2019 ) . Table 1 . The five GenderMag problem-solving style types (rows), and the range of values for each GenderMag persona (columns). These problem-solving styles have empirically statistically clustered by people’s genders (e.g., ( Anderson et al. 2024 ; Burnett et al. 2016 ; Guizani et al. 2022 ; Stumpf et al. 2020 ; Vorvoreanu et al. 2019 ) ). Table 1 shows these five ranges of values, with distinguished endpoints (columns 2 & 4). The set of values in each column are grouped into one of three personas. The five problem-solving style values on the left are assigned to the “Abi (Abigail/Abishek)” persona. The five values on the right are assigned to “Tim (Timara/Timothy).” A mix of values are assigned to “Pat (Patricia/Patrick).” The principle behind GenderMag is that, when technology simultaneously supports the “endpoint” personas Abi and Tim, every problem-solving style value within the ranges those endpoints define is also supported. If those who apply GenderMag identify that something in the technology does not support one of these endpoints, GenderMag defines that occurrence as an inclusivity bug ( Guizani et al. 2022 ) . These inclusivity bugs are problem-solving inclusivity bugs, since they disproportionately impact people with that problem-solving style value. The inclusivity bugs identified by the five problem-solving style types in Table 1 are also gender inclusivity bugs. They are gender inclusivity bugs because these five problem-solving style types capture (statistical) gender differences in how people problem-solve ( Anderson et al. 2024 ; Burnett et al. 2016 ; Guizani et al. 2022 ; Stumpf et al. 2020