Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3791552 ISBN: 979-8-4007-2278-3/2026/04 CCS: Social and professional topics Government technology policy CCS: Software and its engineering Software verification and validation CCS: Applied computing Law

Angela Jin, Alexander Asemota, Dan E. Krane, Nathaniel Adams, Rediet Abebe · 2026

Read Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3791552 ISBN: 979-8-4007-2278-3/2026/04 CCS: Social and professional topics Government technology policy CCS: Software and its engineering Software verification and validation CCS: Applied computing Law by Angela Jin, Alexander Asemota, Dan E. Krane, Nathaniel Adams, Rediet Abebe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10875.txt.

Introduction

Jin and colleagues ask how standard design shapes results, through ASB 018, a standard for auditing probabilistic genotyping software used by United States courts for DNA samples. Weak standards can hide and lend trust to poor systems. Study of ASB 018 and five audit reports finds gaps between what the standard wants and what it allows. Passing audits need not set use limits from seen failures, though the standard aims at that. The authors trace the gaps to fuzzy wording and vague terms, ending with tips for writing standards and judging them.

Catalog record

CREATORAngela Jin, Alexander Asemota, Dan E. Krane, Nathaniel Adams, Rediet Abebe
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108779
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10875

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Compliant But Unsatisfactory: The Gap Between Auditing Standards and Practices for Probabilistic Genotyping Software Conference: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems; April 13–17, 2026; Barcelona, Spain Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26), April 13–17, 2026, Barcelona, Spain DOI: 10.1145/3772318.3791552 ISBN: 979-8-4007-2278-3/2026/04 CCS: Social and professional topics Government technology policy CCS: Software and its engineering Software verification and validation CCS: Applied computing Law Angela Jin, Alexander Asemota, Dan E. Krane, Nathaniel Adams, Rediet Abebe Abstract . AI governance efforts increasingly rely on audit standards: agreed-upon practices for conducting audits. However, poorly designed standards can hide and lend credibility to inadequate systems. We explore how an audit standard’s design influences its effectiveness through a case study of ASB 018, a standard for auditing probabilistic genotyping software—software that the U.S. criminal legal system increasingly uses to analyze DNA samples. Through qualitative analysis of ASB 018 and five audit reports, we identify numerous gaps between the standard’s desired outcomes and the auditing practices it enables. For instance, ASB 018 envisions that compliant audits establish restrictions on software use based on observed failures. However, audits can comply without establishing such boundaries. We connect these gaps to the design of the standard’s requirements such as vague language and undefined terms. We conclude with recommendations for designing audit standards and evaluating their effectiveness. † † cc-license: by 1. Introduction Figure 1. A simplified depiction of forensic lab use of probabilistic genotyping software to produce evidence in a criminal case. The goal of forensic DNA testing is to determine whether a person of interest (e.g., the defendant) could be the source of some or all of the DNA recovered from a crime scene. To this end, (A) a lab analyst first prepares two physical DNA samples: an evidence sample (e.g., a swabbing of a shoe collected as evidence) and a reference profile. (B) The analyst then runs the PGS with these DNA samples as inputs, along with other inputs chosen by the analyst. The software outputs a likelihood ratio (LR)—a statistic that describes similarities observed between the evidence sample and reference profile. (C) Before the LR is presented as evidence at trial, a court may conduct an admissibility hearing to determine whether the evidence is sufficiently reliable to be presented in front of a jury. (D) If the judge deems the evidence admissible, the LR will be presented as evidence at trial. . Flow-chart depicting stages of the forensic DNA testing process and the introduction of the probabilistic genotyping software (PGS) likelihood ratio output as evidence in a criminal case. The flowchart has four stages: (A) DNA profile preparation, (B) Use of PGS to produce likelihood ratio, (C) A potential admissibility hearing, and (D) Presentation of LR as evidence at trial. The figure depicts four inputs to the probabilistic genotyping software: the crime scene DNA sample (i.e., evidence sample), the person of interest's DNA sample (i.e., reference profile), the forensic lab analyst's estimate of the number of contributors, and other analyst inputs. The software outputs a likelihood ratio (LR), which the figure then depicts as being presented in front of a judge. If the judge deems the LR admissible, the LR is shown as trial evidence in front of a jury. There is a box depicting the PGS. There is also a larger box encompassing both the PGS and lab analyst with a dotted line depicting the broader PGS system that takes physical DNA samples as input and outputs a LR. Over the past decade, researchers and practitioners have developed an ecosystem of tools—including guidelines, frameworks, and software toolkits—to promote fairness, transparency, and accountability throughout the AI lifecycle ( 44 ; 29 ; 43 ; 61 ; 49 ; 2 ) . Amidst growing calls for algorithm audits, recent work highlights that such tools can be crucial for ensuring these evaluations result in meaningful accountability ( 56 ; 2 ; 39 ; 62 ; 36 ) . However, tools for responsible AI may fail to achieve their intended outcomes when deployed in complex organizational settings and practices ( 75 ; 6 ; 20 ; 63 ; 44 ; 65 ) . In response, a burgeoning body of work investigates how to design audit tooling that effectively helps audits create accountability ( 56 ; 44 ; 61 ; 29 ; 39 ; 62 ; 2 ) . As researchers and policymakers increasingly call for audit standards (i.e., agreed-upon best practices) ( 25 ; 18 ; 39 ; 62 ; 26 ; 31 ) , we ask: How can the design of an audit standard undermine its effectiveness? We specifically examine how a standard’s formal requirements may enable audits that comply with the letter of the standard while failing to meet the goals envisioned by the standard’s developers. We explore this question through the lens of ASB 018 1 1 1 We study the first edition of ASB 018, which was developed and published in 2020 by the American Academy of Forensic Sciences Standards Board (ASB) ( 4 ) . , a standard for auditing probabilistic genotyping software (PGS) used by U.S. forensic laboratories to analyze DNA samples in criminal cases. ASB 018 provides a critical case study not only because it mirrors audit processes in other domains like hiring ( 62 ; 39 ; 26 ; 24 ) , healthcare ( 54 ; 77 ) , and social services ( 13 ) but also because forensic labs already cite ASB 018 compliance to convince judges that PGS outputs are sufficiently reliable to be presented as evidence in criminal trials ( 71 ) . We investigate ASB 018’s design and effectiveness through two research questions: (RQ1) What gaps exist between the practices ASB 018 envisions and the practices it enables? 2 2 2 We study forensic labs’ audit practices by examining their representation in publicly available audit reports; see Section 4 . , and (RQ2) How does the design of ASB 018 facilitate these gaps? In this work, we analyzed five publicly available PGS audit reports spanning various lab sizes, software versions, and years. We first assessed each audit report for compliance with ASB 018’s formal requirements. After determining that all five audits could be interpreted as compliant, we qualitatively coded ASB 018 and its accompanying factsheet to identify its desired outcomes across five audit stages ( 56 ) : Audit Scope, Standards Identification, Performance Analysis, Post-Audit Judgment, and Audit Communication. We then qualitatively coded each audit report to identify practices that misalign with the standard’s goals (RQ1). Finally, we conducted a second round of coding on ASB 018, this time focusing on the language and organization of its requirements to identify aspects of its design that allow unsatisfactory audits to remain technically compliant (RQ2). Our findings reveal that the analyzed audits are ASB 018 compliant, but unsatisfactory : audit practices consistently fall short of ASB 018’s envisioned goals across all five audit stages, yet still satisfy the standard’s formal requirements . For instance, while ASB 018 envisions testing the broader sociotechnical system, the audits primarily measure narrow software performance. Similarly, although the standard intends for audits to establish boundaries on software use based on failures

Read the complete text