Advancing evaluation of AI systems when humans make the decisions
Alexander D’Amour · 2025
Read Advancing evaluation of AI systems when humans make the decisions by Alexander D’Amour (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-12745701.txt.
Description
Alexander D'Amour comments on new methods for judging AI systems that help humans make high-stakes calls, in a commentary from 2025. Such systems now feed into bail, health, and lending decisions. A core problem is that past data were produced by the very human system the AI aims to improve, so outcomes are known only for cases humans chose.
D'Amour walks through a framework by Ben-Michael and colleagues that tackles this selective labels problem head-on. It sharpens the rough fixes now in use and adds sound ways to estimate key scores. He draws on his own past work in microlending and aims the piece at practitioners used to standard A-B testing.
Catalog record
| CREATOR | Alexander D’Amour |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |