Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing, Bettina Berendt, Pieter Delobelle · 2026
Read Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs by David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing, Bettina Berendt, Pieter Delobelle (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03087.txt.
Description
David Hartmann and five co-authors make fairness auditing of black-box language models cheaper. Such models show systematic biases across demographic groups, and outside auditors can only probe them through costly queries. The authors frame auditing as estimating a fairness metric under uncertainty, and introduce BAFA, an active auditor that plans its queries instead of sampling blindly.
BAFA keeps a set of surrogate models consistent with the answers seen so far and computes uncertainty intervals for the fairness metric. It then picks the queries that narrow those intervals fastest. On two standard fairness case studies, BAFA reached tight error targets with up to forty times fewer queries than stratified sampling, with steadier performance over time and lower variance between runs.
Catalog record
| CREATOR | David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing, Bettina Berendt, Pieter Delobelle |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03087