Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing, Bettina Berendt, Pieter Delobelle · 2026

Read Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs by David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing, Bettina Berendt, Pieter Delobelle (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03087.txt.

Description

David Hartmann and five co-authors make fairness auditing of black-box language models cheaper. Such models show systematic biases across demographic groups, and outside auditors can only probe them through costly queries. The authors frame auditing as estimating a fairness metric under uncertainty, and introduce BAFA, an active auditor that plans its queries instead of sampling blindly.

BAFA keeps a set of surrogate models consistent with the answers seen so far and computes uncertainty intervals for the fairness metric. It then picks the queries that narrow those intervals fastest. On two standard fairness case studies, BAFA reached tight error targets with up to forty times fewer queries than stratified sampling, with steadier performance over time and lower variance between runs.

Catalog record

CREATORDavid Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing, Bettina Berendt, Pieter Delobelle
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03087

Browse the full library · About the Tower of Babel