Sequential Fairness Auditing with Limited Output Access Conference: Fifth European Conference on Algorithmic Fairness; September 02-September 04, 2026; Ghent, BE

Ioannis Pitsiorlas, Martha V. Sourla, Marios Kountouris · 2026

Read Sequential Fairness Auditing with Limited Output Access Conference: Fifth European Conference on Algorithmic Fairness; September 02-September 04, 2026; Ghent, BE by Ioannis Pitsiorlas, Martha V. Sourla, Marios Kountouris (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30338.txt.

Introduction

Ioannis Pitsiorlas, Martha Sourla, and Marios Kountouris give external auditors a statistical method for checking AI fairness while evidence is still arriving. Regulators increasingly demand proof that deployed systems treat groups fairly. Independent auditors usually see only model outputs through queries. Fixed-sample tests fit that setting poorly. Their framework makes auditing sequential and tolerance-aware.

Auditors accumulate evidence from a finite pool of test cases. A sequential generalized likelihood-ratio test tracks the accumulating evidence. Auditing stops once compliance or violation has sufficient support. The method covers decision-based statistical parity and equal opportunity audits. Score and logit based proxy audits extend it when richer outputs exist. Fairness metric choice strongly affects audit efficiency, they find. Richer outputs cut query counts substantially in some regimes. Near decision thresholds, those gains shrink to almost nothing.

Catalog record

CREATORIoannis Pitsiorlas, Martha V. Sourla, Marios Kountouris
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30338

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sequential Fairness Auditing with Limited Output Access Conference: Fifth European Conference on Algorithmic Fairness; September 02-September 04, 2026; Ghent, BE Ioannis Pitsiorlas, Martha V. Sourla, Marios Kountouris Abstract . External evaluations are becoming increasingly central to the governance of AI systems. In practice, however, independent auditors often have limited access to deployed models and must rely on query-based interactions. Most existing fairness evaluation methods assume static datasets and fixed-sample statistical tests, making them poorly suited to real-world auditing scenarios in which evidence must be collected sequentially under query constraints. In this work, we formulate fairness auditing as a tolerance-aware sequential hypothesis-testing problem under limited model output access. We develop a sequential generalized likelihood-ratio framework that allows auditors to accumulate evidence from a finite audit pool and stop once sufficient support for compliance or violation has been obtained. The framework is instantiated for decision-based Statistical Parity and Equal Opportunity audits, and extended to score- and logit-based proxy audits when richer observables are available. Our results show that both the fairness metric and the level of model access significantly affect audit efficiency, and that the benefits of richer output information are not uniform across auditing settings. In particular, richer outputs can substantially reduce the number of queries required for some fairness metrics and operating regimes, while offering limited gains in near-threshold cases. This work provides a practical statistical framework for sequential fairness auditing under realistic deployment constraints. 1. Introduction Algorithmic decision systems are increasingly subject to regulatory oversight. Frameworks such as the EU AI Act ( 6 ) and local bias audit laws ( 12 ) require statistical evidence of compliance for deployed artificial intelligence (AI) systems. In practice, however, external auditors rarely have access to model parameters, training data, or internal documentation. Instead, auditing often occurs through query-based interaction with deployed systems, where auditors observe outputs for selected inputs and must draw conclusions from limited evidence ( 9 ; 4 ) . In this paper, we distinguish between exact black-box audits of decision-based fairness constraints and access-aware proxy audits that use richer observables, such as scores or logits, to study how additional observability affects sequential evidence accumulation. Most current fairness evaluation methods rely on static disparity metrics such as Statistical Parity (SP) or Equal Opportunity (EO) ( 8 ; 7 ) and assess compliance using fixed-sample hypothesis tests ( 15 ; 18 ) . These approaches are typically designed for batch evaluation on pre-collected datasets. In contrast, real-world audits often involve sequential and query-efficient evidence collection under limited query budgets, where auditors adaptively gather observations and must decide whether sufficient evidence has been obtained to certify compliance or detect violations ( 17 ) . Consider an external auditor evaluating a credit approval system for compliance with fairness regulations. The auditor does not have access to the model’s parameters or training data and instead interacts with the system through queries, submitting input profiles, and observing the model’s decisions. Suppose the regulator requires that the disparity in approval rates between demographic groups does not exceed a tolerance threshold \delta . Because each query may correspond to a costly or limited audit interaction, the auditor seeks to determine compliance using as few queries as possible. This setting naturally leads to a sequential statistical decision problem: as evidence accumulates, the auditor must decide whether sufficient evidence exists to certify compliance, detect a violation, or continue collecting observations. Recent work has emphasized auditing under limited access to deployed systems, often framing auditing as a statistical decision problem over observable outputs ( 13 ; 4 ) . However, existing approaches do not provide concrete sequential procedures with explicit stopping rules for efficient auditing under query constraints. Sequential hypothesis testing provides principled tools for decision making under incremental data collection, enabling early stopping with controlled error rates ( 16 ; 10 ) . However, to the best of our knowledge, these methods have not been developed into a concrete generalized likelihood ratio (GLR)-based framework for tolerance-aware fairness auditing under limited access. In this work, we formalize fairness auditing as a tolerance-aware sequential hypothesis testing problem. Given a fairness metric g(f) and regulatory tolerance \delta , we consider the composite hypothesis test H_{0}:g(f)\leq\delta\qquad\text{vs.}\qquad H_{1}:g(f)>\delta. To address this problem, we adopt a GLR framework with sequential stopping boundaries inspired by Wald-type tests. Unlike fixed-sample procedures, the proposed protocol allows evidence to accumulate sequentially and terminates once sufficient statistical support is obtained or a predefined query budget is exhausted. This formulation reflects realistic regulatory settings in which small disparities may be legally permissible, and compliance is defined relative to a tolerance threshold rather than zero disparity. Our empirical study reveals several important phenomena in sequential fairness auditing. First, tolerance-aware auditing exhibits strong boundary effects: models whose fairness gaps lie near the regulatory tolerance often yield inconclusive outcomes within realistic query budgets. Second, richer output-access regimes can substantially reduce audit cost for SP and for some operating regimes, but the gains are not uniform across fairness criteria. In particular, the EO results show that richer observables do not automatically yield faster or more conclusive audits near the regulatory threshold. Third, EO audits are significantly more sample intensive than SP audits because the EO procedure operates on a conditional Y=1 audit pool that is typically smaller, and may therefore provide fewer informative observations and less favorable group counts than the full SP pool. These findings highlight that fairness audit outcomes depend not only on the deployed model but also on the statistical protocol and level of access available to the auditor. By operationalizing fairness auditing as a sequential inference problem, our framework provides a foundation for analyzing the reliability and efficiency of real-world AI audits under realistic deployment constraints. 1.1. Contributions Our contributions are summarized as follows: (1) Sequential formulation of fairness auditing: We formulate fairness auditing under limited model output access as a sequential statistical decision problem, where evidence is collected through query-based interaction and compliance is defined relative to a tolerance threshold. (2) Sequential GLR testing protocol: We develop a sequential GLR audit with explicit stopping rules and operational decision thresholds for composite fairness constraints. (3) Access-aware auditing analysis: We study how different access regimes affect the efficiency of fairness audits, distinguishing between decision-only audits that directly target hard-decision SP/EO estimands and proxy audits based on scores or logits when richer observables are available. (4) Empirical study: Through experiments on benchmark datasets, we identify practical phenomena in sequential fairness auditing, including increased difficulty for conditional metrics such as EO and statistical ambiguity near regulatory thresholds. 2. Related Work 2.1. Statistical Fairness Evaluation Prior work studies algorithmic fairness through statistical disparity metrics that compare predictive behavior across demographic groups. Some examples are SP, which requires equal positive prediction rates across groups, and EO, which requires equal true positive rates conditioned on the ground truth ( 8 ; 7 ) . These metrics and their variants are widely used to quantify group-level disparities in machine learning systems and have become common benchmarks in fairness evaluation. Several approaches incorporate statistical testing to assess whether the observed group disparities are significant. For example, hypothesis testing frameworks for fairness metrics have been studied in ( 15 ; 18 ) . These methods typically construct confidence intervals

Read the complete text