Incentive Aware AI Regulations: A Credal Characterisation

Anurag Singh, Julian Rodemann, Rajeev Verma, Siu Lun Chau, Krikamol Muandet · 2026

Read Incentive Aware AI Regulations: A Credal Characterisation by Anurag Singh, Julian Rodemann, Rajeev Verma, Siu Lun Chau, Krikamol Muandet (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05175.txt.

Introduction

Anurag Singh and colleagues formalize regulating black-box AI services as a mechanism-design problem with two goals: keeping non-compliant providers out while retaining compliant ones. They call that ideal the perfect market outcome, threatened by providers' private information, strategic evasion, and statistical uncertainty in finite-sample evidence. Their sharp characterization says a mechanism achieves the ideal exactly when non-compliant evidence distributions form a closed, convex credal set. Mechanisms built from hypothesis tests can achieve it, and experiments on spurious-feature and fairness-based rules validate the theory.

Catalog record

CREATORAnurag Singh, Julian Rodemann, Rajeev Verma, Siu Lun Chau, Krikamol Muandet
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE124344
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Incentive Aware AI Regulations: A Credal Characterisation Anurag Singh, Julian Rodemann, Rajeev Verma, Siu Lun Chau, Krikamol Muandet Abstract While high-stakes ML applications demand strict regulations, strategic ML providers often evade them to lower development costs. To address this challenge, we cast AI regulation as a mechanism design problem under uncertainty and introduce regulation mechanisms : a framework that maps empirical evidence from models to a license for some market share. The providers can select from a set of licenses, effectively forcing them to bet on their model’s ability to fulfil regulation. We aim at regulation mechanisms that achieve perfect market outcome, i.e. (a) drive non-compliant providers to self-exclude, and (b) ensure participation from compliant providers. We prove that a mechanism has perfect market outcome if and only if the set of non-compliant distributions forms a credal set, i.e., a closed, convex set of probability measures. This result connects mechanism design and imprecise probability by establishing a duality between regulation mechanisms and the set of non-compliant distributions. We also demonstrate these mechanisms in practice via experiments on regulating use of spurious features for prediction and fairness. Our framework provides new insights at the intersection of mechanism design and imprecise probability, offering a foundation for development of enforceable AI regulations. Keywords: AI-Regulation, Mechanism Design, Imprecise Probability, Testing by Betting 1 Introduction As ML systems are increasingly deployed in high-stakes domains ranging from credit scoring ( Baesens et al. 2003 ) to social justice ( Angwin et al. 2022 ) , their associated risks have become real ( Buolamwini & Gebru 2018 ; Laux et al. 2024 ) . Policymakers have reacted to this threat with the development of comprehensive AI governance frameworks, such as the EU AI Act ( Edwards 2021 ) . In practice, the policymakers face a delicate balancing act as the goal of regulation is twofold: (1) preventing the deployment of unsafe models by making the non-compliant model providers self exclude (e.g., those failing fairness or robustness standards) while simultaneously (2) fostering safe innovation by encouraging compliant model providers to participate in the market. We term this the perfect market outcome . Regulators could possibly achieve perfect market outcomes via regulations with white box access to models, i.e., model weights, gradients, including access to training procedure and hyper-parameters ( Shevlane 2022 ; Solaiman 2023 ) . However, in practice, regulators must often regulate models with limited black-box access, as proprietary interests and trade secrets frequently preclude the disclosure of model architectures or training data ( Pasquale 2015 ; Raji et al. 2020 ) . Therefore, some policymakers argue for an outcome-based approach to regulation ( Hadfield & Clark 2023 ; Brundage et al. 2018 ) . To enforce outcome-based regulations, regulators must determine if models fulfil regulations via finite-sample benchmarks, and thus face statistical uncertainty ( Ren et al. 2024 ; Jansen et al. 2024 ) . At the same time, model providers possess much more knowledge than regulators about model’s abilities and pitfalls. This creates an information asymmetry between providers and regulators which can be exploited by the model providers ( Casper et al. 2024 ; Li & Goel 2025 ) . In this work, we formalise this problem by casting AI regulation as a mechanism design problem under uncertainty. We propose regulation mechanisms which push the burden of proof on the providers. Instead of trying to prove “This model is unsafe”, the regulator now judges, “Is the model provider willing to bet their own capital on the model’s safety?” While recent advances in hypothesis testing ( Bates et al. 2023 ; Bates et al. 2022 ) also formalize information asymmetry under uncertainty, they primarily focus on making hypothesis tests robust to strategic behaviours. In contrast, our focus is on addressing the question: under what conditions can a regulator achieve perfect market outcomes using such statistical mechanisms? We provide a complete characterisation, proving that a regulatory requirement is enforceable if and only if the set of non-compliant distributions forms a credal set ( Levi 1980 ) —a closed, convex set of probability measures. The credal set here reflects the conditions on the regulator to be internally consistent with respect to his regulatory requirements. We assume that the regulator claims to achieve a particular market outcome with their regulatory requirements. If the set of non-compliant distributions is not a credal set, any mechanism chosen to enforce regulation will either fail to prevent strategic behaviour or inadvertently deny market access to compliant agents, demonstrating the inherent inconsistency of the regulatory requirements. Our Contributions. We now summarise our main contributions. Firstly, we formalise perfect market outcomes in AI regulation as outcome of an implementable mechanism. Secondly, we provide the full characterisation of perfect market outcome achieving regulatory requirements via credal sets. Our result further shows that quasi-convexity and lower-semicontinuity are a necessary and sufficient conditions for threshold-based regulations to achieve perfect market outcomes. These results provide insights for regulators who wish to use mechanisms to enforce their regulations with perfect market outcomes. Lastly, we derive optimal responses of risk-neutral and risk-averse model provider to the regulation mechanisms. We then develop practical regulation mechanisms, which can achieve perfect market outcomes for different regulatory requirements, and extend them to settings where regulators do not have an explicit representation of the credal set via the testing-by-betting framework ( Shafer 2021 ; Ramdas & Wang 2024 ; Grünwald et al. 2024 ) by allowing the model providers to bet on their model’s ability. We also demonstrate how practical regulation mechanisms can achieve perfect market outcomes on both synthetic and real-world datasets. 2 Preliminaries This section introduces the notation, presents the problem formulation, and reviews the necessary background on imprecise probabilities and mechanism design. 2.1 Notation and Problem Formulation We consider \mathcal{X}\subseteq\mathbb{R}^{d} as our instance space and \mathcal{Y} as our target space where \mathcal{Y}\subseteq\mathbb{R} for regression tasks and \mathcal{Y}=\{1,\dots,K\} for a K -class classification problem. Our focus will be on a supervised learning scenario where the goal is to learn a function f:\mathcal{X}\rightarrow\mathcal{Y} from a hypothesis class \mathcal{H} on the basis of data. The model’s performance is measured via a loss function \ell:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}_{\geq 0} where \ell(f(x),y) is the point-wise error the model f\in\mathcal{H} makes on a data point (x,y)\in\mathcal{X}\times\mathcal{Y} . In our setting, we consider two players: a model designer and a regulator . The model designer is responsible for learning a predictive model from data, whereas the regulator’s role is to ensure that the model complies with prescribed regulatory requirements. Let (\Omega,\mathcal{F}) be an underlying measurable space over which there exists a fixed but unknown data generating process P . We denote the corresponding measure as \mathbb{P} , as well as random variables associated with \mathcal{X} and \mathcal{Y} as X:\Omega\rightarrow\mathcal{X} and Y:\Omega\rightarrow\mathcal{Y} , respectively, such that X(\omega)=x and Y(\omega)=y . In addition, we consider an evidence space \mathcal{Z} with corresponding random variables Z_{f}:\Omega\rightarrow\mathcal{Z} indexed by f\in\mathcal{H} . For example, loss values Z_{f}(\omega)=\ell(f(X(\omega)),Y(\omega)) can be considered as an evidence associated with f . We denote the space of bounded continuous functions on \mathcal{Z} as C(\mathcal{Z}) . We consider the topology between C(\mathcal{Z}) and evidence distribution P through the standard dual pairing \langle\cdot,\cdot\rangle:C(\mathcal{Z})\times\Delta(\mathcal{Z})\to\mathbb{R} , defined by the expectation for some \pi\in C(\mathcal{Z}) : \langle\pi,P\rangle\coloneqq\mathbb{E}_{z\sim P}[\pi(z)]=\int_{\mathcal{Z}}\pi(z)\,dP(z). Consistent

Read the complete text