Instance-Adaptive Hypothesis Tests with Heterogeneous Agents

2025

Read Instance-Adaptive Hypothesis Tests with Heterogeneous Agents (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21178.txt.

Introduction

Flora C. Shi, with Martin J. Wainwright and Stephen Bates, studies hypothesis testing when people in a population differ and hold private information. A single test used for everyone performs worse than a test that could somehow see each person's hidden type, so uniform testing wastes accuracy.

The authors design menus of contracts that pair a test suited to each type with a payment rule, so people reveal their type by their own choice. They fully describe which menus match the ideal performance for any mixed population, and show that revealing information can cost almost nothing compared with one standard test. Numerical examples display better trade-offs between the two kinds of statistical error.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21178

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Instance-Adaptive Hypothesis Tests with Heterogeneous Agents Abstract We study hypothesis testing over a heterogeneous population of strategic agents with private information. Any single test applied uniformly across the population yields statistical error that is sub-optimal relative to the performance of an oracle given access to the private information. We show how it is possible to design menus of statistical contracts that pair type-optimal tests with payoff structures, inducing agents to self-select according to their private information. This separating menu elicits agent types and enables the principal to match the oracle performance even without a priori knowledge of the agent type. Our main result fully characterizes the collection of all separating menus that are instance-adaptive, matching oracle performance for an arbitrary population of heterogeneous agents. We identify designs where information elicitation is essentially costless, requiring negligible additional expense relative to a single-test benchmark, while improving statistical performance. Our work establishes a connection between proper scoring rules and menu design, showing how the structure of the hypothesis test constrains the elicitable information. Numerical examples illustrate the geometry of separating menus and the improvements they deliver in error trade-offs. Overall, our results connect statistical decision theory with mechanism design, demonstrating how heterogeneity and strategic participation can be harnessed to improve efficiency in hypothesis testing. Flora C. Shi † Martin J. Wainwright †,⋆ Stephen Bates † Laboratory for Information and Decision Systems Statistics and Data Science Center EECS † and Mathematics ⋆ Massachusetts Institute of Technology 1 Introduction Hypothesis testing involves making a discrete choice and constitutes the most basic form of decision-making under uncertainty. It occupies a central role in practice, serving as an assessment of evidence in most scientific research. Testing, however, rarely occurs in isolation. In a broader system-level context, outcomes of a test have downstream implications, such as deploying a new drug, adding a new feature to a commercial product, or publishing a scientific finding. While all parties involved are affected, stakeholders may be differentially impacted by the results. In the other direction, upstream of the phase of data collection, these same parties may have additional, potentially private information about the object of study and make choices that affect the data generation. For example, a pharmaceutical company sponsoring a phase III clinical trial for a candidate drug implicitly has a belief about the drug’s efficacy and chooses to run a trial accordingly. In this way, the distribution of the data observed, and thus the performance of any method for statistical decision-making, is altered by strategic behavior of the agents involved. In this paper, we study hypothesis testing based on data collected from a heterogeneous population of agents, and analyze how their strategic behavior impacts the design of optimal statistical tests. More specifically, we consider a principal that seeks to carry out a collection of hypothesis tests. Each test is controlled by an agent that possesses private information—not known to the principal—and this private information can differ across agents. Our main contribution is to show how to design a protocol for hypothesis testing that yields statistically optimal results even when the private information of the agents is not known in advance. As a simple illustration, suppose there are two agent types: a “good” type with a high prior probability of being non-null and a “bad” type with a low prior probability of being non-null. The decision-maker evaluates agents using p -value thresholds, which determine the stringency of the test: smaller thresholds reduce false positives but also reduce true discoveries. For a heterogeneous collection of tests, one natural measure of aggregate performance is based on the false discovery rate and true discovery rate; cf. equations ( 5a ) and ( 5b ). The false discovery rate (FDR) corresponds to the fraction of rejections that are false, while the true discovery rate (TDR) corresponds to the rate of true discoveries across the mixture of types, and captures a notion of overall power. Figure 1: trade-offs between FDR and TDR for a population with two agent types: “good” and “bad”. The light-blue region corresponds to all FDR-TDR pairs that can be achieved by any testing protocol. Its upper boundary (solid red) defines the Pareto frontier that can be achieved by an oracle that knows the agent type associated with each observation, and applies different test thresholds to each. The orange solid curve shows the result of applying a single uniform test to the full agent population. The dotted lines show the performance of applying a single test to only the “good” agents (purple line), and only the “bad” agents (green line); applying these two protocols also requires knowledge of the agent type. Figure 1 illustrates how heterogeneity undermines efficiency. The light-blue region shows all achievable (FDR, TDR) pairs when the decision-maker is permitted to apply different tests to each agent type; the upper boundary of this region (marked in a solid red line) corresponds to the oracle Pareto frontier that could be achieved if the principal were given access to the hidden agent type. A single threshold applied uniformly across both types (orange curve) lies substantially below the Pareto frontier. We can also compare to two other single threshold protocols that do require knowledge of agent type. Applying a test to only the “good” agents yields the purple curve; it has a more favorable trade-off, but still lies below the Pareto frontier. Testing only the “bad” type (green curve) defines the lower boundary of the achievable region. When the agent types are known, it is possible to tailor thresholds to different agent types so as to balance error trade-offs more effectively: moderate thresholds for good type to increase power, and stricter thresholds for bad types to limit false discoveries. It is exactly this tailored choice of thresholds that yields the oracle Pareto frontier (red curve). However, in practice, agent types are not directly observable, so we are led to the following question: Question: Is it possible to match the statistical performance of the oracle even when agent types are unknown a priori? The main contribution of this paper is to answer this question in a constructive way: we both characterize when it is possible to match the oracle and provide a concrete procedure for constructing testing protocols that do so. The key insight in our work is that, because agents are assumed to behave strategically, their actions can reveal private information. We leverage this by offering a menu of possible statistical tests, each paired with an associated payoff structure—together forming what we call a contract. We show that it is possible to design a menu of contracts such that an agent’s choice of contract reveals their private information, and moreover, such that each agent selects the statistical threshold that is optimal for their type. This design yields a testing protocol whose statistical error trade-offs match the oracle Pareto frontier, even though the principal has no a priori knowledge of agent types. 1.1 Our contributions We study a game-theoretic model of hypothesis testing with a heterogeneous population of strategic agents. The decision-maker (principal) designs a hypothesis testing protocol, while agents decide whether to participate based on private information and expected payoffs. Participation requires a fixed cost, and agents receive a reward if they pass the

Read the complete text