Testing Decision Makers without Counterfactuals

Yakov Babichenko · 2026

Read Testing Decision Makers without Counterfactuals by Yakov Babichenko (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02095.txt.

Introduction

Babichenko asks whether an observer can tell which of two agents knows more. A decision maker picks arms in a bandit setting. Only the chosen arm's outcome is observed each round. An adviser gives recommendations in parallel rounds. Their outcomes show only when they match the choice.

Both agents hold partial information about the arms. The observer sees decisions, advice, and realized outcomes. A test must select one agent from that data. Scoring tests assign a number to each observation. The agent with the higher average score is selected. Strategic agents aim to be selected by the test. The study focuses on simple and symmetric cases. Long-run identification is the central question.

Catalog record

CREATORYakov Babichenko
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02095

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Testing Decision Makers without Counterfactuals Yakov Babichenko Abstract A decision-maker (DM) repeatedly makes choices under uncertainty in a bandit environment, where only the realization of the chosen arm is observed. Another competing agent, the adviser (AD), repeatedly provides recommendations, but the realizations of these recommendations are unobserved unless they coincide with the DM’s choice. Both agents possess partial information about the arms’ realizations. The central question we focus on is whether, in the long run, an outside observer can identify which agent is more informed based solely on the observed decisions, recommendations, and arm realizations. A test selects one of the agents based on the observed data. We focus primarily on the class of scoring tests , which assign a numerical score to each observation and select the agent according to the average score. We study strategic agents whose objective is to be selected by the test. For simultaneous arm choices, we show that there exists a scoring test that successfully identifies the more-informed agent. For sequential arm choices, however, no such scoring test exists. Finally, we explore the tension between identifying the more-informed agent and maximizing welfare. A DM whose objective is to pass the test may not necessarily make welfare-maximizing decisions. In a binary-arm environment, we show that no scoring test can simultaneously identify the more informed agent and achieve more than half of the welfare attained by welfare-maximizing decisions. 1 Introduction ”Counterfactual reasoning, which deals with what-ifs, might strike some readers as unscientific. Indeed, empirical observation can never confirm or refute the answers to such questions.” Judea Pearl, The Book of Why: The New Science of Cause and Effect The concern noted by Judea Pearl regarding unobserved counterfactuals is particularly relevant when evaluating the quality of decision-making. If a decision leads to poor consequences, one could always argue that an alternative choice would have produced even worse consequences; a claim that cannot be confirmed or refuted through empirical observation. Many decision problems involve counterfactuals that are difficult, if not impossible, to evaluate, such as geopolitical choices, corporate strategies, or major life decisions. Yet in these situations, it remains desirable to assess the quality of decisions, ideally relying only on empirical observations. As a leading motivation, we focus on geopolitical decisions, though the model we propose applies more broadly to any setting where one agent currently makes decisions on behalf of an entity and another competing agent could potentially replace the current decision-maker in the future. Geopolitical decisions are made by the government (the decision-maker in our model). To evaluate these decisions, one needs a reference point for comparison. A natural reference point, often available in such contexts, is the opposition (the adviser in our model), which proposes alternative courses of action. 1 1 1 Other potential reference points include the performance of comparable countries or of the same country in the past. These, however, are debatable, as it is often unclear which country or which historical period is most comparable to the present situation. There is, however, an inherent asymmetry between the government and the opposition: the consequences of the government’s decisions are observable, while the opposition’s proposals remain counterfactuals whose consequences will never be observed. Decisions of this kind are typically made under uncertainty. The quality of an agent (the government or the opposition) is determined by the information it holds about the decision problem. More concretely, an agent must decide which arm to pull and privately holds information about the possible realizations. Only the realization of the arm pulled by the government is observed. We pose the following question: If one of these agents is more informed about the arms’ realizations than the other, can an outside observer identify which agent it is? Specifically, by more informed we mean that one agent has access to the other’s information and, in addition, possesses private information of her own; see ( 6 ) . In the context of geopolitical decisions, this notion of more informativeness may capture a more accurate perception of reality. By identifying the more informed agent, we mean selecting one of the agents based solely on observable data. In our case, the observable data are given by the government’s decisions, their realized welfare, and the opposition’s advice. We consider the above problem in a repeated environment, where the outside observer has access to a sequence of such observations. Our question is whether it is possible, in the long run, to identify the more-informed agent. A selection rule—namely, a mapping that assigns an agent to every sequence of observations—will be called a test . In the context of geopolitical decisions, a test can be thought of as a guideline for the public to assess the relative quality of the government and the opposition. However, the fact that both agents are being tested can affect their behavior. Each seeks to ensure that, by the end of the interaction, her track record appears favorable. We take an extreme perspective and assume that each agent’s sole objective is to maximize her chances of being selected by the test. This assumption induces a zero-sum game between the agents. The question posed above can thus be reinterpreted as whether there exists a test that identifies the more-informed agent in equilibrium of the zero-sum game induced by the test. Whenever such a test exists, it naturally raises a further question: What behavior does such a test impose on the decision-maker? And how far does she deviate from the welfare-maximizing course of action? The public, in fact, has two objectives: to evaluate the quality of the decision-maker (to select a capable leader in the future) and, at the same time, to incentivize the decision-maker to choose optimal actions (to secure good welfare in the present). If the test induces substantially suboptimal decisions, however, the effectiveness of such an assessment becomes questionable. 1.1 Main Results We primarily focus on the class of scoring tests , which assign a numerical value to each observation (independently of time) and determine the winner by averaging these values. 2 2 2 Equivalently, a scoring test can be interpreted as assigning a score to each agent, with the winner being the agent whose average score is higher. Focusing instead on the difference between the two scores yields an equivalent representation by a single numerical value. This class is natural in that it treats every period symmetrically. We distinguish between simultaneous decisions , where the two agents choose an arm at the same time, and sequential decisions , where one agent reacts to the choice of the other. In the geopolitical context, the sequential-decision setting is arguably more natural: typically, the opposition responds to the decisions made by the government rather than announcing a course of action simultaneously. Our main results are as follows: 1. There exists a scoring test that identifies the more informed agent in the simultaneous-decision environment. This holds even when the environment changes over time—that is, when each period corresponds to a different bandit instance (see Theorem 1 ). 2. No scoring test exists that can identify the more informed agent in the sequential-decision environment. This remains true even when the environment is fixed over time—that is, when the same bandit instance is repeated in

Read the complete text