Randomization Inference in Two-Sided Market Experiments
Jizhou Liu, Azeem M. Shaikh, Panos Toulis · 2025
Read Randomization Inference in Two-Sided Market Experiments by Jizhou Liu, Azeem M. Shaikh, Panos Toulis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06215.txt.
Introduction
Jizhou Liu, Azeem Shaikh, and Panos Toulis build sound statistics for experiments on two-sided markets in a 2025 paper. Platforms like lending or shopping sites randomize offers to learn what works, but buyers and sellers spill effects onto each other. Standard tests can misread that spillover. Their framework keeps exact validity for sharp null claims in finite samples, a strict standard.
For weaker claims, the usual studentized test does not always stay valid as samples grow; the authors show how validity depends on the claim's shape. A two-way variance estimate repairs the problem. Simulations and a micro-lending network experiment confirm the fixes. Careful design keeps platform experiments honest about effects that travel.
Catalog record
| CREATOR | Jizhou Liu, Azeem M. Shaikh, Panos Toulis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 134 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06215
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Randomization Inference in Two-Sided Market Experiments Thanks: The second author acknowledges support from the National Science Foundation through grant SES-2419008. Jizhou Liu, Azeem M. Shaikh, Panos Toulis Abstract Randomized experiments are increasingly employed in two-sided markets, such as buyer–seller platforms, to evaluate the effects of marketplace interventions. These experiments must reflect the underlying two-sided market structure in their design and can therefore be challenging to analyze. In this paper, we develop a randomization inference framework for outcomes from two-sided experiments, with a focus on testing and inference for two-sided spillover effects. Our approach is finite-sample valid under sharp null hypotheses. Regarding weak null hypotheses, we find that the commonly used Neyman-style studentization does not universally ensure asymptotic validity, and we document how it depends on the specific formulation of the null. We then propose a two-way variance estimator for studentization that restores asymptotic validity. We further propose methods to improve testing power by exploiting the two-sided structure of the problem, which we validate empirically. We demonstrate our methods through a series of simulation studies and an applied example from a network experiment in micro-lending. KEYWORDS: Causal inference; Conditional Randomization Test; Two-sided Experiments; Multiple Randomization Design. 1 Introduction Randomized experiments are increasingly used to assess the impact of policies in various online domains, including online marketplaces, and streaming media platforms ( Kohavi et al., 2020 ; Athey and Luca, 2019 ; Thomke, 2003 ) . These environments generate complex datasets through two-sided interactions between multiple types of actors, such as viewers and content creators or buyers and sellers. In this context, policy evaluation is complicated due to market interference , where the policy treatment on certain units may affect the outcomes of different units in the market, including those never assigned to treatment ( Blake and Coey, 2014 ; Toulis and Parkes, 2016 ) . Traditional randomized experiments, designed for single-population settings, often fall short in addressing the complexities arising from interference in two-sided markets ( Masoero et al., 2026 ; Johari et al., 2022 ; Wager and Xu, 2021 ) . To overcome these challenges, a novel class of experimental designs, termed Multiple Randomization Designs , was developed independently by Masoero et al. (2026) ; Bajari et al. (2023) and Johari et al. (2022) , specifically for marketplace experimentation. For example, in an online marketplace with buyers and sellers, the experimenter might randomly assign half of the buyers and half of the sellers to treatment groups, applying a policy intervention (e.g., free shipping) to transactions where both the buyer and the seller are treated. Transactions where either the buyer or the seller (or both) are untreated would not receive the intervention. This design creates variation in the exposure to treatment across different transactions, which facilitates the analysis of causal effects in the presence of interference. In this paper, we propose Fisherian-style randomization tests ( Fisher, 1953 ) tailored specifically for analyzing two-sided market experiments of this kind. We formulate sharp null hypotheses on spillover effects that are naturally aligned with the two-sided structure of these experiments. Our approach adopts the conditional randomization testing framework that restricts attention to a set of “focal units” associated with each tested hypothesis ( Athey et al., 2018 ; Basse et al., 2019 ; Puelz et al., 2021 ) . In this framework, the spillover hypotheses are not sharp in the traditional sense as they cannot be used to impute all potential outcomes. However, with proper conditioning, the hypotheses become sharp and are thus amenable to standard Fisherian-style randomization. We propose two valid randomization testing procedures: a permutation-based procedure, which is straightforward to implement but relies on specific symmetry conditions on the design that may not universally hold; and an alternative procedure that is valid under arbitrary designs but requires sampling from a potentially complex space. For a third null hypothesis on total treatment effects, we identify a novel validity-power trade-off among multiple candidate tests and provide heuristic guidance on improving power. Furthermore, we extend our framework to test weak null hypotheses on average spillover and total treatment effects by incorporating studentized statistics into our randomization tests. While studentization is a well-established technique for ensuring asymptotic validity under weak nulls ( Chung and Romano, 2013 ; DiCiccio and Romano, 2017a ; Zhao and Ding, 2021 ; Wu and Ding, 2021 ; Toulis, 2025 ) , we demonstrate that its effectiveness in two-sided experiments depends crucially on how the null hypothesis is formulated and what variance estimators are used for studentization. Specifically, we show that the commonly used Neyman-style studentization is asymptotically valid under a null hypothesis concerning focal average effects, but it may fail to control size under a global average null due to the additional variance introduced by the two-sided randomization. Then, we propose a modified, two-way variance estimator for studentization that restores asymptotic validity for the global weak null by absorbing the additional sampling variation. Our key technical contribution involves extending the proof techniques of Zhao and Ding (2021) to account for these two-sided dependencies, a finding we validate through extensive simulation studies. This paper contributes to the growing literature on randomized experiments in multi-sided online marketplaces. Bajari et al. (2023) pioneered the introduction of multiple randomization designs to account for complex spillover effects occurring within marketplaces. Masoero et al. (2026) delves deeper into such designs, adopting a Neymanian perspective that emphasizes randomization-based point estimation and proposes conservative variance estimators for statistical inference. By contrast, our work considers the Fisherian perspective that instead focuses on finite-sample exact p -values via randomization-based tests. The rest of the paper is organized as follows. Section 2 describes the setup and notation. Section 3 presents the main results under sharp null hypotheses. Section 4 discusses the randomization tests for the weak null hypotheses. Section 5 examines the finite-sample behavior of our methods through simulations. Section 6 illustrates the proposed inference methods in an empirical application based on the experiment conducted in Comola and Prina (2021) . Section 7 concludes the paper. 2 Setup and Notation We begin by introducing the two-sided randomized design following the two-population buyer-seller example in Masoero et al. (2026) . Consider a marketplace comprising I buyers and J sellers, indexed by i=1,\ldots,I and j=1,\ldots,J , respectively. In the marketplace, buyers and sellers interact, producing an outcome of interest, such as the total number of transactions between the buyer-seller pair within a given time period. We denote this observed outcome between buyer i and seller j as Y_{i,j}\in\mathbb{R} . We denote the set of buyer-seller pairs as \mathbb{U}=\{(i,j):i=1,\dots,I,j=1,\dots,J\} . The I\times J matrix \mathbf{Y}=[Y_{i,j}] captures the outcomes observed in the marketplace. To examine the impact of various interventions, researchers conduct randomized experiments at the level of individual buyer-seller pairs, using a binary treatment assignment, W_{i,j}\in\{0,1\} . These interventions might include incentives such as free shipping or discounts on transaction fees. The matrix \mathbf{W}=[W_{i,j}]\in\{0,1\}^{I\times J} denotes the full treatment assignment. We use Y_{i,j}(\mathbf{w}) to denote the potential outcome for the pair (i,j) under treatment assignment \mathbf{w}\in\{0,1\}^{I\times J} , and \mathbf{Y}(\mathbf{w}) for the full I\times J matrix of potential outcomes under assignment \mathbf{w} . As usual, the observed outcomes and the potential outcomes are related to treatment assignment by the consistency relationship \mathbf{Y}=\mathbf{Y}(\mathbf{W}) . As highlighted