Optimizing Returns from Experimentation Programs

Timothy Sudijono, Simon Ejdemyr, Apoorva Lal, Martin Tingley · 2024

Read Optimizing Returns from Experimentation Programs by Timothy Sudijono, Simon Ejdemyr, Apoorva Lal, Martin Tingley (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05508.txt.

Introduction

Timothy Sudijono and co-authors rethink how online platforms should run their A/B tests. Firms test changes to lift a target measure, but classic testing cares about proof, not size of gain or cost of delay. The team asks how a firm with a pool of past tests should set its rules. They survey work that borrows strength across past tests.

One known frame treats testing as a constrained planning problem. The team solves it with dynamic programming and shows it maps to tuned p-value cutoffs. Under free testing, their math says firms should test far more ideas, with fewer users each, and looser cutoffs. Extensions weigh real costs and other industry wrinkles.

Catalog record

CREATORTimothy Sudijono, Simon Ejdemyr, Apoorva Lal, Martin Tingley
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05508

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimizing Returns from Experimentation Programs Timothy Sudijono, Simon Ejdemyr, Apoorva Lal, Martin Tingley Abstract Experimentation in online digital platforms is used to inform decision making. Specifically, the goal of many experiments is to optimize a metric of interest. Null hypothesis statistical testing can be ill-suited to this task, as it is indifferent to the magnitude of effect sizes and opportunity costs. Given access to a pool of related past experiments, we discuss how experimentation practice should change when the goal is optimization. We survey the literature on empirical Bayes analyses of A/B test portfolios, and single out the A/B Testing Problem [ ADMO+20 ] as a starting point, which treats experimentation as a constrained optimization problem. We show that the framework can be solved with dynamic programming and implemented by appropriately tuning p p -value thresholds. Furthermore, we develop several extensions of the A/B testing problem and discuss the implications of these results on experimentation programs in industry. For example, under no-cost assumptions, firms should be testing many more ideas, reducing test allocation sizes, and relaxing p p -value thresholds away from p = 0.05 p=0.05 . 1 Introduction Experimentation in the private sector – particularly in the online tech industry – broadly differs from experimentation in scientific fields in two ways. Firstly, the goal of the vast majority of experiments in the private sector is to optimize a metric. For example, tech firms optimize measurable business objectives such as long-term retention, sign-up rates, or revenue. Secondly, repositories of past experiments give firms strong priors on treatment effects from related experiments. Empirical evidence [ ADMO+20 , GJ17 , GMR20 , Geo22b , Geo22a , KC24 ] suggests the following stylized fact regarding experiments: treatment effect distributions are typically centered around zero, with significant dispersion around the mean. Figure 1 shows treatment effect densities for three distinct experimentation initiatives at Netflix, corroborating this point. Given this prior on treatment effects, how should an organization conduct experimentation when the goal is to optimize a metric of interest? The current statistical paradigm in industry, null hypothesis statistical testing, can be mismatched with this goal: it is indifferent to utilities and opportunity costs, and it provides no principled justification for Type I and Type II error defaults [ Man19 ] . Building on a growing literature [ ADMO+20 , AMOV23 , GMR20 , GJ17 , Den15 ] , we study an alternative Bayesian decision-theoretic framework and discuss how organizations should restructure current practice towards optimizing returns from experimentation. Figure 1 : Best-fitting Gaussian densities for three programs at Netflix. x -axis is obfuscated. Dashed line indicates x=0 . 1.1 Setup An organization conducting experimentation typically has a pool of untested ideas and limited resources with which to conduct experiments. These organizations have a population of units on which to experiment – an allocation pool – and teams to generate new ideas for experiments and subsequently analyze them. Generally, experimentation in these firms proceeds as follows. A product team generates an idea which may improve a product. The idea is tested using a randomized controlled experiment or A/B test [ KCL+09 ] . Based on the results of the A/B test, the idea is either discarded or shipped – made available to all units in the allocation pool. Often, similar tests on the same product, platform, or population can be grouped into initiatives that we will call experimentation programs . The A/B Testing Problem [ ADMO+20 ] provides a starting point for optimizing the returns from a single experimentation program. It is defined as follows. Suppose the experimentation program consists of I untested ideas or interventions indexed by i=1,\dots,I . The data is not assumed to arrive sequentially. Treatment effects for these ideas are measured in the same metric. We will think of these effects \Delta_{i} as the return associated with idea i . Data from past experiments give credence to a Bayesian assumption that \Delta_{i} are drawn iid for some distribution G . We choose an allocation of units to A/B tests, (n_{i})_{i=1}^{I} , such that \sum_{i}n_{i}=N , the total size of the allocation pool. A unit may only be assigned to one test at a time, making the allocation pool a limited resource. For those tests with n_{i}>0 , we observe the result of the experiment, given by \hat{\Delta}_{i}\sim\textsf{N}\left(\Delta_{i},\frac{\sigma^{2}}{n_{i}}\right). Normality of the observations is justified by the central limit theorem; \sigma represents typical fluctuations of the return metric, which is fairly stable and known in advance. The goal of the A/B testing problem is to select an allocation (n_{i})_{i=1}^{I} and a subset of tests S to ship, S\subseteq[I] , in order to maximize total expected returns to experimentation, \mathbb{E}\left[\sum_{i\in S}u(\Delta_{i})\right], (1) where u is an increasing utility function. We will typically take u(x)=x to be risk-neutral. However u may also capture risk-averse utilities such as u(x)=x+bx\mathbf{1}\left\{x<0\right\},b>0 , inspired by prospect theory [ KT13 ] . 1.2 Related work The idea of optimizing returns from a portfolio of A/B tests is a stark alternative to traditional statistical analysis of experiments. The decision theoretic foundations of this perspective were first discussed in [ ADMO+20 , AMOV23 , GJ17 , Man19 , GMR20 ] . These works also adopt a Bayesian or empirical Bayesian approach to the problem. See Appendix A for an extended discussion of closely related work in the analysis of collections of A/B tests. Most relevant to our work is the A/B Testing Problem in [ ADMO+20 ] , which is the first to recast experimentation as a constrained optimization problem – exactly the A/B Testing Problem in the previous section. By a careful asymptotic analysis, [ ADMO+20 ] characterizes the optimal experimentation strategy in specific settings. For t -distributed priors which are sufficiently heavy-tailed, the optimal allocation is to split N units equally across the I tests, a strategy called lean experimentation : firms should run many low-powered tests. For light-tailed t -distributed priors G with N small enough, the optimal strategy is to go big : it is optimal to allocate all N units to one test. Collectively, the aforementioned literature [ ADMO+20 , AMOV23 , GJ17 , Man19 , GMR20 ] and this paper represent a different approach to experimentation that we will refer to as the return-aware framework , to contrast it with null hypothesis testing. The framework views experimentation as a decision theoretic problem where the goal is to maximize the cumulative impact of experimentation, subject to various constraints. 1.3 Contributions. This article makes original contributions to the return-aware framework and discusses its implications for current practice. We also argue that the return-aware framework is well-suited to experimentation in industry, and worthy of future research. First, we highlight several simple, useful observations about the framework. Practical Solution Methods. The elegant results of [ ADMO+20 , AMOV23 ] leave open the optimal allocation for general priors and arbitrary values of I,N . We show that dynamic programming solves this problem for any choice of I,N , utility function u , and prior G . Connection to p -value Frameworks. In current experimentation practice, A/B tests are usually decided based on p -values and not posterior expectation calculations. We show that tuning the hypothesis testing level \alpha based on the

Read the complete text