Beyond Normality: Reliable A/B Testing with Non-Gaussian Data DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Mathematics of computing Hypothesis testing and con
Junpeng Gong, Chunkai Wang, Hao Li, Jinyong Ma, Haoxuan Li, Xu He · 2025
Read Beyond Normality: Reliable A/B Testing with Non-Gaussian Data DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Mathematics of computing Hypothesis testing and con by Junpeng Gong, Chunkai Wang, Hao Li, Jinyong Ma, Haoxuan Li, Xu He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23666.txt.
Introduction
Junpeng Gong and five co-authors show why the standard t-test can mislead A/B tests that guide product launches and pricing on online platforms. The test assumes roughly normal data and balanced groups, but real metrics are often skewed with long tails, and groups often differ in size. Under those conditions the false positive rate drifts, and teams risk shipping harmful changes.
The authors quantify how skewness, heavy tails, and uneven allocation distort error rates, and they derive formulas for the smallest sample size that keeps the t-test valid. For many online metrics, that size runs into the hundreds of millions. As a remedy for smaller samples, they add an Edgeworth-based correction that sharpens the p-values. Tests on data from a leading A/B testing platform confirm the thresholds and show the corrected method markedly improves reliability in real conditions.
Catalog record
| CREATOR | Junpeng Gong, Chunkai Wang, Hao Li, Jinyong Ma, Haoxuan Li, Xu He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-23666
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Normality: Reliable A/B Testing with Non-Gaussian Data DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Mathematics of computing Hypothesis testing and confidence interval computation Junpeng Gong, Chunkai Wang, Hao Li, Jinyong Ma, Haoxuan Li, Xu He Abstract . A/B testing has become the cornerstone of decision-making in online markets, guiding how platforms launch new features, optimize pricing strategies, and improve user experience. In practice, we typically employ the pairwise t t -test to compare outcomes between the treatment and control groups, thereby assessing the effectiveness of a given strategy. To be trustworthy, these experiments must keep Type I error (i.e., false positive rate) under control; otherwise, we may launch harmful strategies. However, in real-world applications, we find that A/B testing often fails to deliver reliable results. When the data distribution departs from normality or when the treatment and control groups differ in sample size, the commonly used pairwise t t -test is no longer trustworthy. In this paper, we quantify how skewed, long tailed data and unequal allocation distort error rates and derive explicit formulas for the minimum sample size required for the t t -test to remain valid. We find that many online feedback metrics require hundreds of millions samples to ensure reliable A/B testing. Thus we introduce an Edgeworth-based correction that provides more accurate p p -values when the available sample size is limited. Offline experiments on a leading A/B testing platform corroborate the practical value of our theoretical minimum sample size thresholds and demonstrate that the corrected method substantially improves the reliability of A/B testing in real-world conditions. 1. Introduction Online controlled experiments, or A/B testing, are the gold standard for evaluating new product ideas ( Xu et al. 2018 ; Xie and Aurisset 2016 ) , ranking algorithms ( McMahan et al. 2013 ) , pricing policies, and recommendation strategies in online markets ( Kohavi et al. 2020 ; Liu et al. 2021 ) . They have been widely adopted by leading technology companies, including Google, Facebook, Microsoft, LinkedIn, and ByteDance ( Deng and Shi 2016 ; Xu et al. 2015 ; Xie and Aurisset 2016 ) . The validity of A/B testing depends on an accurate p -value, the probability of observing a result as or more extreme than the current one under the null hypothesis ( Lehmann and Romano 2005 ) . In online experimentation, decisions rely on comparing the p -value with a predefined significance level, making its accuracy essential for trustworthy inference and reliable decision making in online markets ( Johari et al. 2017 ) . The two sample t -test ( Welch 1947 ) is widely used in A/B testing to compare the effects of a treatment and a control strategy and to support decision-making. First, the experiment platform collects user-level feedback and computes a t-statistic that summarizes the difference between the treatment and control groups. Second, it evaluates how likely it would be to observe a result at least as extreme as the one seen if the null hypothesis (i.e., no difference between two groups) were true. Then, we reject the null when the reported p -value is at most \alpha , which is set in advance. If the estimated mean for treatment exceeds the mean for control, the effect is interpreted as positive and the platform may launch the new strategy; otherwise, the strategy is not launched. In practice, the reliability of the pairwise t -test heavily depends on the data distribution. Specifically, the results are valid only if the distribution is asymptotically normal ( Hall and Wang 2004 ) . Otherwise, we will obtain an incorrect false positive probability (Type I error rate) that departs from the ideal ones ( Kohavi and Longbotham 2011 ) . For a two-sided test, the Type I error rate should be symmetrically split between the two tails, with each tail accounting for approximately half of the nominal significance level \alpha . When the right-tail Type I error, that is, the probability of falsely approving a positive effect, exceeds half of the nominal level, the platform becomes overly permissive, increasing the risk of launching ineffective or even harmful strategies and degrading user experience. Conversely, when the right-tail Type I error is below half of the nominal level, promising strategies may be prematurely discarded. To confirm the presence of non-normality, we conduct a motivating experiment using real-world data. Specifically, we repeatedly draw independent and identically distributed from the same dataset to form the control and treatment groups, thereby ensuring that the null hypothesis holds. We then compute the test statistic T over 10^{6} replications to obtain its empirical distribution. As shown in Figure 1 , the empirical distribution of the test statistic T is asymmetric and deviates from normality. The right-tail rejection probability (orange) exceeds its nominal \alpha/2 , while the left-tail rejection probability (blue) falls below it. In such cases, the test becomes systematically biased, more likely to approve ineffective strategies, and thus leads to wasted resources and suboptimal decisions. Figure 1 . Empirical density of the Welch’s t -test statistic T (blue), based on real data on the number of videos published by users of a popular online platform, with significance level \alpha=0.05 . The control and treatment sample sizes are 100 and 1,000, and the empirical distribution is compared with the standard normal N(0,1) density (orange). Such deviations from normality are also observed in commonly used online metrics, including click-through rate, the number of videos published by users, and engagement duration. The deviation mainly arises because these metrics are dominated by a large majority of low-activity users and a small minority with extremely high values. To examine the convergence behavior of the t -test under such skewed conditions, Boos and Hughes-Oliver 2000 conducted a systematic investigation to identify the minimum sample size required for accurate inference. Through extensive simulations with common skewed distributions such as Gamma and Weibull, they quantified how the empirical Type I error rate deviates from the nominal level across different degrees of skewness. They established that the deviation in each tail is approximately proportional to the skewness divided by \sqrt{n} , where n denotes the sample size. Finally, they derived a formula for the minimum sample size that maintains the Type I error deviation within a prespecified tolerance. Nevertheless, the formulas were obtained empirically through simulation and regression, and thus lack general validity across diverse data distributions Boos and Hughes-Oliver 2000 . More importantly, online experiments introduce an additional factor that strongly influences the error rate, the imbalance in sample sizes between the treatment and control groups, i.e., unequal allocation. Unequal allocation is common for two main reasons. Firstly, in large scale A/B testing, platforms often assign fewer users to new or high risk strategies to minimize potential revenue loss and avoid negative user experiences. Second, local regulations or operational policies may cap the fraction of users exposed to treatment, leading to unequal allocation. These imbalanced allocations can further distort the actual Type I error rate ( Oldfield 2016 ) . As a result, the empirical rules cannot accurately capture how the required sample size depends on the allocation ratio