The Bias-Variance Tradeoff in Long-Term Experimentation

Daniel Ting, Kenneth Hung · 2025

Read The Bias-Variance Tradeoff in Long-Term Experimentation by Daniel Ting, Kenneth Hung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02792.txt.

Introduction

Daniel Ting and Kenneth Hung study a tension in online experiments: trimming noise makes estimates sharper but can add bias. Methods like winsorization and surrogate metrics do exactly that. Their 2025 paper asks whether such bias is safe when experiments run for years and small errors can pile up over time.

Their model shows the tradeoff survives in the long run. Sharper signal still helps, even when it costs some bias. That shifts the goal from finding one correct unbiased metric to making the best estimates for each decision. The paper also refines launch rules: loose criteria help when a system is far from its best, while strict criteria win as the system matures.

Catalog record

CREATORDaniel Ting, Kenneth Hung
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02792

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Bias-Variance Tradeoff in Long-Term Experimentation Daniel Ting, Kenneth Hung Abstract 1 Introduction As we exhaust methods that reduces variance without introducing bias [ 5 ] , reducing variance in experiments often requires accepting some bias, using methods like winsorization [ 6 ] or surrogate metrics [ 2 ] . While this bias-variance tradeoff can be optimized for individual experiments, bias may accumulate over time, raising concerns for long-term optimization. We analyze whether bias is ever acceptable when it can accumulate, and show that a bias-variance tradeoff persists in long-term settings. Improving signal-to-noise remains beneficial, even if it introduces bias. This implies we should shift from thinking there is a single “correct”, unbiased metric to thinking about how to make the best estimates and decisions when better precision can be achieved at the expense of bias. Furthermore, our model adds nuance to previous findings [ 1 , 4 ] that suggest less stringent launch criterion leads to improved gains. We show while this is beneficial when the system is far from the optimum, more stringent launch criterion is preferable as the system matures. 2 Experiment-based optimization as stochastic gradient descent We study this problem by treating our long term optimization problem like a stochastic gradient descent (SGD) procedure [ 7 ] . Each experiment can be treated as a noisy measurement of the lift, so each launch is a noisy step towards or away from the global optimum. This sets up a stochastic differential equation (SDE) that we can solve. By injecting bias and variance into the procedure, we can study whether the optimality gap increases or decreases. For simplicity, we consider a problem where we optimize a metric by changing a parameter. When the parameter is x_{t} at time t , the observed metric is \mathrm{Metric}(x_{t}) . This contains some sampling noise and has expectation \EE\mathrm{Metric}(x_{t}) , an objective that we wish to optimize. Without loss of generality, we can assume \EE\mathrm{Metric}(x) attains the optimum at x=0 . Since at the optimum, the true objective has zero gradient, we can express the objective as \EE\mathrm{Metric}(x_{t})=\mathrm{optimum}-\frac{1}{2}rx_{t}^{2}, where r captures the scaling of the effect of the parameter x on the outcome. At time t , with probability \nicefrac{{1}}{{2}} we propose to change the parameter x_{t} to x_{t}-1 and with probability \nicefrac{{1}}{{2}} we propose changing it to x_{t}+1 . A change from x_{t} to x_{t}-1 will change the objective by \EE\mathrm{Metric}(x_{t}-1)-\EE\mathrm{Metric}(x_{t})\approx-\nabla\EE\mathrm{Metric}(x_{t})=rx_{t}, (1) which we can measure via an experiment. For simplicity, we assume all experiments to have standard error \sigma . The experiment is then launched if the result is statistically positive, so the probability of changing x_{t} to x_{t}-1 is \PP(Z+rx_{t}/\sigma>c)=\bar{\Phi}(c-rx_{t}/\sigma), where Z is a standard normal variable, \bar{\Phi} is its survival function (or one minus its cumulative distribution function), and c is the hypothesis testing threshold for a z -score at level \alpha , e.g. c=1.64 for 90\% confidence. Similar, a change from x_{t} to x_{t}+1 is launched with probability \bar{\Phi}(c+rx_{t}/\sigma) . In other words, given the parameter is x_{t} at time t , the parameter at time t+1 will be given by X_{t+1}=\begin{cases}x_{t}-1&\text{with probability }\frac{1}{2}\bar{\Phi}(c-rx_{t}/\sigma),\\ x_{t}+1&\text{with probability }\frac{1}{2}\bar{\Phi}(c+rx_{t}/\sigma),\\ x_{t}&\text{otherwise.}\end{cases} Consider the case where it is difficult to detect a treatment effect, and the scale of noise is bigger than the effect size, i.e. \sigma\gg|rx_{t}| . The parameter X_{t+1} will satisfy \displaystyle\EE[X_{t+1}-x_{t}] \displaystyle=-\frac{1}{2}\bar{\Phi}(c+rx_{t}/\sigma)+\frac{1}{2}\bar{\Phi}(c-rx_{t}/\sigma)\approx rx_{t}\phi(c)/\sigma, (2) \displaystyle\var(X_{t+1}-x_{t}) \displaystyle=\EE\left[(X_{t+1}-x_{t})^{2}\right]-\EE[X_{t+1}-x_{t}]^{2} \displaystyle\approx\frac{1}{2}\bar{\Phi}(c-rx_{t}/\sigma)+\frac{1}{2}\bar{\Phi}(c+rx_{t}/\sigma)-o(rx_{t}/\sigma) \displaystyle\approx\alpha. (3) After rescaling time to obtain a continuous time process, these define the drift and diffusion terms of an SDE: dX_{t}=-\frac{r\phi(c)}{\sigma}x_{t}\,dt+\sqrt{\alpha}\,dW_{t}=-\theta x_{t}\,dt+\sqrt{\alpha}\,dW_{t}, (4) where \theta=r\phi(c)/\sigma . This is a well-known SDE for the Ornstein–Uhlenbeck process [ 3 ] , i.e. a mean reverting walk. This is a particularly nice SDE cause it has a closed form solution for its distribution at every time point. The solution and limit distribution are given by X_{t}\sim N\left(x_{0}e^{-\theta t},\frac{\alpha}{2\theta}\left(1-e^{-2\theta t}\right)\right),\qquad X_{\infty}\sim N(0,\alpha/2\theta) (5) Correspondingly, the objectives at time t and as t\to\infty are \displaystyle\EE\mathrm{Metric}(X_{t}) \displaystyle=\mathrm{optimum}-\frac{1}{2}r\left(x_{0}e^{-2\theta t}+\frac{\alpha}{2\theta}\left(1-e^{-2\theta t}\right)\right) (6) \displaystyle\EE\mathrm{Metric}(X_{\infty}) \displaystyle=\mathrm{optimum}-\frac{1}{4}\frac{r\alpha}{\theta} (7) 2.1 Interpreting the SDE solution This equation already describes interesting properties of the stochastic optimization of systems through experimentation and highlight the importance of reducing the variance of lift estimates. • Longer convergence. Increasing \sigma to \sigma^{\prime} means it takes (\sigma^{\prime}/\sigma) times longer to reach the stationary distribution. In other words, you need to run (\sigma^{\prime}/\sigma) times more experiments to reach the same gains. • Suboptimal final state. A large standard deviation \sigma in the lift measurement directly reduces the expected value of the target metric. • Less stringent launch criterion yields faster convergence. While larger \sigma slows down convergence, smaller decision thresholds c>0 increase \phi(c) and \theta , which speeds up convergence as shown in Figure 1 . • More stringent launch criterion yields better final state. For the optimality gap in Equation 7 , we see that \frac{1}{4}\frac{r\alpha}{\theta}=\frac{\sigma}{4}\frac{\bar{\Phi}(c)}{\phi(c)}. The ratio \bar{\Phi}(c)/\phi(c) is the Mills’s ratio for a Gaussian distribution and is (1+o(c))(1/c) as c\to\infty . Figure 1: Distribution of the underlying state X_{n} starting at X_{0}=100 . Launches using statistical significance for a one-sided test at the 5\% level takes twice as long to converge as a one-sided test at the 40\% level. While the first point is completely unsurprising, the second and fourth suggest that as the system becomes more and more optimized, experiments need to be run with higher power and more stringent launch criterion to reach a better final state! Perhaps counterintuitively, the third and fourth points fall in line with previous suggestions to relax the p -value threshold in [ 1 , 4 ] — in a less mature system farther from the stationary state, effect sizes are bigger and a relaxed p -value threshold helps to achieve faster improvements; while in a mature system closer to the stationary state, effect sizes are smaller and a more stringent p -value threshold is preferred. 3 Impact of bias But what happens when you induce bias? For simplicity, we consider two scenarios: • Bias in the updates. The bias shrinks the experiment estimand toward zero by a factor of \gamma_{r} , in exchange for shrinking the standard error by a factor of \gamma_{\sigma} . • Bias in the objective. The metric is a biased measurement on the true estimand. In the first scenario, we should still converge to a limiting distribution with the right mean, while in the second scenario we assume the bias actually changes the mean of the limiting distribution. 3.1 Bias in the updates In the first scenario, the true effect in Equation 1 becomes rx_{t}/\gamma_{r} and the standard error becomes \sigma/\gamma_{\sigma} . The SDE ( Equation 4 ) can be updated by replacing r with r/\gamma_{r} and \theta with \frac{\gamma_{\sigma}}{\gamma_{r}}\theta . By plugging this into the solution ( Equation 5 ), we can see that this scenario affects the solutions in how fast it converges, faster convergence is achieved when \gamma_{r}<\gamma_{\sigma} . In other words, it improves all aspects of the optimization if the signal-to-noise ratio improves so that \frac{r/\gamma_{r}}{\sigma/\gamma_{\sigma}}>\frac{r}{\sigma} . We will denote the improvement in signal-to-noise ratio as \gamma , i.e. \gamma=\gamma_{\sigma}/\gamma_{r} . Furthermore, plugging this in Equations 6 and 7 , the new expected target metric is \EE\mathrm{Metric}(X_{\infty})=\mathrm{optimum}-\frac{1}{4}\frac{r\alpha}{\gamma\theta}. The key takeaway is that any improvement in the signal-to-noise ratio is beneficial, even

Read the complete text