t t -Testing the Waters
Olivier Jeunen · 2025
Read t t -Testing the Waters by Olivier Jeunen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04793.txt.
Description
Online A/B tests guide web products, and the t-test is their workhorse for judging whether a change helped. Olivier Jeunen examines when that workhorse can be trusted, in this 2025 paper. The test rests on a limit theorem that promises near-normal behavior for large samples, yet skewed outcomes make it unclear how large is large enough.
Jeunen proposes a practical check: rerun the test on reshuffled A/A data, where no real effect exists, and see whether the p-values spread evenly, as they should when the assumptions hold. A standard test can verify that spread. He shows how the check flags setups prone to false alarms. The aim is a simple routine that makes everyday web experiments more reliable.
Catalog record
| CREATOR | Olivier Jeunen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04793