t t -Testing the Waters

Olivier Jeunen · 2025

Read t t -Testing the Waters by Olivier Jeunen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04793.txt.

Description

Online A/B tests guide web products, and the t-test is their workhorse for judging whether a change helped. Olivier Jeunen examines when that workhorse can be trusted, in this 2025 paper. The test rests on a limit theorem that promises near-normal behavior for large samples, yet skewed outcomes make it unclear how large is large enough.

Jeunen proposes a practical check: rerun the test on reshuffled A/A data, where no real effect exists, and see whether the p-values spread evenly, as they should when the assumptions hold. A standard test can verify that spread. He shows how the check flags setups prone to false alarms. The aim is a simple routine that makes everyday web experiments more reliable.

Catalog record

CREATOROlivier Jeunen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04793

Browse the full library · About the Tower of Babel