Surrogate-Based Prevalence Measurement for Large-Scale A/B Testing DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea
Zehao Xu, Tony Paek, Kevin O’Sullivan, Attila Dobi · 2026
Read Surrogate-Based Prevalence Measurement for Large-Scale A/B Testing DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea by Zehao Xu, Tony Paek, Kevin O’Sullivan, Attila Dobi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-16111.txt.
Description
Zehao Xu leads a team of four researchers who scale content prevalence measurement across online experiments at Pinterest. Platforms track the share of views tied to content traits, so they can weigh trade-offs and set guardrails in A/B tests. Labeling each experiment, arm, segment, and day with a large language model gives a high-fidelity reference, but the cost is far too high at platform scale.
Their surrogate system keeps one global calibration of machine learning score buckets. It is refreshed from a recurring LLM-labeled stream, and its bucket prevalences are reused across all tests through a per-experiment metric and a dashboard focused on deltas. Since calibration draws on daily samples, it stays true to live traffic as it drifts, adding no new labeling cost. In live use it serves about a hundred experiments each day, covering far more arms than per-test labeling could.
Catalog record
| CREATOR | Zehao Xu, Tony Paek, Kevin O’Sullivan, Attila Dobi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-16111