Surrogate-Based Prevalence Measurement for Large-Scale A/B Testing DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea

Zehao Xu, Tony Paek, Kevin O’Sullivan, Attila Dobi · 2026

Read Surrogate-Based Prevalence Measurement for Large-Scale A/B Testing DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea by Zehao Xu, Tony Paek, Kevin O’Sullivan, Attila Dobi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-16111.txt.

Description

Zehao Xu leads a team of four researchers who scale content prevalence measurement across online experiments at Pinterest. Platforms track the share of views tied to content traits, so they can weigh trade-offs and set guardrails in A/B tests. Labeling each experiment, arm, segment, and day with a large language model gives a high-fidelity reference, but the cost is far too high at platform scale.

Their surrogate system keeps one global calibration of machine learning score buckets. It is refreshed from a recurring LLM-labeled stream, and its bucket prevalences are reused across all tests through a per-experiment metric and a dashboard focused on deltas. Since calibration draws on daily samples, it stays true to live traffic as it drifts, adding no new labeling cost. In live use it serves about a hundred experiments each day, covering far more arms than per-test labeling could.

Catalog record

CREATORZehao Xu, Tony Paek, Kevin O’Sullivan, Attila Dobi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-16111

Browse the full library · About the Tower of Babel