Decision Quality Evaluation Framework at Pinterest DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea CCS: Applied computing Document management and text processing Document filtering CCS: Information systems Data ma

Yuqi Tian, Robert Paine, Attila Dobi, Kevin O’Sullivan, Aravindh Manickavasagam, Faisal Farooq · 2026

Read Decision Quality Evaluation Framework at Pinterest DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea CCS: Applied computing Document management and text processing Document filtering CCS: Information systems Data ma by Yuqi Tian, Robert Paine, Attila Dobi, Kevin O’Sullivan, Aravindh Manickavasagam, Faisal Farooq (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15809.txt.

Description

Tian and colleagues present a Decision Quality Evaluation Framework built and deployed at Pinterest for content safety at scale. Online platforms need robust systems to enforce content safety policies. A key part is judging review calls made by human agents and large language models. That judging is hard due to trade-offs between cost, scale, and trust, plus shifting policy tangle.

The frame centres on a high-trust Golden Set curated by subject matter experts, serving as a ground truth benchmark. An auto smart sampling pipe uses tilt scores to expand data set reach fast. Uses include testing cost-performance trade-offs of various language model agents, data-driven prompt tuning, handling complex policy change, and guarding soundness of policy content rate metrics via ongoing checks. The frame enables a shift from gut assessments to a data-driven and counted practice for managing content safety systems.

Catalog record

CREATORYuqi Tian, Robert Paine, Attila Dobi, Kevin O’Sullivan, Aravindh Manickavasagam, Faisal Farooq
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-15809

Browse the full library · About the Tower of Babel