Decision Quality Evaluation Framework at Pinterest DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea CCS: Applied computing Document management and text processing Document filtering CCS: Information systems Data ma
Yuqi Tian, Robert Paine, Attila Dobi, Kevin O’Sullivan, Aravindh Manickavasagam, Faisal Farooq · 2026
Read Decision Quality Evaluation Framework at Pinterest DOI: XXXXXXX.XXXXXXX Conference: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 9–13, 2026; Jeju, Korea CCS: Applied computing Document management and text processing Document filtering CCS: Information systems Data ma by Yuqi Tian, Robert Paine, Attila Dobi, Kevin O’Sullivan, Aravindh Manickavasagam, Faisal Farooq (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15809.txt.
Description
Tian and colleagues present a Decision Quality Evaluation Framework built and deployed at Pinterest for content safety at scale. Online platforms need robust systems to enforce content safety policies. A key part is judging review calls made by human agents and large language models. That judging is hard due to trade-offs between cost, scale, and trust, plus shifting policy tangle.
The frame centres on a high-trust Golden Set curated by subject matter experts, serving as a ground truth benchmark. An auto smart sampling pipe uses tilt scores to expand data set reach fast. Uses include testing cost-performance trade-offs of various language model agents, data-driven prompt tuning, handling complex policy change, and guarding soundness of policy content rate metrics via ongoing checks. The frame enables a shift from gut assessments to a data-driven and counted practice for managing content safety systems.
Catalog record
| CREATOR | Yuqi Tian, Robert Paine, Attila Dobi, Kevin O’Sullivan, Aravindh Manickavasagam, Faisal Farooq |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-15809