The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems

Richard Ren, Arunim Agarwal, Mantas Mazeika, Cristina Menghini, Robert Vacareanu, Brad Kenstler, Mick Yang, Isabelle Barrass, Alice Gatti, Xuwang Yin, Eduardo Trevino, Matias Geralnik, Adam Khoja, Dean Lee, Summer Yue, Dan Hendrycks · 2025

Read The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems by Richard Ren, Arunim Agarwal, Mantas Mazeika, Cristina Menghini, Robert Vacareanu, Brad Kenstler, Mick Yang, Isabelle Barrass, Alice Gatti, Xuwang Yin, Eduardo Trevino, Matias Geralnik, Adam Khoja, Dean Lee, Summer Yue, Dan Hendrycks (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-03750.txt.

Description

Richard Ren and co-authors build a test that separates honesty from mere correctness in AI in a 2025 paper. A model can be wrong without lying, and can lie while knowing better. Much earlier testing blurred the two, scoring wrong answers as dishonesty.

Their benchmark, MASK, uses a large human-collected dataset designed to catch actual lying: cases where the model's stated answer conflicts with what it demonstrably holds to be true. Pressure prompts probe whether systems abandon their beliefs when pushed. With accuracy and honesty scored apart, developers can finally tell a model that needs more knowledge from one that needs more integrity.

Catalog record

CREATORRichard Ren, Arunim Agarwal, Mantas Mazeika, Cristina Menghini, Robert Vacareanu, Brad Kenstler, Mick Yang, Isabelle Barrass, Alice Gatti, Xuwang Yin, Eduardo Trevino, Matias Geralnik, Adam Khoja, Dean Lee, Summer Yue, Dan Hendrycks
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-03750

Browse the full library · About the Tower of Babel