Evaluating Language Model Reasoning about Confidential Information

Dylan Sam Alexander Robey Andy Zou Matt Fredrikson J. Zico Kolter · 2025

Read Evaluating Language Model Reasoning about Confidential Information by Dylan Sam Alexander Robey Andy Zou Matt Fredrikson J. Zico Kolter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19980.txt.

Description

Dylan Sam and co-authors tested in 2025 whether language models can reason about confidential information. As models act as agents in high-stakes settings, they must follow user-set access rules. The team built a benchmark where a model must judge if a request is authorised by a correct password.

Current open and closed models struggled with this seemingly simple task. Stronger reasoning did not generally help, and reasoning traces often leaked the secret itself, raising doubts about exposing them. Tests grew harder with jailbreak pressure and longer multi-turn chats. The authors conclude that frontier models are not yet suited to guarding confidential information.

Catalog record

CREATORDylan Sam Alexander Robey Andy Zou Matt Fredrikson J. Zico Kolter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-19980

Browse the full library · About the Tower of Babel