SpecEval: Evaluating Model Adherence to Behavior Specifications

Ahmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, Percy Liang · 2025

Read SpecEval: Evaluating Model Adherence to Behavior Specifications by Ahmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, Percy Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02464.txt.

Description

Ahmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, and Percy Liang present a 2025 AI governance paper asking whether foundation models obey the behaviour rules their makers publish. Companies pledge safety limits and personality traits in detailed specification documents, but adherence had never been audited systematically.

Their SpecEval framework parses those documents into testable statements, generates targeted prompts, and uses models as judges of the answers. The key check is three-way consistency among the written specification, model outputs, and the provider's own judge models. Across sixteen models from six developers and over a hundred statements, they find systematic gaps, with compliance shortfalls reaching about a fifth in places. The audit sets a baseline expectation for stated model behaviour.

Catalog record

CREATORAhmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, Percy Liang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE393 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02464

Browse the full library · About the Tower of Babel