SpecEval: Evaluating Model Adherence to Behavior Specifications
Ahmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, Percy Liang · 2025
Read SpecEval: Evaluating Model Adherence to Behavior Specifications by Ahmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, Percy Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02464.txt.
Description
Ahmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, and Percy Liang present a 2025 AI governance paper asking whether foundation models obey the behaviour rules their makers publish. Companies pledge safety limits and personality traits in detailed specification documents, but adherence had never been audited systematically.
Their SpecEval framework parses those documents into testable statements, generates targeted prompts, and uses models as judges of the answers. The key check is three-way consistency among the written specification, model outputs, and the provider's own judge models. Across sixteen models from six developers and over a hundred statements, they find systematic gaps, with compliance shortfalls reaching about a fifth in places. The audit sets a baseline expectation for stated model behaviour.
Catalog record
| CREATOR | Ahmed Ahmed, Kevin Klyman, Yi Zeng, Sanmi Koyejo, Percy Liang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 393 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02464