SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints
Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, Chi Wang, William Yang Wang, Xifeng Yan · 2025
Read SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints by Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, Chi Wang, William Yang Wang, Xifeng Yan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08669.txt.
Description
Zekun Li and co-authors built SOPBench, a test of whether AI agents follow written procedures when they act. Agents increasingly handle customer service work where each service has standard operating procedures and hard constraints. The benchmark covers seven service domains with 167 tools and over 900 test cases, all generated automatically. Each procedure's code becomes a graph of steps, and the same code doubles as judge, checking rule-following without human grading.
Eighteen leading models were tested. Even the best struggled: strong general models passed only thirty to fifty percent of cases, and small models did far worse. One reasoning-focused model clearly led the field. Worryingly, agents could be jailbroken into ignoring the procedures. The authors release the code, data, and over 24,000 recorded agent runs. The message is that following the rulebook is a separate skill from sounding smart, and current agents lack it.
Catalog record
| CREATOR | Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, Chi Wang, William Yang Wang, Xifeng Yan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 152 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08669