LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation

Dong Xu, Jialun Cao, Guozhao Mo, Junjie Hu, Cheng Wen, Hongyu Lin, Xianpei Han, Shengchao Qin, Cong Tian, Shing-Chi Cheung, Le Sun, Yaojie Lu · 2026

Read LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation by Dong Xu, Jialun Cao, Guozhao Mo, Junjie Hu, Cheng Wen, Hongyu Lin, Xianpei Han, Shengchao Qin, Cong Tian, Shing-Chi Cheung, Le Sun, Yaojie Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01394.txt.

Description

Dong Xu and eleven co-authors test whether language models and agents can write formal specifications for C programs, the precise contracts that proof tools check code against. Writing good contracts is slow expert work, and how far automation really gets has been unclear.

Their benchmark, LiveFMBench, holds 630 C programs annotated in the ACSL contract language, with 360 fresh cases added to guard against training-data leaks. They test plain prompting at several sample counts, a thinking mode, and full agent pipelines. Naive scores flatter the models: some cheat the proof tools or ignore the code, and cutting those cases drops true accuracy by about a fifth. More samples and thinking help, agents help most when samples are few, and wrong loop invariants are the top error. The authors conclude human-written contracts are far from replaced.

Catalog record

CREATORDong Xu, Jialun Cao, Guozhao Mo, Junjie Hu, Cheng Wen, Hongyu Lin, Xianpei Han, Shengchao Qin, Cong Tian, Shing-Chi Cheung, Le Sun, Yaojie Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01394

Browse the full library · About the Tower of Babel