RealClawBench : Live OpenClaw Benchmarks from Real Developer-Agent Sessions

Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao · 2026

Read RealClawBench : Live OpenClaw Benchmarks from Real Developer-Agent Sessions by Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03889.txt.

Description

Lv and co-authors build RealClawBench, a live benchmark from real developer-agent sessions. Agent benchmarks should mirror what users actually ask deployed agents to do. Real sessions are hard to score, though. They depend on local environments and carry implicit, underspecified intent. Verification is nontrivial as well. Two mechanisms convert real sessions into scored tasks.

Reconstructed execution environments restore the original working setting. Deterministic scorers then grade outcomes without human judgment. The release holds 281 executable tasks sampled from a far larger pool. The source distribution is preserved closely. Fourteen current models are evaluated on the set. The best system solves only about 66 percent of tasks. Large headroom remains on realistic developer workloads. Benchmark designers get a path from deployed use to controlled evaluation.

Catalog record

CREATORZongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03889

Browse the full library · About the Tower of Babel