RealClawBench : Live OpenClaw Benchmarks from Real Developer-Agent Sessions
Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao · 2026
Read RealClawBench : Live OpenClaw Benchmarks from Real Developer-Agent Sessions by Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03889.txt.
Description
Lv and co-authors build RealClawBench, a live benchmark from real developer-agent sessions. Agent benchmarks should mirror what users actually ask deployed agents to do. Real sessions are hard to score, though. They depend on local environments and carry implicit, underspecified intent. Verification is nontrivial as well. Two mechanisms convert real sessions into scored tasks.
Reconstructed execution environments restore the original working setting. Deterministic scorers then grade outcomes without human judgment. The release holds 281 executable tasks sampled from a far larger pool. The source distribution is preserved closely. Fourteen current models are evaluated on the set. The best system solves only about 66 percent of tasks. Large headroom remains on realistic developer workloads. Benchmark designers get a path from deployed use to controlled evaluation.
Catalog record
| CREATOR | Zongwei Lv, Zhewen Tan, Yaoming Li, Yilun Yao, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Weihong Lin, Tong Yang, Guangxiang Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03889