AcademiClaw: When Students Set Challenges for AI Agents

2026

Read AcademiClaw: When Students Set Challenges for AI Agents (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02661.txt.

Introduction

Junjie Yu and seventy-seven co-authors introduce AcademiClaw, a benchmark of tasks that university students set for AI agents. Prior benchmarks in this agent ecosystem covered assistant-level work only. AcademiClaw instead draws eighty complex, long-horizon tasks from real student workflows: homework, research projects, competitions, and personal projects that current agents failed to solve well.

The tasks were picked from hundreds of student submissions by expert review and span more than twenty-five fields, from olympiad-level math and linguistics to heavy reinforcement learning and full system debugging. Each runs in an isolated sandbox and is scored by multi-part rubrics, with a separate safety audit. Six frontier models were tested, and even the best passed only a little over half the tasks. Sharp gaps by field and differing model strategies round out the findings.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.02661

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AcademiClaw: When Students Set Challenges for AI Agents Abstract Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We introduce AcademiClaw , a bilingual benchmark of 80 complex, long-horizon tasks sourced directly from university students’ real academic workflows—homework, research projects, competitions, and personal projects—that they found current AI agents unable to solve effectively. Curated from 230 student-submitted candidates through rigorous expert review, the final task set spans 25+ professional domains, ranging from olympiad-level mathematics and linguistics problems to GPU-intensive reinforcement learning and full-stack system debugging, with 16 tasks requiring CUDA GPU execution. Each task executes in an isolated Docker sandbox and is scored on task completion by multi-dimensional rubrics combining six complementary techniques, with an independent five-category safety audit providing additional behavioral analysis. Experiments on six frontier models show that even the best achieves only a 55% pass rate. Further analysis uncovers sharp capability boundaries across task domains, divergent behavioral strategies among models, and a disconnect between token consumption and output quality, providing fine-grained diagnostic signals beyond what aggregate metrics reveal. We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community, driving progress toward agents that are more capable and versatile across the full breadth of real-world academic demands. All data and code are available at https://github.com/GAIR-NLP/AcademiClaw . Junjie Yu † * 1,3 Pengrui Lu † * 1,2,3 Weiye Si † * 1,3 Hongliang Lu * 1 Jiabao Wu * 1 Kaiwen Tao * 1 Kun Wang * 1 Lingyu Yang * 1 Qiran Zhang * 1 Xiuting Guo * 1 Xuanyu Wang * 1 Yang Wang * 1 Yanjie Wang * 1 Yi Yang * 1 Zijian Hu * 1 Ziyi Yang * 1 Zonghan Zhou * 1 Binghao Qiang 1 Borui Zhang 1 Chenning Li 1 Enchang Zhang 1 Feifan Chen 1 Feng Jian 1 Fengyin Sun 1 Hao Qiu 1 Hao Zheng 1 Haoran Zhu 1 Hongyu Liu 1 Jianbin Deng 1 Jiaxin Song 1 Jiaying Chi 1 Jiayou Shi 1 Jie Fang 1 Jinghui Zhong 1 Jingyu Zhou 1 Jinze Li 1 Junfeng Yi 1 Junyan Yu 1 Junzhi Xue 1 Ni Song 1 Pengyi Chen 1 Qi Chen 1 Quansheng Li 1 Rui Tao 1 Shenghai Gong 1 Shenhang Lu 1 Tianqi Shen 1 Tianxiang Zhu 1 Tiehan Kang 1 Tingyu Li 1 Wendi Wu 1 Xiao Shen 1 Xiao Zhou 1 Xiaotao Zhang 1 Xinrong Li 1 Xuankun Yang 1 Xun Zhang 1 Yan Li 1 Ye Lu 1 Yi Wang 1 Yibo Zhou 1 Yichi Zhang 1 Yihao Sun 1 Yijun Huang 1 Yixin Zhu 1 Yixuan Wu 1 Yuchen Sun 1 Yue Wu 1 Yuheng Sun 1 Yukun Li 1 Yutian Tu 1 Yuxuan Qin 1 Yuzhuo Wu 1 Zeyu Li 1 Zhengyu Lou 1 Zhenning Ran 1 Zizhu He 1 Pengfei Liu † ‡ 1,2,3 1 Shanghai Jiao Tong University 2 SII 3 GAIR 0 0 footnotetext: † Project Lead * Core Contribution ‡ Corresponding Author 1 Introduction The emergence of large language model (LLM) based autonomous agents has transformed software development, data analysis, and complex workflow automation. Commercial systems such as Claude Code ( Anthropic 2025 ) and Codex ( OpenAI 2025 ) popularized this paradigm by equipping LLMs with tool-use capabilities ( Schick et al. 2023 ) —executing shell commands, editing files, searching codebases, and browsing the web—building on the ReAct framework that interleaves reasoning with action ( Yao et al. 2023 ) . More recently, OpenClaw ( OpenClaw Community 2026 ) has emerged as the most widely adopted open-source agent framework, attracting a rapidly growing developer community through its extensible tool system and support for arbitrary LLM backends. As adoption accelerates across both commercial and open-source ecosystems, rigorous evaluation of what agents can and cannot do becomes both urgent and consequential. Existing agent benchmarks have made significant progress along individual capability axes—SWE-bench ( Jimenez et al. 2024 ) grounds evaluation in real GitHub issues, WebArena ( Zhou et al. 2023 ) tests web navigation in realistic browser environments—and the growing OpenClaw ecosystem has catalyzed a wave of companion benchmarks targeting diverse evaluation criteria (§ 2 ). However, as Figure 1 illustrates, benchmarks within the OpenClaw ecosystem have thus far focused exclusively on assistant-level tasks—email triage, calendar management, project scaffolding from templates—operations that, while practically useful, require neither deep domain expertise nor sustained multi-step reasoning. This narrow evaluation scope has reinforced a prevailing perception of OpenClaw as primarily an assistant-level tool, leaving a critical gap: no existing OpenClaw benchmark systematically evaluates agents on the kind of complex, knowledge-intensive work that constitutes much of real academic and professional practice—mathematical proofs, GPU-intensive model training, cross-framework debugging, or scientific data analysis requiring domain-specific judgment. Addressing this gap with a rigorous academic-level benchmark is essential for revealing where OpenClaw agents truly fall short on harder, domain-intensive problems and for providing the open-source community with actionable diagnostic signals to advance the framework beyond its current assistant-oriented scope toward a more comprehensive and versatile agent. Figure 1: Task complexity comparison: Claw-Eval vs. AcademiClaw. Claw-Eval focuses on assistant-level routines, whereas AcademiClaw targets tasks requiring deep academic expertise and sustained multi-step reasoning. We introduce AcademiClaw , a bilingual benchmark of 80 complex, long-horizon tasks designed to bridge this gap. Rather than having researchers or annotators design tasks top-down, we adopt a bottom-up collection strategy: undergraduate students contribute problems from their real academic workflows—course assignments, research projects, competitions, and personal projects—that they found current AI agents unable to solve effectively (§ 3.1 ). This user-sourced methodology yields naturally calibrated difficulty at the frontier of AI capability, spanning 25+ professional domains from CMO mathematical proofs and IOL linguistics olympiad problems to GPU-intensive reinforcement learning and literary knowledge extraction. Each task executes in an isolated Docker container and is scored by a multi-dimensional rubric combining six complementary verification techniques—deterministic checks, code execution, LLM-as-judge, vision LLM assessment, end-to-end browser testing, and structured-output validation—enabling fine-grained diagnosis of agent capabilities beyond binary pass/fail (§ 3.4 ). We evaluate six frontier models—Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Qwen3.5-397B, and MiniMax M2.7—under identical conditions via the OpenClaw agent framework (§ 4.2 ). Even the best-performing model achieves only a 55% pass rate (score \geq 75 out of 100), confirming that academic-level tasks pose a substantial challenge to current frontier agents. Beyond aggregate performance, our analysis reveals phenomena that lighter-weight benchmarks leave hidden: over 22% of tasks exhibit capability boundaries where scores swing by up to 90 points across models on the same task; agents handle generative tasks well but struggle systematically with formal reasoning, with olympiad-level problems remaining universally unsolved; and token consumption varies by over 5\times across models yet shows near-zero correlation with quality ( r=-0.03 ), indicating that reasoning depth rather than computational effort drives performance. We further identify three distinct behavioral phenotypes —read-first, execute-first, and minimalist—that differ markedly in efficiency and safety profiles (§ 4.4 ). In summary, our main contributions are as follows: • We construct AcademiClaw, the first academic-level benchmark within the OpenClaw ecosystem, comprising 80 bilingual tasks sourced directly from university students’ real academic workflows across 25+ domains, including 16 GPU-intensive tasks absent from

Read the complete text