ClawMark : A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

ClawMark Team github.com/evolvent-ai/ClawMark claw-mark.com · 2026

Read ClawMark : A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents by ClawMark Team github.com/evolvent-ai/ClawMark claw-mark.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23781.txt.

Introduction

Meng and colleagues introduce ClawMark, a benchmark for language-model agents working as persistent coworkers across multiple days in a changing world. Its 100 tasks span 13 professional scenarios in a stateful sandbox where emails, calendars, and records evolve between turns, scored by deterministic checkers without any model judge. The best strict task success is only 20 percent, and performance drops after the first outside environment change.

Catalog record

CREATORClawMark Team github.com/evolvent-ai/ClawMark claw-mark.com
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50781
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23781

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ClawMark : A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents ClawMark Team github.com/evolvent-ai/ClawMark claw-mark.com Abstract Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce ClawMark , a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1,537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation. \papertags \evolventtag Benchmark \evolventtag Coworker Agents \evolventtag Multimodal 1 Introduction Language-model agents are moving from one-shot task solvers toward persistent coworkers that stay with a human across many working days. The surrounding environment keeps evolving while the agent works: new emails arrive, schedules shift, knowledge-base records are updated, and key evidence is distributed across images, scanned PDFs, audio, video, and spreadsheets. Frameworks such as OpenClaw and Claude Code already make this coworker-agent usage pattern increasingly operational. What remains under-measured is a benchmark setting that jointly evaluates agents on these properties, rather than under a single static session. Existing agent benchmarks still fall short of this setting in three important ways. First, they evaluate at a time instant rather than across a time interval : most existing benchmarks score an agent within a single session in which the environment is implicitly assumed not to change between steps. A coworker, by contrast, works across an interval where the environment can evolve between turns: a file read in step 1 may be different in step 2. Second, this assumption is reinforced by a static-environment design [ 18 , 2 , 16 ] : later state changes, when present at all, typically follow from the agent’s own actions, rather than from exogenous updates coming from outside the interaction loop. Third, inputs are text-centric : although some benchmarks have added images [ 6 , 15 ] , real office work often depends on raw multimodal evidence. The capabilities that distinguish a coworker (persistent state tracking, adaptation to external change, and multimodal evidence integration) therefore remain insufficiently measured. ClawMark is a benchmark for coworker agents targeting exactly this regime. Each task is a multi-turn workflow spanning multiple in-universe workdays (one turn per working day), executed against five stateful sandboxed services: filesystem, email, calendar, knowledge base, and spreadsheet. By “stateful sandboxed services” we mean stateful runtime services running inside the benchmark sandbox (Docker-mounted filesystem, GreenMail SMTP/IMAP, a Notion-compatible knowledge base, a Google-Sheets-compatible spreadsheet, and a Radicale CalDAV server), rather than static logs, cached snapshots, or real third-party production endpoints. Between turns the environment changes independently of the agent, through announced events (which we call loud events ) and unannounced silent mutations , and evidence is delivered untranscribed. Scoring is fully rule-based (§ 3.2 ) rather than LLM-as-judge over the agent’s prose. The current release contains 100 tasks across 13 professional scenarios. Table 1 : Structural comparison of agent benchmarks along the three axes of § 1 plus the scoring choice. Multimodal : None / Partial (images only) / Full (audio, video, scanned PDFs, images, spreadsheets; we use “Full” here for full coverage of raw non-text office artifacts rather than as a strict modality count). Multi-Day : does each task span multiple in-universe days? Environment : does external state mutate between turns independently of the agent (Dynamic) or not (Static)? Verification : how is task success determined? Among the representative benchmarks compared here, ClawMark is the only one with Multi-Day = Yes, Environment = Dynamic, and Multimodal = Full under rule-based scoring. Benchmark # Tasks # Scenarios Multimodal Multi-Day Verification Environment WebArena [ 20 ] 812 5 None No Rule-based Static OSWorld [ 15 ] 369 9 Partial No Rule-based Static Terminal-Bench [ 10 ] 89 \sim 6 None No LLM-as-judge Static MCPMark [ 14 ] 127 5 None No Rule-based Static Claw-Eval [ 19 ] 300 9 Full No Rule + LLM Static ClawsBench [ 8 ] 153 15 Partial No LLM-as-judge Static ClawMark (Ours) 100 13 Full Yes Rule-based Dynamic Our contributions are threefold: • We present ClawMark , a benchmark for coworker-agent evaluation that combines multi-turn multi-day tasks, a stateful sandboxed service environment, exogenous between-turn environment changes, and deterministic rule-based scoring in a single executable setting. • We operationalise a no-LLM-as-judge scoring protocol: 1,537 deterministic Python checkers inspect post-execution service state, and each task is admitted to the released corpus only after two independent re-runs produce bit-identical checker verdicts and diagnostic messages, giving a deterministic alternative to model-judged scoring. • We benchmark seven frontier agent systems on this setting. The top weighted score is 75.8 and the top strict Task Success is 20.0 (these are two different metrics on a 0–100 scale; see § 3.2 ), and both metrics leave room to improve. The per-scenario picture is in Figure 1 . Because tasks unfold across multiple turns and external state changes between them, models with similar aggregate scores can follow meaningfully different adaptation trajectories over time. On the 73 tasks with exactly three turns, six of the seven evaluated models drop on Day 2, while the Claude Sonnet 4.6–GPT-5.4 gap narrows from +6.5 percentage points on Day 1 to +4.0 percentage points on Day 3 (§ 6.1 ). We release the benchmark, the evaluation harness, and the construction pipeline to support reproducible evaluation of the next generation of coworker agents. Figure 1 : ClawMark results overview. Left: main leaderboard across seven frontier models under the single-run protocol (§ 5.1 ); Claude Sonnet 4.6 leads at 75.8 weighted score and the top strict Task Success is 20.0; both metrics leave room to improve. Right: distribution of the 100 tasks across the 13 professional scenarios; the benchmark covers specialised domains including legal assistance, investment analysis, and EDA that prior agent benchmarks have not reached. 2 Related work 2.1 Agent benchmarks Table 1 positions ClawMark against representative agent benchmarks. Realistic web and computer-use benchmarks such as WebArena [ 20 ] , Mind2Web [ 1 ] , VisualWebArena [ 6 ] , and OSWorld [ 15 ] establish strong single-episode evaluation settings. Related benchmarks extend tool coverage or execution domains, for example MCPMark [ 14 ] , MCP-Bench [ 12 ] , SWE-bench [ 5 ] , AgentBench [ 9 ] , GAIA [ 11 ] , and Terminal-Bench [ 10 ] , but still largely evaluate progress within a fixed episode. Multi-turn benchmarks such as tau-bench

Read the complete text