OccuBench : Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation

Xiaomeng Hu, Yinger Zhang, Fei Huang, Jianhong Tu, Yang Su, Lianghao Deng Yuxuan Liu Yantao Liu Dayiheng Liu Tsung-Yi Ho, [6pt] Qwen Team, Alibaba Group, [3pt] Equal contribution Work done during internship at Qwen Team Corresponding author · 2026

Read OccuBench : Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation by Xiaomeng Hu, Yinger Zhang, Fei Huang, Jianhong Tu, Yang Su, Lianghao Deng Yuxuan Liu Yantao Liu Dayiheng Liu Tsung-Yi Ho, [6pt] Qwen Team, Alibaba Group, [3pt] Equal contribution Work done during internship at Qwen Team Corresponding author (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10866.txt.

Introduction

Hu and colleagues offer OccuBench, a test of 100 real job tasks across 10 fields and 65 domains, run with language simulators that fake work settings. A pipeline auto-makes test cases that are solvable, graded in difficulty, and varied in texts. OccuBench scores task success and toughness under planted faults. Testing 15 top models from 8 families, they find no one model tops all fields, and hidden faults prove harder than plain errors. Bigger models and more reasoning effort raise scores. Strong agents are not always strong simulators, so simulator quality is key.

Catalog record

CREATORXiaomeng Hu, Yinger Zhang, Fei Huang, Jianhong Tu, Yang Su, Lianghao Deng Yuxuan Liu Yantao Liu Dayiheng Liu Tsung-Yi Ho, [6pt] Qwen Team, Alibaba Group, [3pt] Equal contribution Work done during internship at Qwen Team Corresponding author
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55815
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10866

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OccuBench : Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation Xiaomeng Hu, Yinger Zhang, Fei Huang, Jianhong Tu, Yang Su, Lianghao Deng Yuxuan Liu Yantao Liu Dayiheng Liu Tsung-Yi Ho, [6pt] Qwen Team, Alibaba Group, [3pt] Equal contribution Work done during internship at Qwen Team Corresponding author Abstract AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench , a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains , enabled by Language Environment Simulators (LESs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LES-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks. Project Page \cdot Code \cdot Data Correspondence: {xmhu23,tyho}@cse.cuhk.edu.hk , {zhangyinger.zye,feihu.hf,tujianhong.tjh}@alibaba-inc.com 1 Introduction AI agents are increasingly expected to perform professional work across diverse occupational domains: triaging emergency patients, auditing financial reports, scheduling factory production lines, responding to network intrusions, processing customs declarations, and coordinating wildfire evacuations. These scenarios represent the highest-value applications of AI agent technology, where autonomous decision-making through multi-step tool use can augment or replace costly human expertise. However, a fundamental evaluation gap exists: the professional domains where agents would deliver the most value are precisely the domains where no benchmarks exist . Consider the following questions that no existing benchmark can answer: • Can an agent triage patients in an emergency department? No public environment exists. • Can an agent manage a nuclear reactor safety alert? No benchmark covers this. • Can an agent process customs import declarations? No API is available. • Can an agent control greenhouse irrigation based on sensor data? No testbed exists. This is not a collection of edge cases; it is the default state for the vast majority of professional work. Existing agent benchmarks are structurally unable to address these domains due to several fundamental limitations: The Untestable Majority. The professional domains where AI agents are most needed, including healthcare, finance, legal, manufacturing, energy, governance, and logistics, which are bound to enterprise systems with no public APIs, no external access, and irreversible real-world consequences. Current benchmarks are confined to domains with available environments: WebArena ( 46 ) to web browsing, OSWorld ( 40 ) to desktop operations, SWE-bench ( 16 ) to code repositories, and TAU-bench ( 44 ) to retail and airline APIs. The result is a severe evaluation blind spot covering the vast majority of high-value professional work. Prohibitive Scaling Cost. Even within covered domains, each benchmark is constrained by its environment implementation. Adding a new domain to WebArena requires deploying and configuring entire web applications; extending TAU-bench requires integrating new real APIs or manually writing simulators. This engineering overhead makes scaling to dozens or hundreds of professional domains practically infeasible. No Robustness Evaluation. Real-world environments are noisy: APIs time out, data arrives incomplete, services degrade silently. Yet existing benchmarks evaluate agents exclusively on the “happy path,” providing no systematic assessment of how agents handle environmental faults. This gap is critical for production deployment decisions. Our Approach: Language Environment Simulator. Our key observation is that the environment itself can be simulated by an LLM . Given a configuration c=(system\_prompt,tool\_schema,initial\_state,state\_description) , an LLM becomes a stateful, interactive environment simulator, i.e., a Language Environment Simulator (LES). As long as an LLM understands the operational logic of a domain, it can simulate tool responses for that domain. This transforms environment construction from an engineering problem into a configuration problem, extending benchmark coverage from “domains with public environments” to “any domain an LLM can understand.” Based on LESs, we present OccuBench , a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains , with 382 evaluation instances . Each scenario corresponds to a real human job role (emergency triage nurse, operations engineer, customs officer, production scheduler), ensuring that evaluation results directly reflect an agent’s fitness for professional work. OccuBench evaluates agents along two complementary dimensions: 1. Task Completion : Multi-step decision-making across 10 industry categories, revealing each model’s cross-industry capability profile. 2. Environmental Robustness : Performance under controlled fault injection, including explicit errors (timeouts, 500s), implicit degradation (truncated data, missing fields), and mixed faults, quantifying an agent’s resilience to real-world environmental noise. We evaluate 15 frontier models spanning 8 model families and make the following key findings: • No single model dominates all industries : Gemini 3.1 Pro leads in Education (84%) and Science (81%) but struggles in Healthcare (62%); Claude Opus 4.6 excels in Transportation (77%) but trails in Commerce (53%). Every model has blind spots invisible to single-domain benchmarks. • Implicit faults are harder than explicit faults : Average performance under E2 (implicit, 53.4%) drops far more than E1 (explicit, 62.6%) from E0 (67.5%). Implicit faults lack overt error signals and require agents to independently detect data degradation, a capability most models lack. • Scaling consistently improves performance : Larger models outperform smaller variants within every family, newer generations outperform older ones, and higher reasoning effort yields better results. GPT-5.2 improves by 27.5 points from none to xhigh effort. • Strong agents are not necessarily strong simulators : GPT-5.2 ranks first as an agent (79.6%) but produces the worst environment simulation quality. When using a sufficiently capable simulator, pairwise ranking agreement reaches 85.7%, confirming that LES-based evaluation is reliable. 2 Related Work Agent Benchmarks. Agent benchmarks can be categorized by environment type. Web environments : WebArena ( 46 ) deploys real websites for browser-based tasks; VisualWebArena ( 18 ) and Mind2Web ( 9 ) extend to visual and cross-domain web interaction; WorkArena ( 11 ) targets enterprise knowledge work on ServiceNow; BrowseComp ( 37 ) evaluates deep web navigation. OS and mobile environments : OSWorld ( 40 ) provides full operating system virtual machines; AndroidWorld ( 34 ) benchmarks mobile app automation; MobileBench ( 8 ) evaluates LLM-based mobile agents. Code environments : SWE-bench ( 16 ) evaluates repository-level issue resolution; InterCode ( 42 ) provides interactive coding with execution feedback; Terminal-Bench ( 23 ) tests agents in real terminal environments. Tool and API environments : TAU-bench ( 44 ) evaluates tool-agent-user interaction; BFCL ( 30 ) benchmarks function calling; AgentBench ( 21 ) covers 8 distinct environments; ToolLLM (

Read the complete text