AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios
xbench.org · 2026
Read AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios by xbench.org (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20613.txt.
Introduction
The xbench team presents AgentIF-OneDay, a benchmark that tests whether AI agents follow instructions in daily-life tasks. Agent capability on long, complex work keeps growing, with strong results in coding, deep research, and hard problem solving. Yet general users rarely feel those gains in everyday scenarios.
The benchmark works at task level, scoring how well agents carry out ordinary instructions. Daily tasks involve messy context and small steps that lab tests skip. A shared yardstick for them could steer agent design toward usefulness that people notice in normal life, not only in demos.
Catalog record
| CREATOR | xbench.org |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20613
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios xbench.org Abstract The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow, demonstrating exceptional performance in coding, deep research, and complex problem-solving evaluations. However, in daily scenarios, the perception of these advanced AI capabilities among general users remains limited. We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks necessary to cover the daily work, life, and learning activities of a broad demographic. To address this, we propose \method , aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks. These tasks require not only solving problems through dialogue but also understanding various attachment types and delivering tangible file-based results. The benchmark is structured around three user-centric categories: Open Workflow Execution , which assesses adherence to explicit and complex workflows; Latent Instruction , which requires agents to infer implicit instructions from attachments; and Iterative Refinement , which involves modifying or expanding upon ongoing work. We employ instance-level rubrics and a refined evaluation pipeline that aligns LLM-based verification with human judgment, achieving an 80.1% agreement rate using Gemini-3-Pro. \method comprises 104 tasks covering 767 scoring points. We benchmarked four leading general AI agents and found that agent products built based on APIs and ChatGPT agents based on agent RL remain in the first tier simultaneously. Leading LLM APIs and open-source models have internalized agentic capabilities, enabling AI application teams to develop cutting-edge Agent products. \xbenchdata [Code] github.com/xbench-ai \xbenchdata [Data] huggingface.co/xbench-ai \xbenchdata [Correspondence][email protected] † † * Equal contribution. 1 Introduction AI agents driven by Large Language Models (LLMs) have recently exhibited remarkable progress in autonomous reasoning, planning, and tool usage. Although agents have achieved success in handling increasingly complex tasks within vertical domains like vibe coding ( Jimenez et al. 2023 ; Zan et al. 2025 ) , deep research ( Wei et al. 2025 ; Chen et al. 2025 ) and complex problem solving ( Phan et al. 2025 ) , these advancements are not yet fully palpable to ordinary users. We posit that current evaluations lack the necessary diversity to reflect real-world scenarios across work, life, and learning. Addressing this gap, we propose \method , a framework to assess the efficacy of AI agents in assisting general users with daily tasks through language instructions. \method demands the delivery, analysis, and modification of comprehensive work outputs, structured around three key dimensions: 1. Open Workflow Execution: These tasks assess the agent’s ability to execute automated processes precisely and exhaustively when the user provides a clear and detailed operational procedure. This tests the agent’s robustness in long-context processing, avoidance of "instruction forgetting," and suppression of hallucinations. 2. Latent Instruction Inference: This dimension requires the agent to autonomously induce and infer implicit, unstated general rules or constraints from task attachments and apply them to new tasks. 3. Iterative Refinement: These tasks simulate a multi-turn, progressive collaborative process between the user and the agent. The agent must perform precise content updates and adjustments based on supplementary or corrective instructions from the user upon existing outputs, measuring the efficacy of its state maintenance and human-machine collaboration. We control the distribution of topics in the evaluation set to balance alignment with authentic user needs and maximization of task diversity . The former includes common demands such as daily office software usage, life planning, and research, helping us understand the correlation between agent capabilities and real-world value. The latter primarily increases the difficulty of "hack" evaluation metrics through out-of-domain tasks, including game strategy research and the automation of scientific research workflows. Beyond manually annotated tasks, we designed a File-centered Automated Agentic Pipeline for task generation. We utilize the ChatGPT agent to collect information-dense attachments with high potential for question generation. Using human-annotated tasks as seeds, we extract and augment workflow frameworks. Finally, we synthesize the attachments and workflows to produce \method evaluation items that are logically coherent and deeply correlated with the attachment information. Furthermore, this methodology is extensible to broader agentic data synthesis applications. We employ Instance-level rubrics for evaluation and have enhanced the consistency between LLM-based Verifiers and human scoring through pipeline design. For instance, we implemented visual parsing for PPT and HTML files and utilized Vision-Language Models (VLMs) as verifiers. Additionally, for rubrics requiring real-time verification, the model defaults to a Search Mode. The utilization of superior multimodal foundation models has significantly improved the accuracy of the discriminator. When employing Gemini-3-Pro as the judge model, the consistency between \method scoring points and human scoring reaches 80.6%. \method comprises 104 tasks and covers 767 scoring points. Through systematic testing and analysis of mainstream AI agent products on this benchmark, we aim to reveal the strengths and bottlenecks of current agent products and agentic models, providing empirically valuable references for the future optimization of AI agents. Simultaneously, the high-quality, finely annotated instruction data accumulated by \method holds significant potential as training data for reinforcement learning. 2 Related Work Instruction Following Benchmarks. Instruction following benchmarks are crucial for evaluating the ability of large language models to execute tasks based on specific instructions. For instance, IFEval ( Zhou et al. 2023 ) assesses a model’s capability using a set of verifiable instructions to ensure accuracy. Another notable benchmark, FOFO ( Xia et al. 2024 ) , classifies tasks into various domains and imposes specific format requirements, such as the Prescription Format Standard, to test a model’s precision. To further push the boundaries of evaluation, ComplextBench ( Wen et al. 2024 ) generates instructions through a compositional approach, which is specifically designed to enhance the assessment of a model’s understanding of more complex commands. Agent Benchmarks. Agent benchmarks are designed to evaluate the ability of AI agents to perform complex, real-world tasks that often require a combination of skills ( Liu et al. 2023 ; Hu et al. 2024 ) . GAIA ( Mialon et al. 2023 ) , for example, assesses an agent’s reasoning and tool-using capabilities by having it complete a variety of real-world tasks. Similarly, VisualWebArena ( Koh et al. 2024 ) focuses on an agent’s ability to execute tasks specifically within a web environment, which is a common and important domain. Additionally, tau-Bench ( Yao et al. 2024 ) provides another dedicated framework for evaluating a model’s capacity for effective tool usage. Evaluation on Real-World Economic Tasks. Recent research has shifted focus from general abilities to evaluating AI’s potential for economic labor automation. Handa et al. 2025 analyzed millions of user conversations, revealing that AI usage is predominantly concentrated in software development and writing tasks, with 57% of interactions focusing on augmenting human capabilities rather than full automation. Similarly, Chatterji et al. 2025 examined consumer usage patterns of ChatGPT, finding that while non-work interactions are growing, the model provides significant economic value as a decision-support tool for knowledge-intensive tasks. To rigorously assess professional capabilities, Patwardhan et al. 2025 introduced GDPval, a benchmark evaluating AI on long-horizon, multi-modal tasks across 44 occupations. Complementing this, Mazeika et al. 2025 proposed the Remote Labor Index (RLI) based on real-world freelance projects, finding that current frontier agents achieve only a 2.5% automation