Measuring AI Ability to Complete Long Tasks

Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn GarciaMax Hasin , Sami Jawhar , Megan Kinniment , Nate Rush , Sydney Von ArxRyan Bloom , Thomas Broadley , Haoxing Du , Brian Goodrich , Nikola Jurkovic , Luke Harold Miles , Seraphina Nix , Tao Lin , Chris Painter , Neev Parikh , David Rein , Lu · 2025

Read Measuring AI Ability to Complete Long Tasks by Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn GarciaMax Hasin , Sami Jawhar , Megan Kinniment , Nate Rush , Sydney Von ArxRyan Bloom , Thomas Broadley , Haoxing Du , Brian Goodrich , Nikola Jurkovic , Luke Harold Miles , Seraphina Nix , Tao Lin , Chris Painter , Neev Parikh , David Rein , Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14499.txt.

Introduction

Thomas Kwa and a large team measure AI progress in human units: how long a task can be before today's models fail half the time, in a paper from 2025. They timed experts on research-engineering tasks and shorter jobs, then ran frontier models on the same work. The yardstick is the 50 percent completion time horizon.

For a leading 2025 model, that horizon sat near fifty minutes. Across models since 2019 it has doubled about every seven months, perhaps faster lately. Gains came mostly from reliability, recovery from mistakes, and better tool use. Extrapolating, the authors warn, puts month-long software tasks within AI reach in about five years. They discuss the limits of the measure and its safety implications plainly.

Catalog record

CREATORThomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn GarciaMax Hasin , Sami Jawhar , Megan Kinniment , Nate Rush , Sydney Von ArxRyan Bloom , Thomas Broadley , Haoxing Du , Brian Goodrich , Nikola Jurkovic , Luke Harold Miles , Seraphina Nix , Tao Lin , Chris Painter , Neev Parikh , David Rein , Lu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14499

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring AI Ability to Complete Long Tasks Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn GarciaMax Hasin , Sami Jawhar , Megan Kinniment , Nate Rush , Sydney Von ArxRyan Bloom , Thomas Broadley , Haoxing Du , Brian Goodrich , Nikola Jurkovic , Luke Harold Miles , Seraphina Nix , Tao Lin , Chris Painter , Neev Parikh , David Rein , Lucas Jun Koba Sato , Hjalmar Wijk , Daniel M. ZieglerElizabeth Barnes , Lawrence Chan [2ex] Model Evaluation & Threat Research (METR) Abstract Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear. To quantify the capabilities of AI systems in terms of human capabilities, we propose a new metric: 50%-task-completion time horizon . This is the time humans typically take to complete tasks that AI models can complete with 50% success rate. We first timed humans with relevant domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024. The increase in AI models’ time horizons seems to be primarily driven by greater reliability and ability to adapt to mistakes, combined with better logical reasoning and tool use capabilities. We discuss the limitations of our results—including their degree of external validity—and the implications of increased autonomy for dangerous capabilities. If these results generalize to real-world software tasks, extrapolation of this trend predicts that within 5 years, AI systems will be capable of automating many software tasks that currently take humans a month. 1 Introduction Figure 1: The length of tasks (measured by how long they take human professionals) that generalist autonomous frontier model agents can complete with 50% reliability has been doubling approximately every 7 months for the last 6 years (Section 4 ). The shaded region represents 95% CI calculated by hierarchical bootstrap over task families, tasks, and task attempts. Even if the absolute measurements are off by a factor of 10, the trend predicts that in under a decade we will see AI agents that can independently complete a large fraction of software tasks that currently take humans days or weeks (Section 7 ). In the last five years, frontier AI systems have undergone a dramatic transformation in capabilities, evolving from basic text generation [ 1 ] to autonomously executing complex multi-hour machine learning research projects [ 2 ] . Sufficiently capable AIs could perform dangerous, highly complex actions like autonomous development of chemical, biological, radiological or nuclear weapons (CBRN) and self-replication and adaptation outside human control [ 3 ] . Understanding AI capabilities helps inform the development of safety guardrails as systems become increasingly powerful. In particular, many frontier AI developers have committed to using measures of specific AI capabilities to determine the necessary risk mitigations for their frontier AI systems. 1 1 1 An updated list of frontier AI safety policies can be found at: https://metr.org/faisc . Robust benchmarks that can accurately track and forecast AI capabilities thus form the foundation for responsible AI governance and risk mitigation. However, existing benchmarks face several key limitations. First, they often consist of artificial rather than economically valuable tasks. Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans, 2 2 2 For example, HellaSwag [ 4 ] and Humanity’s Last Exam [ 5 ] were both generated by adversarially filtering problems against the best performing language models available at the time. biasing the comparison to human performance. Most critically, individual benchmarks saturate increasingly quickly [ 6 ] , and we lack a more general, intuitive, and quantitative way to compare between different benchmarks, 3 3 3 As with our tasks, SWE-bench Verified [ 7 ] does come with human-estimated task completion times. We use SWE-bench Verified tasks and accompanying time estimates to validate our main result in Section 6.3 . which prevents meaningful comparison between models of vastly different capabilities (e.g., GPT-2 versus o1). As a consequence, while the last few years have seen dramatic increases in AI performance on many individual benchmarks, understanding the progress of AI capabilities in general has required estimating the qualitative difficulty of the latest benchmarks AI systems can pass. Figure 2: Our methodology for measuring AI agent time horizon. First, we create a diverse task suite of 170 tasks. Second, we have both humans and AI agents (consisting of an AI model and a scaffold) attempt these tasks, recording the time taken by successful humans and the success rate for AI agents. Third, we fit a logistic model to find the time horizon at which each AI agent has a 50% chance of success, and plot this against the release date of the model. We propose tracking AI progress over time using the task completion time horizon : the duration of tasks that models can complete at a certain success probability, providing an intuitive measure of real-world capability compared to humans. As models may not reliably complete all tasks of a given length, we operationalize this by measuring the X%-(task completion) time horizon –the length of tasks that models can complete approximately X% of the time. We prototype this methodology using three datasets designed to capture skills required for research or software engineering (Section 3.1 ), totaling 170 tasks with a wide range of difficulty: HCAST [ 8 ] , RE-Bench [ 2 ] , and Software Atomic Actions (SWAA), a new suite of shorter software tasks that can measure pre-2023 models (Section 3.1.3 ). Using skilled human baseliners, we estimate the duration that a domain knowledgeable human (without task-specific context) takes to complete these tasks (Section 3.2 ). We evaluate the performance of 13 frontier models from 2019 to 2025 on these tasks (Section 3.3 ). Using methodology inspired by human psychometric studies, we then estimate the duration of tasks that models can complete with 50% success rate—the 50% time horizon (Section 4.1 ). We find that the 50% time horizon has been growing exponentially from 2019–2025 on our tasks, with a doubling time of approximately seven months (Figure 1 ). We compare our main result with our exploratory results on non-SWAA tasks, finding that the 2023–2025 horizon growth rate is consistent with the 2019–2025 rate to within 11% (Section 6.1 ). We also measure the 80% time horizon of models (Figure 6 ) and find a similar trend, though horizons are roughly 5x shorter. This progress appears to be driven by several key factors: improved logical reasoning capabilities, better tool use capabilities, and greater reliability and self-awareness in task execution (Section 5 ). 4 4 4 Code to reproduce our figures can be found at: https://github.com/METR/eval-analysis-public We also note several limitations of current systems—notably, performance is much lower on less structured, “messier” tasks (Section 6.2 ). Since our tasks do not perfectly represent the average segment of intellectual labor by researchers and software engineers, this raises the question of external validity (Section 6 ): whether the exponential trend holds on real-world tasks. We include results from three supplementary external validity

Read the complete text