Crashing Waves vs. Rising Tides: Preliminary Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks

Matthias Mertens, Adam Kuzee, Brittany S. Harris, Harry Lyu, Wensu Li, Jonathan Rosenfeld, Meiri Anto, Martin Fleming, Neil Thompson · 2026

Read Crashing Waves vs. Rising Tides: Preliminary Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks by Matthias Mertens, Adam Kuzee, Brittany S. Harris, Harry Lyu, Wensu Li, Jonathan Rosenfeld, Meiri Anto, Martin Fleming, Neil Thompson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01363.txt.

Introduction

Mertens and colleagues characterize automation as a continuum between crashing waves, abrupt jumps across narrow tasks, and rising tides, broad improvement. From 6000 tasks drawn from the O*NET taxonomy and over 60000 evaluations by workers, they find little evidence of crashing waves. Instead, rising tides are the form of progress. Performance is high and improving rapidly. In 2024-Q2, models completed tasks taking humans 1.5 hours with 60 percent success, rising above 70 percent by 2025-Q3. If trends persist, frontier models could complete tasks at sufficient quality with 88 to 97 percent success by 2030.

Catalog record

CREATORMatthias Mertens, Adam Kuzee, Brittany S. Harris, Harry Lyu, Wensu Li, Jonathan Rosenfeld, Meiri Anto, Martin Fleming, Neil Thompson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115904
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.01363

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Crashing Waves vs. Rising Tides: Preliminary Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks ∗ *Corresponding authors: Matthias Mertens, [email protected] ; Neil Thompson, [email protected] . Address of all authors: We thank David Autor for his insightful comments. We are grateful to Annie Lin, Amelia Michael, Peter Olkhovets, and Tiffany Wang for their excellent work as research assistants. We also thank Justin Viola for excellent software engineering work. Funding for this research was provided by Open Philanthropy and a technology company. Matthias Mertens, Adam Kuzee, Brittany S. Harris, Harry Lyu, Wensu Li, Jonathan Rosenfeld, Meiri Anto, Martin Fleming, Neil Thompson Abstract We propose that AI automation is a continuum between: (i) crashing waves where AI capabilities surge abruptly over small sets of tasks, and (ii) rising tides where the increase in AI capabilities is more continuous and broad-based. We test for these effects in preliminary evidence from an ongoing evaluation of AI capabilities across over 3,000 broad-based tasks derived from the U.S. Department of Labor O*NET categorization that are text-based and thus LLM-addressable. Based on more than 17,000 evaluations by workers from these jobs, we find little evidence of crashing waves (in contrast to recent work by METR), but substantial evidence that rising tides are the primary form of AI automation. AI performance is high and improving rapidly across a wide range of tasks. We estimate that, in 2024-Q2, AI models successfully complete tasks that take humans approximately 3-4 hours with about a 50% success rate, increasing to about 65% by 2025-Q3. If recent trends in AI capability growth persist, this pace of AI improvement implies that LLMs will be able to complete most text-related tasks with success rates of, on average, 80%–95% by 2029 at a minimally sufficient quality level. Achieving near-perfect success rates at this quality level or comparable success rates at superior quality would require several additional years. These AI capability improvements would impact the economy and labor market as organizations adopt AI, which could have a substantially longer timeline. \defbibfilter appendixOnlyFilter segment=1 and not segment=0 1 Introduction Figure 1 : Crashing Waves vs Rising Tides in AI Automation (a) Crashing Waves (b) Rising Tides Notes: Diagram of the distinction between AI automation that comes as “Crashing Waves" (Panel (a)) and “Rising Tides" (Panel (b)). Recent evidence by 11 suggests that, as models improve, AI capabilities surge abruptly for tasks that previously appeared out of reach, as if a “crashing wave” suddenly reaches them (our characterization), as shown in Figure 1 (a). 1 1 1 11 study 170 research and software-engineering tasks and document rapid growth in the maximum task duration (measured as human completion time) that LLMs can solve at a 50% success rate. Subsequent work by 14 extends this analysis to additional stylized benchmarks. By contrast, we focus on non-deterministic, realistic, and representative labor-market tasks. In this paper, we contrast this crashing wave phenomenon with a "rising tide" (Figure 1 (b)), in which performance is lifted more broadly across the task space. The central difference between the two phenomena is the slope of the relationship between AI success on tasks and (log) task duration. For crashing waves, this relationship can be well described by a steep logistic curve. 2 2 2 Task duration can plausibly relate to the serial dependence of tasks: longer tasks may require completing more coupled sequential sub-steps. We formalize this interpretation in Section 4.2 . For related discussions, see, for instance, 4 . AI progress is then a rightward shift of the curve, which translates into large, concentrated automation for tasks near the tipping point due to sudden improvements in capabilities of what systems can do. In practice, this would lead to harsh surprises for human workers. Over just a short period of time, they would observe AI models going from nearly always failing to nearly always succeeding. By contrast, rising-tide automation has a flatter success–duration relationship, with AI performance being more similar across tasks of different lengths. This would still be represented by a logistic curve, but a much flatter one. The same amount of AI progress would then translate into more gradual automation under the rising-tide view, such that individual workers are less likely to be blindsided by AI. A rising tide could, however, still be quite disruptive if it happens quickly. The main insight of this paper is that, across a large set of realistic and representative labor-market tasks addressable by LLMs, the downward slope between task success and task duration is, on average, surprisingly flat — i.e., more consistent with a rising tide rather than a crashing wave. Our analysis draws upon a broader ongoing research effort that collects novel evaluations of LLM outputs by domain experts across more than 40 models and covering over 20,000 unique task examples ("instances") based on more than 10,000 O*NET tasks that are at least partially text-based ( 15 ). 3 3 3 GDPval from OpenAI has done something related, but for an order of magnitude fewer occupations ( 16 ). Outputs are scored by human evaluators with relevant on-the-job experience. Given the paper’s primary focus on automation, we center our analysis on success measures defined by expert evaluations indicating that the LLM output required no human intervention to be at least minimally-successful. The survey is ongoing; while we have already collected a substantial share of the data, the results reported here should be interpreted as preliminary . Leveraging thousands of task instances that range from minutes to multiple weeks of human work, we find: 1. The success-duration curve is relatively flat — consistent with a rising tide of AI automation. This pattern holds across models of different sizes, as well as different model vintages across time. Automation within particular "job families" (e.g., management or community and social service) also follows the same rising-tide pattern in most cases. There are, however, meaningful differences in the slope of the success-duration curves across job families, as one would expect given the differences in task structure (see Section 4.2 ). 2. AI performance is high. Across all LLMs in our survey, we find strong capabilities. Models can do a minimally-sufficient job without human edits on roughly half to three-quarters of potential tasks presented to them. And, for example, by 2024-Q3, frontier models were already able to achieve 50% success rates on (LLM-addressable) tasks that take humans about a day. 4 4 4 As discussed later, these findings will not translate directly to shares of job automation, because of sampling issues, ”last mile” costs ( 6 ), and other reasons. 3. AI performance is improving rapidly across a wide range of task durations. Our estimates imply that between 2024-Q2 and 2025-Q3, frontier models went from achieving a 50% success rate on 3- to 4-hour tasks to 1-week tasks, and achieving a 70% success rate on 1-minute tasks to 1-hour tasks. When estimating a linear-trend model across all periods, the implied "doubling time," or the calendar time between model releases needed for newer models to achieve the same success rate on tasks which are twice as long, equals 3.8 months and is estimated with relatively high precision. Relative to prior benchmark-based evidence ( 11 ; 14 ), these improvement rates are rapid,

Read the complete text