“Are We Done Yet?”: A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents

Marta Sumyk, Oleksandr Kosovan · 2025

Read “Are We Done Yet?”: A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents by Marta Sumyk, Oleksandr Kosovan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20067.txt.

Introduction

Marta Sumyk and Oleksandr Kosovan ask a simple question with a tool to answer it: are we done yet? Computer use agents can operate digital interfaces on their own, but they often cannot tell reliably whether the task they were given is finished. The authors build an evaluation and feedback framework around that gap.

Vision-language models judge task completion directly from screenshots paired with task descriptions. The supporting dataset covers dozens of built-in macOS applications and more than a thousand human-labeled tasks across many scenarios. The judge reaches solid accuracy at spotting success, and feeding its verdicts back to the agent raises overall task success by a clear margin. Vision-based checking thus works as a self-correction channel for computer use agents.

Catalog record

CREATORMarta Sumyk, Oleksandr Kosovan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

“Are We Done Yet?”: A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents Marta Sumyk, Oleksandr Kosovan Abstract Computer Use Agents (CUAs) are designed to autonomously operate digital interfaces, yet they often fail to reliably determine whether a given task has been successfully completed. We present an autonomous evaluation and feedback framework that leverages Vision–Language Models (VLMs) to assess task completion directly from screenshots and task descriptions. Our dataset covers 42 42 built-in macOS applications and 1,260 1{,}260 human-labeled tasks, covering a wide range of scenarios. Our framework achieves up to 73 % 73\% classification accuracy in task success detection and yields an average relative improvement of 27 % 27\% in the overall task success rate of CUAs when evaluator feedback is applied. These results demonstrate that vision-based evaluation can serve as an actionable feedback mechanism that significantly improves the reliability and self-correction of autonomous computer-use agents. † † footnotetext: This work has been accepted to appear at the AAAI 2026 Workshop on Trust and Control in Agentic AI (TrustAgent). Introduction In recent years, Computer Use Agents (CUAs) ( 14 ; 10 ; 8 ) have emerged as a promising paradigm for enabling AI systems to autonomously interact with digital environments, perceiving screen states and performing actions such as clicking, typing, and executing commands to accomplish user-specified goals. Despite their generality and service-agnostic design, a key limitation remains: CUAs often struggle to reliably determine whether a task has been successfully completed. This shortcoming manifests in two critical ways: • The agent declares a task complete when it is not, undermining user trust and overall reliability ( 15 ; 13 ) . • The agent successfully completes the task but fails to recognize this, leading to redundant actions and unnecessary computational overhead ( 13 ) . To address these challenges, this work proposes a method for autonomous evaluation of task completion for macOS CUAs, aimed at improving both the success rate and the reliability of CUAs. We focus on the macOS environment for two main reasons. First, it remains an underexplored domain in the study of screen representations and CUAs ( 9 ) . While prior work has primarily examined web and mobile environments ( 11 ; 6 ; 5 ) , desktop operating systems have received considerably less attention. Desktop interfaces are also inherently more complex to interpret: they typically contain a larger number of visual elements than mobile UIs and lack structured unified representations such as Hypertext Markup Language (HTML) trees available in web environments ( 9 ) . Second, we choose macOS as an ideal starting point because it offers a controlled yet diverse collection of built-in applications that enable systematic benchmarking of task execution and evaluation. In the future, we aim to extend our framework to other desktop operating systems, including Windows, Linux, and cross-platform web interfaces, toward developing general-purpose and reliable CUAs. To sum up, the contributions of this work are as follows: • We introduce a diverse, human-labeled dataset comprising 1{,}260 tasks across 42 built-in macOS applications. • We propose a methodology for autonomous evaluation of task completion, achieving up to 73\% accuracy and improving success rate of CUAs by an average of 27\% relative percentage points on average. Related Works Computer Use Agents CUAs are autonomous agents designed to interact directly with Graphical User Interfaces (GUIs), performing actions such as clicking, typing, scrolling, dragging or navigating web pages. Early examples include browser-based assistants and recent prototypes such as OpenAI’s Computer Use tool 2 2 2 https://openai.com/index/computer-using-agent/ , which integrate vision-language reasoning with low-level action execution. Similar lines of research explore autonomous UI navigation ( 4 ) , multi-modal planning ( 6 ) , and end-to-end web automation benchmarks ( 5 ) . Unlike API-based or function-calling agents that require explicit integration with each service, CUAs operate in a service-agnostic manner: they perceive and act directly on the screen, enabling interaction with any digital environment without additional engineering effort. This design paradigm makes CUAs inherently more flexible and scalable, capable of generalizing across software systems and interfaces ( 15 ; 13 ) . However, this generality also introduces new challenges in reasoning and verification. Because CUAs rely solely on visual observations, they can fail silently or partially when confronted with unexpected interface states, visual occlusions, or distribution shifts ( 4 ; 5 ; 6 ) . This highlights the need for reliable mechanisms to assess whether the intended task has actually been completed, especially when the task cannot be trivially reduced to log-based success signals. Autonomous Evaluation of Task Completion A persistent challenge across all types of agentic systems is evaluating whether a goal has truly been achieved ( 17 ; 2 ; 18 ) . Reliable evaluation is fundamental for measuring performance, enabling self-improvement, and establishing user trust of agents. In this work, we adapt this broader evaluation challenge to the domain of CUAs. The problem of determining whether a CUA has successfully completed a task has not been extensively explored ( 13 ) . While prior work has focused primarily on improving action planning and interface understanding ( 4 ; 6 ; 5 ) , it has paid little attention to how to tell when a task is actually completed. A notable exception is script-based evaluation, used in the OSWorld benchmark ( 16 ) . However, this approach relies on manually written verification scripts for each task, which severely limits scalability and makes real-time evaluation impractical. Maintaining reliable automated evaluation across hundreds of GUI tasks requires substantial manual effort, since even minor interface or environment changes can break the scripts and invalidate results. Also, a closely related effort is the work of 11 , which proposes an autonomous evaluation and refinement framework for digital agents. Their method focuses on automatically assessing and improving web-based and simulated agents by reasoning over structured representations of page elements and textual feedback. Although their approach demonstrates that model-based evaluators can significantly accelerate agent learning, it operates primarily in browser and synthetic environments where the interface semantics and success states are explicitly defined. In contrast, our setting involves real desktop interfaces, specifically macOS, where the screens are harder to parse due to variety and number of elements and also have no universal way of representation as in the web HTML ( 9 ) . This makes our work a complementary extension of autonomous evaluation to unstructured, multimodal environments that better mirror real-world computer use. In contrast, the question of task completion by an agent has been more systematically studied in other domains, particularly in robotics. In robotics, recent work such as AutoEval ( 17 ) introduces autonomous evaluation frameworks for manipulation policies, reducing the reliance on human annotators or scripted success detectors. It reaches both high agreement with human annotations and reduces the human annotation time by 99\% . Our work builds on this line of research but adapts it to the domain of CUAs. Unlike physical robotics tasks, task completion in digital environments often lacks a straightforward ground-truth signal: for instance, whether “sending an email” was completed correctly may not be directly observable from logs alone. Inspired by AutoEval, we propose to use vision-language models as evaluators that judge whether the current desktop state corresponds

Read the complete text