CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning

Marta Sumyk, Oleksandr Kosovan · 2026

Read CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning by Marta Sumyk, Oleksandr Kosovan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10577.txt.

Introduction

Marta Sumyk and Oleksandr Kosovan test whether vision-language models can audit computer-use agents, software that carries out desktop tasks from plain instructions. Five models judged task success from an instruction and the final screen state, across three benchmarks on macOS, Windows, and Linux. The study scored accuracy, calibration of confidence, and agreement between models. Leading models were accurate and well calibrated, yet all slipped in complex or mixed settings, and even the best disagreed a lot. Auditing itself, they argue, needs reliability checks.

Catalog record

CREATORMarta Sumyk, Oleksandr Kosovan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27470
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10577

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents CCS: Human-centered computing Human computer interaction (HCI) CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning Marta Sumyk, Oleksandr Kosovan Abstract . Computer-Use Agents (CUAs) are emerging as a new paradigm in human-computer interaction, enabling autonomous execution of tasks in desktop environment by perceiving high-level natural-language instructions. As such agents become increasingly capable and are deployed across diverse desktop environments, evaluating their behavior in a scalable and reliable manner becomes a critical challenge. Existing evaluation pipelines rely on static benchmarks, rule-based success checks, or manual inspection, which are brittle, costly, and poorly aligned with real-world usage. In this work, we study Vision-Language Models (VLMs) as autonomous auditors for assessing CUA task completion directly from observable interactions and conduct a large-scale meta-evaluation of five VLMs that judge task success given a natural-language instruction and the final environment state. Our evaluation spans three widely used CUA benchmarks across macOS, Windows, and Linux environments and analyzes auditor behavior along three complementary dimensions: accuracy, calibration of confidence estimates, and inter-model agreement. We find that while state-of-the-art VLMs achieve strong accuracy and calibration, all auditors exhibit notable performance degradation in more complex or heterogeneous environments, and even high-performing models show significant disagreement in their judgments. These results expose fundamental limitations of current model-based auditing approaches and highlight the need to explicitly account for evaluator reliability, uncertainty, and variance when deploying autonomous CUAs in real-world settings. † † footnotetext: This work has been accepted to appear at the HEAL @ CHI 2026 Worshop on Human-centered Evaluation and Auditing of Language Models. 1. Introduction Recent advances in large language models and multimodal perception have given rise to Computer-Use Agents (CUAs): autonomous systems that can operate Graphical User Interfaces (GUIs) by translating high-level natural-language instructions into sequences of actions such as clicking, typing, scrolling, and dragging ( 21 ) . From a Human–Computer Interaction (HCI) perspective, CUAs extend a long line of work on interface agents and intelligent user interfaces, where users attribute intent, agency, and social meaning to interactive systems rather than viewing them as purely functional tools ( 7 ) . Recent systems further demonstrate that large vision-language models can act as unified controllers for complex desktop environments, generalizing across applications, tasks, and operating systems without relying on handcrafted rules ( 26 ) . As a result, CUAs offer a service-agnostic alternative to traditional robotic process automation, reducing brittleness and maintenance costs while supporting a broader range of real-world tasks ( 21 ) . Beyond automation, CUAs hold particular promise for accessibility and inclusive interaction. When paired with natural-language or voice interfaces, they enable users with motor, visual, or cognitive impairments to complete multi-step tasks through language alone ( 25 ; 28 ) . More broadly, CUAs can reduce cognitive and interaction burdens for non-technical users, older adults, and individuals facing language or executive-function challenges ( 3 ) . As CUAs are increasingly deployed in real-world settings, rigorous evaluation prior to deployment becomes essential. However, assessing CUA behavior remains a fundamental challenge. Existing evaluation pipelines rely on static benchmarks, rule-based success checks, or manual inspection, all of which are costly to maintain, brittle to interface changes, and poorly aligned with real-world usage ( 27 ) . Such approaches typically yield coarse success signals and provide limited insight into partial task completion, user-acceptable failures, or performance under realistic UI variation. These limitations are especially concerning given that CUAs act autonomously on users’ behalf, often across multiple applications and involving sensitive data. In this work, we study Vision-Language Models (VLMs) as autonomous auditors for CUAs. Rather than relying on internal agent states or handcrafted evaluation logic, VLM-based auditors assess task completion directly from observable evidence by judging whether a natural-language instruction has been satisfied in the final GUI state. We conduct a large-scale meta-evaluation of VLM auditors across multiple operating systems and benchmarks, analyzing their accuracy, confidence calibration, and inter-model agreement. By treating evaluation as a first-class problem, our study characterizes the reliability and limitations of model-based auditing and highlights key challenges for the safe and robust deployment of CUAs in real-world settings. 2. Related Works 2.1. Computer-Use Agents and GUI Automation Research on CUAs builds on a long history of GUI automation, robotic process automation (RPA), and intelligent user interfaces. Early systems relied on handcrafted rules, application-specific scripts, DOM trees, or accessibility APIs to automate repetitive tasks. While effective in controlled settings, these approaches were brittle to interface changes, required substantial manual maintenance, and failed to generalize across applications or operating systems ( 3 ) . Recent work has shifted toward learning-based approaches that operate directly on multimodal observations of the interface, typically combining screenshots with natural-language task instructions. This paradigm enables agents to interact with graphical user interfaces through the same perceptual and control channels available to human users. Systems such as SeeAct ( 29 ) , InfiGUIAgent ( 16 ) , SEAGENT ( 24 ) , and UI-TARS ( 26 ) demonstrate that large vision-language models can act as general-purpose GUI controllers in a wide range of desktop and mobile environments. Collectively, these results show that CUAs can achieve substantial cross-application and cross-platform generalization without relying on application-specific APIs or predefined workflows. By treating the GUI as an executable visual environment rather than a structured programmatic interface, CUAs represent a departure from traditional automation pipelines and enable more flexible, service-agnostic interaction with existing software ecosystems. 2.2. CUA as a New HCI Concept CUAs introduce an emerging interaction paradigm in which users delegate high-level goals to autonomous agents that perceive, reason, and act directly within existing GUIs. Unlike traditional interaction models based on direct manipulation ( 22 ) , CUAs function as intermediaries that execute tasks on the user’s behalf through the same visual and control channels available to humans. From an HCI perspective, CUAs build upon earlier work on interface agents and intelligent user interfaces, which explored how software agents could assist users through recommendations, reminders, or adaptive behavior ( 17 ; 12 ) . These systems, however, typically played a supportive or advisory role and relied on structured application access, predefined workflows, or handcrafted rules. In contrast, modern CUAs are designed for end-to-end task execution: given a natural-language instruction, the agent must interpret user intent, observe the current interface state, plan a sequence of actions, and adapt its behavior in dynamic and partially observable environments. This shift places CUAs within the tradition of mixed-initiative interaction and human–automation collaboration, where control is shared between humans and autonomous systems ( 15 ; 9 ; 8 ) . However, CUAs push this paradigm further by substantially reducing direct user oversight during task execution. The graphical user interface becomes an executable environment rather than a passive display, and interaction is reframed as a sequential decision-making process over perceptual inputs and actions such as clicking, typing, scrolling, or dragging. This framing aligns CUAs with agent-based models of perception–action loops in interactive systems ( 20 ) . At the same time, increased autonomy introduces challenges central to HCI research on trust, safety, and usability. Prior work shows that reduced human control can lead to loss of transparency,

Read the complete text