OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents

Pengzhou Cheng, Zheng Wu *, Zongru Wu, Tianjie Ju, Aston Zhang, Zhuosheng Zhang †, Gongshen Liu † · 2025

Read OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents by Pengzhou Cheng, Zheng Wu *, Zongru Wu, Tianjie Ju, Aston Zhang, Zhuosheng Zhang †, Gongshen Liu † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16465.txt.

Introduction

Pengzhou Cheng and co-authors give screen-operating AI agents a sense of when to ask for help. Agents that click and type through apps on their own can run past the point where they should pause, a flaw the authors call over-execution. Ambiguous orders, pop-ups, and hijacked screens make blind autonomy risky. Their 2025 paper presents OS-Kairos, an agent that scores its own confidence at each step.

Two mechanisms drive the design. A probing step labels training episodes with step-by-step confidence scores, and a second step teaches the agent to act alone when confident and hand control to a human when not. On a new test set of tangled scenarios and on two public benchmarks, the agent beat rivals by wide margins in task success. The authors aim for computer help that is effective and safe in real use, not just in clean demos.

Catalog record

CREATORPengzhou Cheng, Zheng Wu *, Zongru Wu, Tianjie Ju, Aston Zhang, Zhuosheng Zhang †, Gongshen Liu †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16465

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents Pengzhou Cheng, Zheng Wu *, Zongru Wu, Tianjie Ju, Aston Zhang, Zhuosheng Zhang †, Gongshen Liu † Abstract Autonomous graphical user interface (GUI) agents powered by multimodal large language models have shown great promise. However, a critical yet underexplored issue persists: over-execution , where the agent executes tasks in a fully autonomous way, without adequate assessment of its action confidence to compromise an adaptive human-agent collaboration. This poses substantial risks in complex scenarios, such as those involving ambiguous user instructions, unexpected interruptions, and environmental hijacks. To address the issue, we introduce OS-Kairos, an adaptive GUI agent capable of predicting confidence levels at each interaction step and efficiently deciding whether to act autonomously or seek human intervention. OS-Kairos is developed through two key mechanisms: (i) collaborative probing that annotates confidence scores at each interaction step; (ii) confidence-driven interaction that leverages these confidence scores to elicit the ability of adaptive interaction. Experimental results show that OS-Kairos substantially outperforms existing models on our curated dataset featuring complex scenarios, as well as on established benchmarks such as AITZ and Meta-GUI, with 24.59% ∼ \sim 87.29% improvements in task success rate. OS-Kairos facilitates an adaptive human-agent collaboration, prioritizing effectiveness, generality, scalability, and efficiency for real-world GUI interaction. The dataset and codes are available at https://github.com/Wuzheng02/OS-Kairos . 1 Introduction Figure 1: Illustration of GUI agents executing a complex shopping instruction across two paradigms: (a) Autonomous , where the agent cannot complete the task independently; (b) Adaptive , where the agent dynamically adjusts its autonomy based on confidence levels. Multimodal large language models (MLLMs) have been explored to develop graphical user interface (GUI) agents capable of analyzing the screen and performing human-like behaviors on operating systems Hong et al. (2024) ; Zhang et al. (2024a) ; Wang et al. (2024a) . Existing efforts in building GUI agents have focused on the autonomous mode Niu et al. (2024) ; Zhang et al. (2024b) ; Nguyen et al. (2024) ; Liu et al. (2024) , with improved capabilities such as grounding ( Wu et al., 2024b ; Qin et al., 2025 ) and reasoning ( Zhang and Zhang, 2024 ; Zhang et al., 2024b ; Liu et al., 2025 ) . Despite exciting progress, we observe that existing GUI agents exhibit significant over-execution issues — the agent executes tasks in a fully autonomous way, without adequate assessment of its action confidence to compromise an adaptive human-agent collaboration. As shown in Figure 1 (a), popular GUI agents such as OS-Atlas ( Wu et al., 2024b ) are unable to click the filters button correctly, causing unexpected interruptions and task failure. Over-execution poses significant challenges in complex real-world scenarios (Examples shown in Figure 2 ), highlighting fundamental limitations in current GUI agents. First, ambiguous instructions from the user leads to information absence in GUI automation (e.g., account logout). Second, existing GUI agents depend heavily on the foundation MLLMs and therefore suffer from unexpected interruptions when executing complex instructions. Besides, these models will also generate hallucinations Sridhar et al. (2023) and shortcut predictions Wu et al. (2024b) ; Zhu et al. (2024) . Third, environmental hijacks , such as network connection failure and pop-up hijacking) Ma et al. (2024a) . To address these challenges, we are motivated to integrate confidence scoring into the foundation model, allowing adaptive human intervention for GUI agents (Figure 1 (b)). Concretely, we introduce OS-Kairos, an adaptive GUI agent capable of predicting confidence levels at each interaction step and efficiently determining whether to act autonomously or seek human intervention. OS-Kairos incorporates two key mechanisms: (i) collaborative probing that annotates confidence scores at each interaction step; (ii) confidence-driven interaction that utilizes these confidence scores to enhance the ability of adaptive interaction. To annotate the confidence scores for the probed GUI agents in real-world scenarios, we first design a collaborative confidence probing framework. Inspired by Chen et al. (2024a) , this framework integrates a layout parsing model Tang et al. (2019) and the most capable proprietary model, GPT-4o Achiam et al. (2023) to function as a critic model. The critic model is used to supervise plan scheduling and confidence score based on our curated instructions to address complex scenarios. This framework is the first toolkit designed to identify when human intervention is necessary, generate confidence scores, and facilitate the automated construction of GUI trajectories. To further integrate confidence scoring into the probed GUI agent, we validate and refine these GUI trajectories and then fine-tune the model. This approach ensures action prediction accuracy while improving adaptability of human intervention. Experimental results in complex scenarios show that OS-Kairos achieves state-of-the-art performance with action type success rate of 99.88%, action success rate of 95.90%, and task success rate of 88.20%. Also, we confirm OS-Kairos’s effectiveness on two well-established GUI benchmarks: Meta-GUI Sun et al. (2022) and AITZ Zhang et al. (2024b) . Comprehensive analysis reveals that OS-Kairos prioritizes effectiveness, generality, scalability, and efficiency, making it a competitive agent for real-world GUI interactions. Our work makes the following key contributions: (i) We introduce OS-Kairos, an adaptive GUI agent that predicts the confidence level of each interaction step and effectively decides whether to act autonomously or seek human intervention. (ii) We propose a collaborative confidence probing framework for dynamically identifying the confidence scores of the GUI agents in typical complex real-world scenarios, while automatically generating high-quality GUI trajectory. (iii) We employ confidence-driven interaction to integrate confidence scoring into the GUI agent that forms adaptive human intervention without compromising action prediction. (iv) We demonstrate that OS-Kairos substantially outperforms existing models on both our curated dataset featuring complex scenarios and well-established benchmarks, with merits of effectiveness, generality, scalability, and efficiency. 2 Related Works Our work falls into the field of MLLM-powered agents. This section will first review the recent progress in building GUI agents and then discuss the capability probing approaches for GUI agents. 2.1 MLLM-powered GUI Agents The rise of MLLMs has redefined the paradigm for GUI agents, enabling them to analyze complex screen layouts and generate accurate actions in a more human-like way Zhang et al. (2024a) . Importantly, this paradigm is a non-intrusive manner without reliance on complex, platform-specific scripts or predefined workflows. Notable examples across different platforms include SeeAct Zheng et al. (2024) and WebRL Qi et al. (2024) for web navigation, AppAgent Zhang et al. (2023) , Auto-UI Zhang and Zhang (2024) , and CoCoAgent Ma et al. (2024b) for mobile interactions, and ScreenAgent Niu et al. (2024) for Windows OS applications. This paper investigates the over-execution of MLLM-powered GUI agents on mobile devices. Early efforts to build GUI agents rely on the availability of commercial MLLMs. These agents can be built through prompt learning based on GPT-4o or Gemini-Pro Vision, e.g., AppAgent Zhang et al. (2023) and Mobile-Agent Wang et al. (2024a) . However, practitioners are concerned about the costs associated with API requests and the delays in inference on mobile devices. Recent studies have focused on fine-tuning to optimize foundation models. On the one hand, they work on performing fine-grained visual understanding Bai et al. (2023) , model scaling laws Chen et al. (2024b) , multimodal information integration Hong et al. (2024) , and GUI grounding enhancements Wu et al. (2024b) ; Qin

Read the complete text