Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning

Hai-Ming Xu, Qi Chen, Lei Wang, Lingqiao Liu · 2024

Read Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning by Hai-Ming Xu, Qi Chen, Lei Wang, Lingqiao Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10840.txt.

Introduction

Hai-Ming Xu and co-authors ground screen parts with attention maps in a 2024 paper. GUI agents must find the button or text a user names in a screen shot. Past craft fine-tunes a model for that hunt. Their TAG path skips the tune step. A set prompt makes a stock sight model look; its own attention maps then pool to a point call. One open model hosts the tests.

TAG matched tuned tools on the bench grades. Text hunts shone most. Raw point calls from the same model fell far shy of the map path. The authors pitch attention as a free sense the models hold.

Catalog record

CREATORHai-Ming Xu, Qi Chen, Lei Wang, Lingqiao Liu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10840

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning Hai-Ming Xu, Qi Chen, Lei Wang, Lingqiao Liu Abstract Recent advancements in Multimodal Large Language Models (MLLMs) have generated significant interest in their ability to autonomously interact with and interpret Graphical User Interfaces (GUIs). A major challenge in these systems is grounding—accurately identifying critical GUI components such as text or icons based on a GUI image and a corresponding text query. Traditionally, this task has relied on fine-tuning MLLMs with specialized training data to predict component locations directly. However, in this paper, we propose a novel Tuning-free Attention-driven Grounding (TAG) method that leverages the inherent attention patterns in pretrained MLLMs to accomplish this task without the need for additional fine-tuning. Our method involves identifying and aggregating attention maps from specific tokens within a carefully constructed query prompt. Applied to MiniCPM-Llama3-V 2.5, a state-of-the-art MLLM, our tuning-free approach achieves performance comparable to tuning-based methods, with notable success in text localization. Additionally, we demonstrate that our attention map-based grounding technique significantly outperforms direct localization predictions from MiniCPM-Llama3-V 2.5, highlighting the potential of using attention maps from pretrained MLLMs and paving the way for future innovations in this domain. Code is available at https://github.com/HeimingX/TAG.git . 1 Australian Institute for Machine Learning, The University of Adelaide 2 University of Wollongong {hai-ming.xu, qi.chen04, lingqiao.liu}@adelaide.edu.au, [email protected] 1 Introduction The integration of artificial intelligence with Graphical User Interfaces (GUIs) holds tremendous potential to transform how humans interact with software systems. Leading this innovation are Multimodal Large Language Models (MLLMs) ( OpenAI 2023 ; Reid et al. 2024 ; Anthropic 2024 ) , which have shown exceptional capabilities in interpreting GUIs across various applications. A crucial task in AI for GUIs is GUI grounding—accurately identifying and localizing key components such as text and icons—since this is fundamental to enabling the automated operation of GUIs. While MLLMs excel at understanding GUI images, the precise grounding of GUI elements remains challenging. Current state-of-the-art solutions often improve the GUI grounding capabilities of MLLMs through fine-tuning on specialized datasets, as demonstrated in works like ( Hong et al. 2023 ; Cheng et al. 2024 ) . In these methods, the MLLM directly predicts the location of GUI elements. In contrast, our approach takes a different path by leveraging the inherent attention patterns of a pretrained MLLM, utilizing its existing spatial awareness and attention mechanisms to achieve accurate GUI grounding without the need for additional fine-tuning. We propose a novel Tuning-free Attention-driven Grounding (TAG) approach that carefully selects and aggregates attention patterns from MiniCPM-Llama3-V 2.5, a state-of-the-art MLLM, to perform GUI element grounding. Our method begins by identifying specific tokens from either the user input query or the model-generated response and then propagates the corresponding attention values back to the image plane. To further enhance performance, we implement a selective mechanism to filter out irrelevant attention heads, ensuring that only the most relevant attention is utilized for accurate grounding. We compare our approach with existing tuning-based methods, and our results demonstrate that utilizing attention patterns from a pretrained model can achieve accurate GUI element grounding. Additionally, our approach significantly improves text localization. These findings highlight the untapped potential of leveraging inherent model capabilities and open the door to more robust, scalable, and efficient applications of MLLMs in GUI automation. Figure 1: Illustration of MiniCPMV2.5’s strong GUI image understanding but poor element localization. Our attention-driven GUI grounding leverages its inherent attention to enhance localization accuracy without fine-tuning, as shown on the right. 2 Related Works Multimodal Large Language Models for GUI Agents The use of MLLMs ( Liu et al. 2024a ; Bai et al. 2023 ; Yao et al. 2024 ; Liu et al. 2024b ; Zhu et al. 2023 ; Lu et al. 2024 ; Wang et al. 2023 ) as GUI agents marks significant progress in AI’s ability to interact with GUI. These models understand user queries and images, enabling them to perform tasks across various platforms, from desktops to mobiles. Recent works in this domain have explored various applications, from automating routine tasks on desktop interfaces ( Hong et al. 2023 ; Wu et al. 2024 ; Kil et al. 2024 ; He et al. 2024 ; Kapoor et al. 2024 ; Xie et al. 2024 ) to providing interactive assistance on mobile platforms ( Ma, Zhang, and Zhao 2024 ; Nong et al. 2024 ; Wang et al. 2024b ; Wang et al. 2024c ; You et al. 2024 ) . These applications highlight the potential of MLLMs to act as autonomous agents that can understand and execute user commands across different platforms. However, the challenge often lies in effectively training these models to handle the intricacies and variability of GUIs without extensive domain-specific tuning. Grounding in GUI Agents Grounding in GUI agents ( Cheng et al. 2024 ; Li, Mitchell, and Myers 2020 ; Liu et al. 2024c ; Gao et al. 2024 ; Wang et al. 2024a ; Li and Li 2023 ; Wang, Li, and Li 2023 ; Li et al. 2020 ) involves the model’s ability to locate and identify interface elements accurately, which is essential for effective interaction. Traditional methods ( Cheng et al. 2024 ; Chen et al. 2024 ; Hong et al. 2023 ; Fan et al. 2024 ) typically require fine-tuning on detailed, annotated datasets. Recent research has explored both supervised and unsupervised techniques to enhance grounding accuracy, such as the SeeClick model ( Cheng et al. 2024 ) fine-tunes on GUI-specific datasets. However, these methods can suffer from scalability issues and overfitting. Our work contributes to this field by proposing a tuning-free approach that leverages pre-trained MLLMs’ inherent attention mechanisms to associate text queries with visual elements, offering a scalable and adaptable grounding solution. 3 Our Method Figure 2: Overall pipeline of our TAG approach in Sec. 3.4 (top) and the self-attention selection module in Sec. 3.5 (bottom). 3.1 Preliminary GUI Grounding GUI grounding is a crucial task for agents that interpret and interact with graphical user interfaces (GUIs). It demands that systems comprehend users’ text queries, such as “I want to book a dental appointment on Tuesday”, analyze GUI screenshots, and accurately pinpoint the relevant components. While recent advancements in MLLMs have shown potential in understanding both textual queries and visual GUI layouts, they often encounter challenges in precisely localizing elements without the aid of additional tools like OCR ( Wang et al. 2024b ) or Set-of-Mark ( Yang et al. 2023 ; Wang et al. 2024b ) techniques. Thus, SOTA methods typically rely on fine-tuning MLLMs with specialized training data to directly achieve accurate element localization ( Hong et al. 2023 ; Cheng et al. 2024 ) . MiniCPMV2.5 and Its Attention Map MiniCPMV2.5 (i.e., MiniCPM-Llama3-V 2.5) is a SOTA MLLM that integrates a vision encoder, a token compression module, and the Llama3 language model. It supports high-resolution images up to 1344×1344 pixels with any aspect ratio, making it well-suited for precise GUI grounding tasks. To manage the large number of visual tokens generated from high-resolution inputs, the model uses cross-attention to compress thousands of vision patch embeddings into a fixed-size ( Q ) set of visual query tokens. These visual query tokens are then processed alongside

Read the complete text