AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent

Neeraj Anand, Rishabh Jain, Sohan PatnaikBalaji KrishnamurthyMausoom SarkarMedia and Data Science Research, Adobe · 2025

Read AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent by Neeraj Anand, Rishabh Jain, Sohan PatnaikBalaji KrishnamurthyMausoom SarkarMedia and Data Science Research, Adobe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00846.txt.

Introduction

Neeraj Anand and coauthors at Adobe present AFRAgent in a 2025 paper on phone automation. Demand is growing for software that can operate mobile interfaces on its own. Visual language models have moved GUI agents from writing instructions for people toward carrying out the tasks themselves.

Screens vary in size and sharpness, which trips up agents that must tap small targets. AFRAgent uses adaptive feature renormalization, a way to rescale what the model sees, with high-resolution screens in mind. Sharper, better-scaled perception could make phone agents more reliable at real tasks like booking, ordering, and settings changes.

Catalog record

CREATORNeeraj Anand, Rishabh Jain, Sohan PatnaikBalaji KrishnamurthyMausoom SarkarMedia and Data Science Research, Adobe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00846

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent Neeraj Anand, Rishabh Jain, Sohan PatnaikBalaji KrishnamurthyMausoom SarkarMedia and Data Science Research, Adobe Abstract There is a growing demand for mobile user interface (UI) automation, driven by its broad applications across industries. With the advent of visual language models (VLMs), GUI automation has progressed from generating text-based instructions for humans to autonomously executing tasks, thus optimizing automation workflows. Recent approaches leverage VLMs for this problem due to their ability to 1) process on-screen content directly, 2) remain independent of device-specific APIs by utilizing human actions (e.g., clicks, typing), and 3) apply real-world contextual knowledge for task understanding. However, these models often have trouble accurately identifying widgets and determining actions due to limited spatial information in vision encoder features. Additionally, top-performing models are often large, requiring extensive training and resulting in inference delays. In this work, we introduce AFRAgent , an instruct-BLIP-based multimodal architecture that achieves superior performance in GUI automation while being less than one-fourth the size of its nearest competitor. To enhance image embeddings in the large language model (LLM) pipeline, we propose an adaptive feature renormalization-based (a token-level affine transformation) technique that effectively enriches low-resolution image embeddings and fuses high-resolution details. We evaluate AFRAgent on Meta-GUI and AITW benchmarks, establishing a new state-of-the-art baseline for smartphone automation. 1 1 footnotemark: 1 The code is available at https://github.com/neerajanand321/AFRAgent † † footnotetext: ∗ equal contribution † Contact: neeraja,rishabhj,[email protected] 1 Project Webpage: Website 1 Introduction Figure 1 : AFRAgent outperforms existing methods in UI automation on AITW dataset, achieving state-of-the-art results with significantly fewer parameters and reduced computational overhead. Advancements in deep learning have made previously challenging tasks achievable, particularly with the emergence of transformer-based architectures. Progress in natural language processing (NLP), computer vision, and multimodal models has significantly enhanced our capacity to address complex tasks. Among these tasks, GUI automation is crucial due to its wide-ranging real-world applications and substantial economic impact. A model must accomplish a user-defined objective, such as scheduling calendar events or preparing presentations on specific topics on a desktop or mobile operating system. While this problem has been explored even before the emergence of large language models (LLMs) [ 47 ] , the development of LLMs has led to notable improvements in action prediction accuracy [ 57 ] , strategic planning for sequential steps [ 22 , 46 ] , and generalization across various environments and applications [ 46 ] . Previous approaches, such as verbalization of screen content combined with prompt engineering and fine-tuning [ 12 , 39 ] , have leveraged LLMs to predict necessary subsequent actions. However, since user interfaces are fundamentally intended for visual engagement, text-only methods prove inadequate compared to multimodal models incorporating spatial information. Large visual language models (VLMs) have demonstrated effectiveness across diverse tasks, including optical character recognition (OCR) [ 4 ] , image captioning [ 1 ] , visual grounding [ 52 , 34 ] , question answering [ 3 ] , and reasoning [ 9 ] . Recently, VLMs have also demonstrated promising potential in automating user-defined workflows. For instance, Auto-GUI [ 56 ] introduced a chain-of-thought approach that combines past action steps with future planning to determine the following action. SeeClick [ 11 ] emphasized the importance of identifying icons and widgets on the screen, pretraining their network with GUI-grounded data to enhance alignment for automation tasks. CogAgent [ 16 ] incorporated both low and high-resolution vision encoders to capture fine-grained details from mobile screenshots, while CoCo-Agent [ 31 ] provided layout information as input to the LLM and simplified action prediction into a more semantic structure to improve performance. Despite these advances, current methods have several limitations. First, many approaches rely on external tools like OCR and icon detectors [ 54 , 41 ] to transform the interface into text-based inputs for language models. This dependence can introduce inefficiencies, as lengthy inputs increase processing time, risk exceeding model input limits, and may suffer from information loss or conversion errors. Additionally, extracting layout information from third-party applications can be challenging. In contrast, AFRAgent efficiently processes mobile screenshots directly without needing external tools. Secondly, current approaches rely on computationally intensive models with parameter counts ranging from 7B [ 31 ] to 18B [ 16 ] , making edge deployment difficult despite its critical importance for preserving privacy in UI interactions. Our approach significantly reduces this computational burden by utilizing a more efficient 4B parameter model while maintaining competitive performance, making edge deployment more feasible. Thirdly, existing UI automation methods typically use either Llava-style projections [ 31 , 26 ] or learnable query embeddings [ 28 ] to extract screen information. However, recent research [ 17 , 55 ] suggests that a combination of direct projection and learnable query embeddings provides enhanced results. Lastly, existing VLMs often incorporate high-resolution information through methods that substantially increase memory and computational overhead, such as increasing the size of the vision encoder [ 10 ] , splitting images into multiple crops [ 27 ] resulting in additional LLM tokens, or using cross-attention to integrate information at each layer of the LLM [ 16 ] . To address these issues, we propose an Adaptive Feature Renormalization (AFR) technique that effectively merges image embeddings with learned query projections without adding a significant computational burden. Additionally, we enhance the InstructBLIP architecture by integrating AFR to combine low-resolution and high-resolution features via Q-Former, demonstrating the adaptability of our approach. In summary, our work makes the following technical contributions • We propose Adaptive Feature Renormalization, a novel technique to enrich the learned query projections with the image embeddings effectively. • We adapt InstructBlip architecture to fuse high resolution information into the low-resolution embeddings via AFR. • Our 4-billion parameter model achieves state-of-the-art results on key GUI benchmarks, significantly surpassing larger models and demonstrating superior performance(Fig 1 ). • We provide comprehensive quantitative and qualitative evidence, along with ablation studies, to demonstrate the effectiveness of each component. 2 Related Works Agents are systems that perceive their environment, make decisions, and pursue goals [ 47 ] . Recent advances in language models have improved agents’ adaptability and generalization [ 32 , 45 ] , enabling memory, planning, and applications in domains like software development [ 23 ] , scientific research [ 6 ] , and web-based tasks, especially autonomous GUI navigation using (M)LLMs. Autonomous GUI Navigation Autonomous navigation in graphical user interfaces (GUIs) leverages these advancements to enable agents to interact with and control complex digital environments. LLM-based methods such as WebAgents [ 15 ] , Mind2Web [ 12 ] , WebArena [ 58 ] , m-BASH [ 39 ] , and AutoWebGLM [ 21 ] utilize textual representations of screens (e.g., HTML, accessibility trees, OCR) along with task goals to guide navigation. These models effectively process and summarize verbose structures, but they struggle with challenges like HTML verbosity, high token counts, and loss of fine-grained visual details, limiting their ability to fully replicate human-like interactions with GUIs. In contrast, multimodal approaches combine textual and visual information to improve agent performance. Models like WebGum [ 14 ] , AppAgent [ 53 ] , Mobile-Agent [ 44 ] , Auto-GUI [ 56 ] , UI-TARS [ 35 ]

Read the complete text