CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang · 2026

Read CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training by Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23559.txt.

Introduction

Chen and six co-authors build ReCAP, a screen agent that can solve CAPTCHAs. Native GUI agents read screenshots and act directly, yet CAPTCHAs trip them up. Special solver pipelines exist, but they cannot do general screen work. ReCAP trains on machine-made interaction trails paired with reasoning text. A testbed of seven CAPTCHA types stresses the skills. Failed trails become self-correction lessons, teaching recovery from mid-task slips. ReCAP lifts CAPTCHA success a lot over its base agents. General screen skill stays strong in the tests.

Catalog record

CREATORYuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61640
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23559

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang Abstract GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices. Despite rapid progress on general GUI tasks, CAPTCHA solving remains a major challenge. On the other hand, although specialized CAPTCHA solving pipelines exist, they cannot handle general GUI tasks. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI agent that can robustly solve modern, interactive CAPTCHA challenges, while preserving their performance as a general GUI agent. We first develop a dynamic CAPTCHA system spanning seven representative CAPTCHA types, designed to stress primitive and complementary capabilities for CAPTCHA solving (e.g., robust OCR under heavy noise and text stylization, fine-grained visual understanding, and precise control). Then, we develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories paired with reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data, training agents to reflect on errors and correct their actions online. Across held-out test sets, ReCAP improves CAPTCHA-solving success from roughly 30% to 80%, while maintaining strong performance on general GUI-agent benchmarks. † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Performance on CAPTCHA and general GUI agent benchmarks. Left: CAPTCHA solving performance across seven challenge types. Right: Performance on general GUI agent benchmarks. ReCAP-32B consistently outperforms baseline GUI agents and prior frameworks on all seven CAPTCHA challenges while maintaining strong general GUI capabilities. CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a critical security mechanism for protecting online services from automated attacks ( 1 ) . Their design has evolved rapidly in response to advances in machine learning. Modern CAPTCHAs increasingly rely on complex visual understanding and interactive behaviors like continuous control and semantic reasoning, rather than static text recognition alone ( 20 ) . Recent GUI agents including OpenCUA ( 24 ) , UI-TARS ( 19 ) , and Qwen3-VL ( 29 ) have demonstrated strong ability on general-purpose GUI interaction tasks, including web browsing, desktop and mobile control, and application usage. Large-scale trajectory datasets from Aguvis ( 28 ) , GUI World ( 3 ) , and AgentNet ( 24 ) have enabled models to acquire broad GUI grounding and interaction skills. Despite this progress, these agents are not explicitly designed for unique challenges posed by modern CAPTCHAs. Consequently, they tend to underperform on CAPTCHA solving as they lack complementary skills required for CAPTCHA solving, including robust OCR under heavy noise and various text stylization, fine-grained visual understanding, and precise control. Solving CAPTCHA has become a challenging task for evaluating the robustness and generalization capabilities of GUI agents ( 27 ) , as CAPTCHAs may naturally appear during daily computer-use. Yet, there is a lack of public and systematic research on studying CAPTCHA solving in native GUI agents. Several recent systems have explored general-purpose approaches to solving modern CAPTCHAs, including Halligan ( 22 ) and Oedipus ( 5 ) . Although effective on specific CAPTCHA benchmarks, these systems frame CAPTCHA solving as a standalone problem, and their transferability to unseen CAPTCHA variants and broader GUI interaction remain limited. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI agent that can robustly solve modern, interactive CAPTCHA challenges. Our approach is enabled by a dynamic, interactive CAPTCHA system spanning seven representative CAPTCHA types: Text, Compact Text, Icon Match, Icon Selection, Paged, Slider, and Image Grid. They are procedurally generated to capture diverse visual layouts and interaction patterns and to stress complementary CAPTCHA-solving capabilities. Building on this system, we develop a scalable data collection and curation pipeline that automatically generates large-scale successful interaction trajectories paired with chain-of-thought (CoT) reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further exploit failed trajectories to construct self-correction data, training agents to reflect on errors, adjust their reasoning, and revise actions toward the correct solution. To mitigate the imbalance caused by the large disparity in sequence lengths, we adopt a weighted training objective that assigns equal total weight to reasoning and action tokens. Across held-out test sets in our dynamic CAPTCHA system, ReCAP-32B improves CAPTCHA solving success rates from roughly 30% to 80%. ReCAP models also achieve substantial gains on real-world CAPTCHAs benchmark consisting of 26 CAPTCHA variants, while maintaining strong performance on general GUI agent benchmarks. Finally, we present ablation studies and analyses that validate the contributions of CoT reasoning and self-correction data. By open-sourcing our dataset, model, and code, we aim to provide a foundation for future research on CAPTCHA-capable GUI agents. 2 Related Works Figure 2 : The suite of CAPTCHA challenges in our dynamic CAPTCHA system, designed to train fundamental CAPTCHA-solving primitives. The challenges are grouped into four core interaction primitives: Optical Character Recognition (OCR), Continuous Control (Dragging), Spatial Localization (Clicking), and Visual Semantic Comprehension. GUI Agent The development of Vision-Language Models (VLMs) has catalyzed a shift toward GUI agent capable of navigating graphical user interfaces. Early work in this domain primarily addressed grounding language instructions to static UI components using supervised learning on datasets of screen-action pairs. However, the field has rapidly transitioned toward end-to-end agents that operate on raw pixel observations to generate low-level control sequences (e.g., clicks, scrolls, and typing) ( 17 ; 32 ) . GUI agent benchmarks such as Mind2Web ( 6 ) , ScreenSpot ( 4 ) , and Android Control ( 14 ) have shifted the focus toward open-ended interaction across diverse operating systems and web environments. Modern agents increasingly leverage high-resolution encoders and specialized architectures to handle fine-grained visual details. For instance, CogAgent ( 13 ) and Ferret-UI ( 30 ) introduce specialized vision towers to improve the perception of small UI elements. Concurrently, the integration of structured reasoning has become a standard. Recent state-of-the-art models like UI-TARS ( 19 ) and OpenCUA ( 24 ) employ autoregressive formulations that interleave visual perception with chain-of-thought reasoning. This trend has been further reinforced by advances in multimodal foundation models, including large open-weight vision-language models that support direct action generation and tool use ( 2 ; 29 ) . Overall, the field is rapidly converging toward general-purpose GUI agents that can handle diverse interactive tasks using a unified perception-reasoning-action interface. Evolution of CAPTCHAs CAPTCHAs were originally introduced as text-recognition challenges that exploited the gap between human perception and machine OCR capabilities ( 1 ) . More recently, reasoning CAPTCHAs mark a shift from simple pattern recognition toward tasks requiring logical reasoning, contextual understanding, and multi-step interactions ( 9 ; 8 ) . Unlike traditional CAPTCHAs, their designs explicitly target cognitive abilities that remain difficult for automated solvers. As deep learning significantly improved OCR performance, text-based CAPTCHAs became largely ineffective, motivating a transition from ”hard-to-read” to ”hard-to-reason” challenges ( 21 ; 12 ) . Modern systems, such as Google’s reCAPTCHA v2, employ rich visual-semantic tasks including image classification, object localization, and spatial reasoning ( 20 ) . Beyond static puzzles, contemporary CAPTCHAs increasingly incorporate behavioral signals and multi-step interactive sequences to distinguish human-like

Read the complete text