CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare
Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh, Numan Saeed, Sriparna Saha, Xiuying Chen, Salman Khan, [0.4em] Indian Institute of Technology Patna Mohamed bin Zayed University of AI (MBZUAI) [0.2em] · 2026
Read CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare by Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh, Numan Saeed, Sriparna Saha, Xiuying Chen, Salman Khan, [0.4em] Indian Institute of Technology Patna Mohamed bin Zayed University of AI (MBZUAI) [0.2em] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24157.txt.
Introduction
Ghosh and co-authors present CarePilot, a multi-agent system that automates long healthcare computer tasks. They also build CareFlow, a human-annotated benchmark of long workflows in medical tools. It covers annotation software, DICOM viewers, health record systems, and lab systems. Plain vision-language models do poorly on these long tasks. CarePilot uses an actor-critic design. The actor picks the next step from the screen and memory, while the critic checks each action and corrects it. Scores beat strong baselines by about 15 and 3 percent.
Catalog record
| CREATOR | Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh, Numan Saeed, Sriparna Saha, Xiuying Chen, Salman Khan, [0.4em] Indian Institute of Technology Patna Mohamed bin Zayed University of AI (MBZUAI) [0.2em] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55929 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare Akash Ghosh, Tajamul Ashraf, Rishu Kumar Singh, Numan Saeed, Sriparna Saha, Xiuying Chen, Salman Khan, [0.4em] Indian Institute of Technology Patna Mohamed bin Zayed University of AI (MBZUAI) [0.2em] Abstract Multimodal agentic pipelines are transforming human–computer interaction by enabling efficient and accessible automation of complex, real-world tasks. However, recent efforts have focused on short-horizon or general-purpose applications (e.g., mobile or desktop interfaces), leaving long-horizon automation for domain-specific systems, particularly in healthcare, largely unexplored. To address this, we introduce CareFlow, a high-quality human-annotated benchmark comprising complex, long-horizon software workflows across medical annotation tools, DICOM viewers, EHR systems, and laboratory information systems. On this benchmark, existing vision–language models (VLMs) perform poorly, struggling with long-horizon reasoning and multi-step interactions in medical contexts. To overcome this, we propose CarePilot , a multi-agent framework based on the actor–critic paradigm. The Actor integrates tool grounding with dual-memory mechanisms, long-term and short-term experience, to predict the next semantic action from the visual interface and system state. The Critic evaluates each action, updates memory based on observed effects, and either executes or provides corrective feedback to refine the workflow. Through iterative agentic simulation, the Actor learns to perform more robust and reasoning-aware predictions during inference. Our experiments show that CarePilot achieves state-of-the-art performance, outperforming strong closed-source and open-source multimodal baselines by approximately 15.26% and 3.38% , on our benchmark and out-of-distribution dataset, respectively. The code and dataset for this project are available at: Carepilot . 1 Introduction Figure 1 : CareFlow is a large-scale benchmark for evaluating multimodal agents on a range of real healthcare software. It enables execution-based evaluation and interactive reasoning across DICOM viewers, image annotation tools, EMR/EHR, and LIS platforms. Each task pairs a natural-language goal with GUI screenshots representing authentic clinical workflows. In today’s digital world, software systems powered by large language models (LLMs) and vision language models (VLMs) form the backbone of modern activity, shaping how humans learn, collaborate, analyze data, and create content across domains, interfaces, and tools 6 ; 26 ; 22 ; 15 ; 13 ; 18 ; 14 ; 19 ; 16 ; 17 ; 1 ; 24 ; 25 ; 5 ; 4 . Recent advances in large multimodal models (LLMs) have enabled autonomous software agents that can follow high-level natural language instructions to operate real applications, making complex computer use more accessible and efficient 51 ; 53 . However, building agents that can reliably execute long-horizon workflows, spanning dozens of interdependent steps under partial observability, remains a major challenge. A key obstacle is the lack of realistic and interactive benchmarks that capture the heterogeneity of operating systems, interfaces, and domain-specific software environments 50 ; 37 . Moreover, such long-horizon multimodal agents require robust grounding and memory mechanisms to make informed decisions at each step, a difficult problem that demands effective use of tool representations, contextual reasoning, and long-term memory integration 46 ; 39 ; 47 ; 29 . Healthcare software ecosystems are inherently broad and workflow-centric, spanning DICOM servers/viewers, image-computing and annotation tools, EMR/EHR systems, and laboratory information systems (LIS) 32 ; 28 . Day-to-day clinical use often requires chaining 10–15 dependent actions, for example, opening a study, configuring views, annotating or measuring, exporting artifacts, and updating records, while adhering to data integrity, audit trails, and strict privacy policies. These platforms are highly heterogeneous and policy-constrained, and they evolve frequently: user-interface updates, custom deployments, and institution-specific configurations make agents that overfit to surface layouts brittle. This combination of heterogeneity, long-horizon dependencies, and strict compliance requirements makes healthcare a natural yet uniquely challenging testbed for long-horizon GUI agents 40 . Despite recent progress on long-horizon multimodal agents in Android, desktop, and web environments 44 ; 52 ; 50 ; 38 , there remains no standardized public benchmark for healthcare or clinical settings that reflects how users interact with multiple medical softwares. This absence of domain-grounded evaluation makes it difficult to assess how current agents generalize to healthcare-specific tasks typically performed by trained medical professionals 37 . Addressing this gap is essential for developing robust, trustworthy multimodal agents capable of operating safely and efficiently in clinical software ecosystems. With this motivation, we introduce CareFlow , a healthcare-specific long-horizon benchmark that evaluates complex workflows requiring domain knowledge of specialized software. CareFlow contains tasks spanning 8–24 consecutive decisions, executed over sequences of GUI screenshots from real medical softwares. At each timestep t , the agent receives the current screenshot, the task instruction, and a condensed history of prior states/actions, and must predict the next semantic action to advance the workflow. A key challenge in building such benchmarks lies in constructing long-horizon queries that are both high-quality and faithfully reflect real-world software usage. To ensure realism, we collaborated closely with domain experts to draft seed instructions covering the core operations they routinely perform. For each instruction, we recorded detailed step-by-step workflows required to complete the corresponding task. We then filtered and refined these trajectories to retain high-frequency, high-value procedures that are critical in everyday clinical practice. For example, in medical image annotation, we focused on 3D Slicer 12 , one of the most widely adopted open-source tools for volumetric analysis, and curated representative workflows for annotation, segmentation, and measurement tasks. To enable multimodal LLMs to tackle complex, domain-specific, long-horizon workflows in healthcare software ecosystems, we propose CarePilot , a memory- and tool-augmented multi-agent framework inspired by the actor–critic paradigm 20 . At time step t , the Actor (a multimodal LM) receives the current screenshot and instruction, invokes lightweight tool modules (e.g., zoom/crop, OCR, UI/object detection) to obtain grounding signals, and predicts the next semantic action. A dual-memory design underpins the system: the long-term memory compacts the history up to t\!-\!1 (key states, actions, outcomes), and the short-term memory records the most recent decision and feedback at time t . The Critic evaluates the Actor’s proposal, updates both memories with observed effects, and issues corrective feedback, during training comparing the Actor’s action to reference traces and during evaluation relying on execution outcomes or verifier feedback. If accepted, the action advances the workflow; if revised, the Actor re-plans. At time t\!+\!1 , the Actor conditions on the refreshed memory and grounding signals to produce a more informed action. Our work makes the following key contributions: • Problem Formulation. We define a new task of long-horizon computer automation for healthcare software : given a natural-language goal and a sequence of screenshots, an agent must predict step-by-step actions to complete real clinical workflows. • Benchmark. We present CareFlow , an expert-annotated benchmark of long-horizon healthcare workflows comprising 8–24 steps for each task encompassing four major clinical systems. Each task is labeled with interface invariant semantic actions and verified using artifact/state based checks. • Framework. We propose CarePilot , a multi-agent framework built on the actor–critic paradigm that integrates tool grounding with dual memories (long and short-term) for robust next-action prediction. • Evaluation. Extensive experiments across all CareFlow domains show that CarePilot achieves state-of-the-art results, improving task accuracy upto 15.26% over strong open- and closed-source baselines. 2 Related Work Figure 2 : Overview of the CarePilot framework. An Actor–Critic multi-agent architecture governs hierarchical decision-making for long-horizon healthcare workflows. At each step, the Actor observes the