Towards Computer-Using Personal Agents
Piero A. Bonatti, John Domingue, Anna Lisa Gentile, Andreas Harth, Olaf Hartig, Aidan Hogan, Katja Hose, Ernesto Jimenez-Ruiz, Deborah L. McGuinness, Chang Sun, Ruben Verborgh, Jesse Wright · 2025
Read Towards Computer-Using Personal Agents by Piero A. Bonatti, John Domingue, Anna Lisa Gentile, Andreas Harth, Olaf Hartig, Aidan Hogan, Katja Hose, Ernesto Jimenez-Ruiz, Deborah L. McGuinness, Chang Sun, Ruben Verborgh, Jesse Wright (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15515.txt.
Introduction
Piero Bonatti and a large team set out a plan for personal agents that use a computer's screen tools, with a store of the user's own data held apart. Their paper is from two thousand twenty-five. Plain agents of this kind drive apps such as browsers to do tasks for a user. Many tasks need personal data. A shared store gives the user more grip on that data. Such agents can also do more tasks and team up for joint work.
Catalog record
| CREATOR | Piero A. Bonatti, John Domingue, Anna Lisa Gentile, Andreas Harth, Olaf Hartig, Aidan Hogan, Katja Hose, Ernesto Jimenez-Ruiz, Deborah L. McGuinness, Chang Sun, Ruben Verborgh, Jesse Wright |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15515
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Computer-Using Personal Agents Piero A. Bonatti, John Domingue, Anna Lisa Gentile, Andreas Harth, Olaf Hartig, Aidan Hogan, Katja Hose, Ernesto Jimenez-Ruiz, Deborah L. McGuinness, Chang Sun, Ruben Verborgh, Jesse Wright Abstract . Computer-Using Agents (CUA) enable users to automate increasingly-complex tasks using graphical interfaces such as browsers. As many potential tasks require personal data, we propose Computer-Using Personal Agents (CUPAs) that have access to an external repository of the user’s personal data. Compared with CUAs, CUPAs offer users better control of their personal data, the potential to automate more tasks involving personal data, better interoperability with external sources of data, and better capabilities to coordinate with other CUPAs in order to solve collaborative tasks involving the personal data of multiple users. 1. Introduction Advances in Generative AI, and particularly Large Language Models (LLMs), have led to the recent release of various Computer-Using Agents ( CUAs ) that automatically operate a user’s computer on their behalf. These agents use multimodal capabilities to interact with graphical interfaces via simulated mouse and keyboard inputs. Prominent commercial examples of CUAs include OpenAI’s Operator, Google’s Jarvis, and new functionalities in Anthropic’s Claude. Potential use cases for CUAs involve personal and often sensitive data, such as credit card details for purchases, passport numbers for flight booking, addresses for deliveries, and allergy information for dinner reservations. While modern browsers sometimes store personal data to autocomplete web forms, CUAs could additionally take context into account (e.g., selecting between a home or work address, depending on the purchase) and go beyond simple autocompletion. Passing personal data to CUAs raises valid concerns about how such data might be (mis)used. Currently, OpenAI’s Operator invokes a takeover mode for tasks involving sensitive data (e.g., log-in or payment details): the user is required to fill the details in manually ( Operator2025 ) . Such measures target users’ concerns about how their personal information will be used by CUAs. OpenAI themselves state that Operator is “still learning, evolving and may make mistakes” ( Operator2025 ) . There are thus many open questions relating to the use of personal user data by CUAs. Conversely, there are many potential benefits to users if CUAs are empowered with personal data. CUAs could autofill forms with personal data for users in a context-aware and potentially generative manner, automating a tedious task. CUAs could potentially enrich personal data with public data to better solve tasks. The CUAs of multiple users could negotiate to achieve a mutually beneficial result based on their users’ personal context and preferences. Towards providing users more oversight over their personal data while enabling higher levels of automation for complex tasks, we propose Computer-Using Personal Agents ( CUPAs ): a Computer-Using Agent (CUA) that has controlled access to a structured repository of private information relating to a user . This concept is illustrated in Figure 1 . Specifically, we propose to instantiate the repository as a Personal Knowledge Graph ( PKG ) representing the user’s personal data, which would facilitate the specification by users on how the CUA can access and use these data. This PKG can collect more personal data over time, with policies also evolving to reflect the user’s fluctuating trust in the system ( afroogh2024trust ) . Looking further forward, one can then imagine a scenario where CUPAs interact with websites and services via the underlying Web APIs instead of through a vision model, where CUPAs can assist in recommendations and negotiations based also on interactions with similar users and/or users’ CUPAs. We provide a road-map towards realising this vision of CUPAs, discussing what is achievable now with current technology, and what gaps must be addressed via further research and development. Figure 1. Computer-Using Personal Agent Computer-Using Personal Agent "The image depicts the flow of Computer-Using Personal Agent." 2. User Scenario Sam is expecting Jane over for dinner at 8pm, and is thinking about preparing Thai food. Sam is pre-diabetic, while Jane has a shellfish allergy. Sam requests that his CUPA to generates some suggestions of Thai recipes for the occasion. Consulting Sam’s schedule, the CUPA recommends to filter recipes requiring more than an hour to prepare based on when he finishes work and his commute time. Sam agrees, and the CUPA starts to retrieve and present shellfish-free recipes of Thai food that are quick to prepare. Upon consulting external sources of nutritional information and recipes on the Web, the CUPA flags some recipes as being above the postprandial glucose threshold recommended by Sam’s doctor (<180 mg/dL), or as having high glycaemic indices (>70). Sam asks his CUPA to find out what recipe Jane might like. As Sam and Jane are friends, Sam’s CUPA can send the candidate recipes to Jane’s CUPA to see what she might like. Jane’s CUPA suggests to avoid some recipes that include coriander (listed in some recipes as cilantro), which Jane hates. Sam’s agent enforces his glucose thresholds and flags ingredients with high glycaemic indices, using external food and recipe knowledge graphs (e.g., the FoodKG ( haussman2019 ) ) to find the alternative ingredients. Of the remaining options, Sam’s agent suggests a tofu green curry recipe that catches Sam’s eye. Since the recipe is flagged for having a high glycaemic index (78), the agent asks Sam if he might consider replacing jasmine rice with cauliflower rice as a healthier option. Sam refuses the substitution as it is a special occasion. Sam requests his CUPA to order the ingredients from a local supermarket. Since green bell peppers are out of stock, the CUPA suggests to replace them with yellow bell peppers. Sam agrees, and the CUPA prepares the order for delivery to Sam’s home address, soliciting Sam’s confirmation. Later that night, Sam and Jane enjoy their dinner of Thai green curry. After Jane leaves, Sam suffers some slight heartburn. He requests his CUPA to order antacids and additionally registers the fact that green curry dishes may cause Sam heartburn for future reference. 3. State of the Art Personal data play an increasingly important role in modern life ( Mortier2016 ; Birch2021 ) . Early works ( Jones2012 ) characterise such data based on the concept of six senses : owned by me, about me, directed to me, sent by me, already experienced by me, and useful to me. More restrictive definitions only include data created by the individual ( Bergman2016 ) , or that the individual cares for/about ( 24824 ; Cushing2022 ) . Much literature has been dedicated to Personal Information Management systems (PIMs), which deal with the acquisition, organisation, maintenance, retrieval and sharing of personal data ( jones2007personal ) . Notable PIM technologies include blockchain systems ( Zyskind2015 ; Faber2019 ) , systems capturing user behaviour on multiple user devices ( Lin2022 ; Schroder2022 ) , and end-user prototypes ( Chaudhry2015 ; Mortier2016 ; Kalokyri2018 ) . Personal Knowledge Graphs (PKGs) ( Chakraborty2022 ; Chakraborty2023 ; skjaeveland2024ecosystem ) further apply a graph abstraction to personal data, opening up possibilities for declarative access policies, deductive inference, and integration with external Knowledge Graphs. Towards taking fuller advantage of such data, AI-powered agents show much promise, particularly those that can automate tasks currently performed by the user. Robotic Process Automation (RPA) ( Aalst ; Silva2022 ) automates interactions with