Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
Gaole He, Gianluca Demartini, Ujwal Gadiraju · 2025
Read Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant by Gaole He, Gianluca Demartini, Ujwal Gadiraju (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-01390.txt.
Introduction
He, Demartini, and Gadiraju test trust and team ends when chat aides plan then act. Chat tools have spread to day life since ChatGPT's rise in fame. With out tools for set jobs, aides can aid hands in day work. Those jobs span flight books and clock tools in the day set. Aides show hope as day aids, yet trust facts stay dim. Plan and step acts by such aides ground that dim zone.
Past work backs man-in-loop plan forms for hard plan tasks. Their test took 248 hosts in a sim world for day tasks. Six stock tasks, with split risk tiers, formed the test frame. Flight books and card pays sit in that risk set in the work. Aides plan step by step, then act step by step there. Host say at each stage forms the trust read base.
Catalog record
| CREATOR | Gaole He, Gianluca Demartini, Ujwal Gadiraju |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-01390
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant Gaole He, Gianluca Demartini, Ujwal Gadiraju Abstract . Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose ( e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of ‘ LLM-modulo ’ setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study ( N N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them ( e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword — (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents. Conference: CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706598.3713218 ISBN: 979-8-4007-1394-1/25/04 CCS: Human-centered computing Empirical studies in HCI CCS: Computing methodologies Artificial intelligence Figure 1. Illustration of the human-AI collaboration with plan-then-execute LLM agents. Illustration of the human-AI collaboration with plan-then-execute LLM agents. The planning stage mainly consists of two steps: (1) LLMs generate a draft plan (2) user edit the planning. Based on the edited plan, we get a step-wise plan (three steps in illustration). Then, in the execution stage, LLM agents will generate a sequence of actions, which is one-on-one mapping based on the plan's primary steps. Users will join this process to decide whether they approve the predicted action or choose to be involved (with manual specification of action or give text feedback to LLM agents). After the step-by-step execution of the plan, the task is solved. † † cc-license: by 1. Introduction Autonomous agents have been regarded as a research focus for artificial intelligence (AI) over the last century ( Albrecht and Stone, 2018 ) . With the wish that autonomous agents can make our life better, many autonomous agents have been designed as virtual personal assistants ( Kepuska and Bohouta, 2018 ) . These AI assistants ( e.g., Siri) perform well (albeit imperfectly) in following user instructions to execute low-risk tasks like playing a song, reporting weather forecasts, or searching for an image to support everyday tasks. However, on tasks entailing potential risks ( e.g., monetary payments or hiring an employee), humans hesitate to trust such AI systems due to loss aversion ( Tversky and Kahneman, 1991 ) and algorithmic aversion ( Hou and Jung, 2021 ; Mahmud et al., 2022 ; Dietvorst et al., 2015 ) . Only when users can obtain a sense of control by being able to modify the outcomes of imperfect AI can they overcome such algorithm aversion and be willing to collaborate with imperfect AI systems ( Dietvorst et al., 2018 ) . With the recent rise of large language models (LLMs) in natural language understanding and generation ( Zhao et al., 2023 ) , researchers have started to analyze LLM-based agents and their applicability in a plethora of tasks ( Xi et al., 2023 ; Wang et al., 2024b ) . The term ‘ LLM agent ’ refers to an artificial entity based on LLMs that perceives its context, makes decisions, and then takes actions in response ( Xi et al., 2023 ) . Compared to existing deep learning and LLM-based methods ( e.g., chaining multiple LLMs ( Wu et al., 2022 ) ), LLM agents provide more flexibility in task solving and user interaction, which makes them suitable for daily assistance. This is primarily due to three reasons. First, with a planning module, LLM agents can generate a dynamic plan based on the tools provided ( Xi et al., 2023 ; Wang et al., 2024b ) . Such plans are typically defined in a logical structure — step-wise plans, which can be easily understood by humans. Second, with LLMs as a core control module, users can access and interact with external toolkits via a more natural interaction ( i.e., conversation) with LLM agents ( Bommasani et al., 2021 ; Zhao et al., 2023 ) , reducing manual control efforts over function-specific tools. For example, LLM agents can complete time-consuming jobs like information seeking and information filtering ( e.g., searching for a flight in itinerary planning) based on specific user needs. Third, the Markov decision process of LLM agents can generate a sequence of actions ( i.e., using external toolkits) 1 1 1 In our study, the usage of one tool is the same as executing one action. Therefore, we refer to a tool and action interchangeably. as output. Paired with an understanding of actions and necessary parameters for the interaction with the LLM agents, users can get involved in the real-time execution of tasks with LLM agents and fix potential problems while benefiting from task delegation ( Lubars and Tan, 2019 ) . Based on an intuitive framework for task delegation, Lubars et al. ( Lubars and Tan, 2019 ) found that user trust can play an important role in human delegation behaviors to AI systems. However, there is a relatively limited understanding of user trust development and calibration in collaboration with LLM agents. There is also a growing debate in the machine learning and AI research communities about whether LLMs can be truly considered as reasoning and planning agents ( Kambhampati et al., 2024 ) . With this in the backdrop, existing work on automated task completion has revealed that LLM agents can exhibit promising performance in handling complex tasks like playing games ( Wang et al., 2023a ) , answering complex questions ( Zhuang et al., 2024 ) , and in simulating social behavior ( Park et al., 2023 ) . However, such agents are still far from perfect. Due to the probabilistic nature of LLMs, there is much uncertainty in automating LLM agents for tasks with high risks attached. To avoid unintended or unexpected consequences, there is a need for user control over the real-time execution process. Through an empirical study of LLM planning capabilities, planning experts found that “LLMs’ ability to generate executable plans autonomously is rather limited” ( Valmeekam et