Agent-S: LLM Agentic workflow to automate Standard Operating Procedures

Mandar Kulkarni · 2025

Read Agent-S: LLM Agentic workflow to automate Standard Operating Procedures by Mandar Kulkarni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15520.txt.

Introduction

Mandar Kulkarni proposes in 2025 an AI agent that carries out standard operating procedures on its own. In customer care, such a procedure spells out the steps a human agent takes to solve a case. Each step is either a talk with the user or a call to a software tool, and the chart of steps sets the path.

His design, Agent-S, uses three task-tuned language models backed by a shared action store, a memory of past steps, and links to tools and knowledge sources. When stuck, the agent can retry a step or look up help. Tests on three seller support procedures from online retail showed the agent held up in messy, real-style cases.

Catalog record

CREATORMandar Kulkarni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15520

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Agent-S: LLM Agentic workflow to automate Standard Operating Procedures Mandar Kulkarni Abstract AI agents using Large Language Models (LLMs) as foundations have shown promise in solving complex real-world tasks. In this paper, we propose an LLM-based agentic workflow for automating Standard Operating Procedures (SOP). For customer care operations, an SOP defines a logical step-by-step process for human agents to resolve customer issues. We observe that any step in the SOP can be categorized as user interaction or API call, while the logical flow in the SOP defines the navigation. We use LLMs augmented with memory and environments (API tools, user interface, external knowledge source) for SOP automation. Our agentic architecture consists of three task-specific LLMs, a Global Action Repository (GAR), execution memory, and multiple environments. SOP workflow is written as a simple logical block of text. Based on the current execution memory and the SOP, the agent chooses the action to execute; it interacts with an appropriate environment (user/API) to collect observations and feedback, which are, in turn, inputted to memory to decide the next action. The agent is designed to be fault-tolerant, where it dynamically decides to repeat an action or seek input from an external knowledge source. We demonstrate the efficacy of the proposed agent on the three SOPs from the e-commerce seller domain. The experimental results validate the agent’s performance under complex real-world scenarios. 1 Introduction Large Language Models (LLMs) have demonstrated great potential in logical reasoning. However, LLMs fall short when tackling more sophisticated tasks that involve interaction with multiple environments. AI agents built on LLMs can control the path to solving a more complex task. Figure 1: Proposed architecture of the SOP Agentic workflow. A Standard Operating Procedure (SOP) defines a systematic workflow for human agents. It is a pre-defined step-by-step process designed to help human agents resolve recurring customer issues. When a consumer reaches out to customer care, a human agent identifies the issue (i.e., intent) and follows the sequence of steps from the SOP. While analyzing the SOPs and customer-human agent interactions from the historical e-commerce data, we observed that any step in the SOP can be categorized as follows. 1. User interaction : In this, human agents interact with the consumers. They ask the user relevant questions, interpret the answers, collect relevant information (e.g., product/listing ID, etc), and convey messages. 2. Status check : Based on the information collected from the user, human agents check the status of product/listing ID on the dashboard (where data is populated through backend API calls). Based on the output of the user interaction or status check, an agent decides the next action as per the SOP. If the current step in the SOP is executed successfully, the agent proceeds to the next step. The next action would again involve a user interaction or status check. If the current step has failed, an agent attempts to complete it either by repeating the current step or any of the previous dependent steps e.g., if during a status check, the agent realizes that the user has provided an invalid listing ID, the agent would repeat the step of asking user for the listing ID. In this paper, we propose an LLM based agentic workflow for executing SOPs to automate a human agent process for customer care support. Both user interaction and status check steps can be automated using LLMs augmented with external environments such as API tools and user interface. The LLMs can generate the question to the user, it can interpret the user inputs, extract relevant information, it can generate status messages and acknowledgments. The status check step can be automated with the combination of API tools and LLMs. An API tool can help retrieve the information from the database, and an LLM can effectively utilize it to make decisions. Equipped with these capabilities, LLMs-based agents can, therefore, help to navigate the entire SOP flow in an automated manner. Technically, the SOP can be thought of as a Directed Acyclic Graph (DAG), where each node represents the sop step (state), and the edges represent the possible branches from the current state. Fig. 1 shows the proposed architecture of the SOP agent. The architecture consists of three LLM components, a State-Decision-LLM: to decide the next action (state) in the SOP, an Action-Execution-LLM: to help execute the currently selected action and a User-Interaction-LLM: to interpret user inputs and provide acknowledgments. The inputs to the state decision LLM are the sop workflow and execution memory. In our approach, the SOP workflow can simply be written as a logical block of text with indents depicting sub-flows. Table 1 shows an SOP for listing blocked issue from the e-commerce seller domain. Execution memory records the history of execution where each entry contains the selected action and its corresponding observation and feedback. The observation and feedback are obtained from the environment with which the agent interacts to execute the action. In the case of an API call, observation contains the (parsed) API response or error message, while if asking for the user’s input, the observation contains the spell corrected user’s input from the user-interaction-LLM. Feedback indicates whether the current action has been executed successfully or not. The Global Action Repository (GAR) table contains a set of possible actions and the corresponding required information to execute the actions. Using sop workflow and current execution memory, the state-decision-LLM decides the next action to be executed. GAR provides the required info to action-execution-LLM to execute the action. Depending on the type of the action, the action-execution-LLM interacts with API tool or user interface. The observation and feedback are generated and inputted to the execution memory, and the state-decision- LLM predicts the next action. The process is repeated till the termination. From historical user-human interactions data, we also noticed that users often ask questions/doubts regarding the requested information, e.g., when asked to provide a listing ID, they typically ask where/how to find their listing ID. We incorporate an external knowledge source in the architecture to seamlessly answer the user queries within the SOP flow. We use an in-house Retrieval Augmented Generation (RAG) system with domain documents to get the answer. When the agent identifies that the user is asking a question, it formulates a search query, fetches an answer from the RAG, and continues the SOP flow from where it branched. To validate the effectiveness of the agent, we extensively experimented with three SOPs from the seller domain: why the listing is blocked, want to update the email ID, and why brand approval is rejected. We evaluated the SOP agent on synthetic and live chat sessions. To mimic real world scenario, we synthetically generated a large variation of user inputs, all possible API responses and failure scenarios and verified the robustness of the agent. After a successful synthetic evaluation, we conducted in-house experiments with live chat sessions. The experimental results demonstrated that the LLM agent can effectively navigate the SOP flow. The agent is scheduled to go live for all sellers after engineering integration. We provide all LLM prompts in the Appendix for replicability. 2 LLM Agentic workflow for SOP automation Fig. 1 shows the proposed architecture. The architecture has the following main components. 2.1 Global Action Repository

Read the complete text