Learning Selective LLM Autonomy from Copilot Feedback in Enterprise Customer Support Workflows

Nikita Borovkov, Elisei Rykov, Olga Tsymboi, Sergei Filimonov, Nikita Surnachev · 2026

Read Learning Selective LLM Autonomy from Copilot Feedback in Enterprise Customer Support Workflows by Nikita Borovkov, Elisei Rykov, Olga Tsymboi, Sergei Filimonov, Nikita Surnachev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23855.txt.

Introduction

Borovkov and six colleagues deploy a system that automates enterprise customer-support workflows inside a business process platform. It learns a next-action policy from interface traces, trains a critic on copilot feedback to set abstention levels, and runs only high-confidence steps in the background, deferring uncertain calls to operators. One operator can thus supervise many sessions at once. In production it automated 45 percent of sessions and cut handling time by nearly two-fifths with no quality loss.

Catalog record

CREATORNikita Borovkov, Elisei Rykov, Olga Tsymboi, Sergei Filimonov, Nikita Surnachev
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61699
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23855

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Selective LLM Autonomy from Copilot Feedback in Enterprise Customer Support Workflows Nikita Borovkov, Elisei Rykov, Olga Tsymboi, Sergei Filimonov, Nikita Surnachev Abstract We present a deployed system that automates end-to-end customer support workflows inside an enterprise Business Process Management (BPM) platform. The approach is scalable in production and reaches selective automation within two weeks for a new process, leveraging supervision already generated at scale: structured per-case UI interaction traces and low-overhead copilot feedback, where operators either accept a suggestion or provide a correction. A staged deployment pipeline trains a next UI action policy, learns a critic from copilot feedback to calibrate abstention, and executes only high-confidence steps in the background while deferring uncertain decisions to operators and resuming from the updated UI state. This setup lets one operator supervise multiple concurrent sessions and be interrupted only when the system is uncertain. The system operates on a schema-driven view of the BPM interface and includes monitoring and safe fallbacks for production. In production, it automated 45% of sessions and reduced average handling time by 39% without degrading support quality level. 1 Introduction LLM-based agents promise end-to-end automation by reasoning and acting with tools ( 37 ; 25 ) and show strong results on web and GUI interaction benchmarks ( 34 ; 3 ) . However, deploying such agents in enterprise customer support remains hard: they interleave many model calls with UI actions, so small errors propagate over long horizons; multi-step control loops are costly at scale, and frequent UI and process changes require conservative uncertainty handling to avoid quality regressions. Traditional automation avoids some of these risks but is expensive to build and maintain, relying on manually authored dialogue flows ( 20 ) or brittle Robotic Process Automation (RPA) scripts ( 14 ) . We present a deployed alternative for an enterprise BPM system: instead of hand-authoring automation, we learn workflows directly from operator behavior, reaching selective automation within two weeks without manual rule authoring. Our setting provides two scalable supervision signals: each case yields structured UI interaction traces, and copilot mode provides lightweight feedback: operators either accept a suggestion or correct it. Using a staged deployment procedure inspired by imitation learning ( 24 ; 23 ) , we progress from offline training on traces to copilot supervision and then to gated background execution, where only high-confidence actions are automated. The system is fully deployed and serves production traffic; we report online A/B results on approximately 15k customers per group, demonstrating a 39% reduction in operator active time without degrading support quality. Our system combines an LLM policy trained on serialized state–action traces, a critic trained on trajectory feedback labels to support abstention ( 7 ; 13 ) , and a controller that executes only above a calibrated confidence threshold, otherwise deferring to the operator and resuming from the updated UI state. To improve robustness under UI change, the system operates on a schema-driven JSON representation of the BPM interface rather than screenshots. Our main contribution is a production-scalable staged deployment process that turns BPM UI traces and lightweight copilot feedback into safe, gated background execution. We also report the key engineering components—schema-driven state/action serialization, deployment safeguards (abstention, data drift handling, fallbacks, monitoring), and an online evaluation protocol with ablations—together with measured gains in operator active time. 2 Related Work Customer support automation has long relied on deployed dialogue and agent-assist systems that suggest replies, retrieve internal knowledge, or summarize conversations for operators 5 ; 17 ; 6 . Newer datasets also capture customer-support interactions, from transactional dialogues to more natural service conversations 1 ; 11 . Most prior work targets conversation-level assistance or knowledge access, while our setting requires safe execution of multi-step UI procedures inside an enterprise BPM interface. Early approaches framed UI and workflow control as behavior cloning from interaction logs 19 . Over long horizons, covariate shift makes errors compound, which motivates interactive imitation learning methods such as DAgger and AggreVaTe that gather expert supervision on states reached by the learned policy 24 ; 23 ; 30 . GUI agents also differ in what they observe. Text-based methods act on structured interface signals such as HTML, DOM, or accessibility trees 9 ; 12 ; 15 , while multimodal agents additionally use screen information 16 ; 35 ; 31 . In deployed systems, raw DOM is often verbose and fragile under UI changes, which motivates intermediate representations and stronger action grounding 33 ; 10 . We use the more controlled nature of enterprise BPM to extract a schema-driven JSON state from the DOM that keeps only task-relevant entities and actionable controls, enabling monitoring and reliable tool calling without screenshots. This aligns with industry efforts that restructure support policies and workflows into LLM-friendly formats 29 . Our deployment loop is DAgger-like. Operator rejections trigger corrective supervision in the same UI state, and we execute selectively with low-confidence handoff to reduce drift over long procedures. Instead of preference-based alignment methods, we use lightweight operator feedback to train a critic and manage the error–coverage trade-off needed for safe background automation 26 ; 21 ; 27 . 3 Framework Figure 1: Staged deployment for selective automation. Logging collects operator trajectories with UI context to train an initial policy. Copilot deploys the policy to propose actions with human oversight on critical steps, producing accept or override feedback. Selective automation adds a critic that scores policy proposals and executes only critic-approved critical actions. This stage runs in calibration with mandatory human review at the finalizing step and transitions to automation where sessions can be executed end-to-end under calibrated threshold(s) \tau . Our approach involves a staged deployment algorithm, progressing through three distinct phases: logging, copilot, and selective automation, each building on the data gathered in the previous stage, allowing for gradual integration into the production environment. The deployment is designed to collect data continuously at each stage, ensuring that each transition is based on robust operational feedback. Problem Setting Operators resolve customer issues via a BPM system integrating full-history chat, scenario search, a workflow workspace for guided UI interactions (e.g., credit requests, account checks, data updates), and auxiliary tabs showing customer products and statuses. Multiple scenarios can run concurrently, with operators interleaving structured UI actions and free-form chat, relying on both encoded business rules and tacit knowledge. A support session is a trajectory of states s_{t} and actions a_{t} : \{s_{t},a_{t}\}_{t=1}^{T} , where s_{t} comprises chat, UI, scenario, and auxiliary context, and a_{t} spans UI navigation and interaction, tool invocation, chat messages, or session closure. The automation objective is selective: maximize automated step coverage while deferring uncertain actions to humans to maintain correctness and customer experience. Data Operator sessions are fully logged, capturing UI state, events such as clicks, edits, submissions, and procedure openings, chat streams, and metadata including timestamps, business line, and routing details. These traces are serialized into structured state-action pairs for behavior cloning. State serialization produces JSON snapshots of the last 30 chat messages, relevant UI context such as customer profile, active scenario, and global announcements, along with operator actions. Actions fall into four categories — message composition, navigation, selection and form-filling, and session management — each targeting a specific interface element identified by its unique control_id (see Table 4

Read the complete text