Flow-Bench : Towards Conversational Generation of Enterprise Workflows
Evelyn Duesterwald, Siyu Huo, Vatche Isahagian, K.R. Jayaram, Ritesh Kumar · 2025
Read Flow-Bench : Towards Conversational Generation of Enterprise Workflows by Evelyn Duesterwald, Siyu Huo, Vatche Isahagian, K.R. Jayaram, Ritesh Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11646.txt.
Introduction
Evelyn Duesterwald and co-authors build tools that turn plain instructions into business workflows, in a 2025 paper. Companies want software that converts a manager's words into formal process charts, and language models make that plausible. The team contributes a benchmark dataset pairing instructions with structured process definitions, so competing tools can be scored fairly.
Their own system first translates instructions into an intermediate form written like Python code, which then converts cleanly into the standard process languages used by industry. They used the benchmark to grade parts of their system across eight language models of different sizes. The authors hope the pair of contributions speeds research on process automation that novices and experts alike can drive by conversation.
Catalog record
| CREATOR | Evelyn Duesterwald, Siyu Huo, Vatche Isahagian, K.R. Jayaram, Ritesh Kumar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11646
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Flow-Bench : Towards Conversational Generation of Enterprise Workflows Evelyn Duesterwald, Siyu Huo, Vatche Isahagian, K.R. Jayaram, Ritesh Kumar Abstract Business process automation (BPA) that leverages Large Language Models (LLMs) to convert natural language (NL) instructions into structured business process artifacts is becoming a hot research topic. This paper makes two technical contributions – (i) Flow-Bench , a high quality dataset of paired natural language instructions and structured business process definitions to evaluate NL-based BPA tools, and support bourgeoning research in this area, and (ii) Flow-Gen , our approach to utilize LLMs to translate natural language into an intermediate representation with Python syntax that facilitates final conversion into widely adopted business process definition languages, such as BPMN and DMN. We bootstrap Flow-Bench by demonstrating how it can be used to evaluate the components of Flow-Gen across eight LLMs of varying sizes. We hope that Flow-Gen and Flow-Bench catalyze further research in BPA making it more accessible to novice and expert users. 1 Introduction With many enterprises relying on BPA to standardize their work, enhance their operational efficiency and reduce human error, BPA tools grew to a $11.84B industry and are projected to grow to $26B in 2028 ( Marketwatch 2022 ) . In contemporary BPA tools, users use visual drag-and-drop interfaces and reusable templates to construct workflows, decision models, and document process logic, adhering to standard notations such as BPMN ( Grosskopf et al. 2009 ; Chinosi and Trombetta 2012 ) and DMN ( Biard et al. 2015 ) . However, even sophisticated low-code BPA platforms frequently necessitate intervention from software engineers to ensure robustness, handle intricate integrations, and implement customized logic not covered by generic templates. The complexity inherent in configuring detailed integration tasks and writing low-level transformation logic remains daunting for novice programmers and tedious even for seasoned developers. Recent efforts have explored leveraging natural language interfaces to simplify BPA authoring. LLMs have demonstrated substantial potential for automating code generation tasks by translating high-level user intents into executable artifacts. However, in our experience, even state-of-the-art LLMs are not effective at generating BPA workflows, both due to the lack of BPMN training data and the extensive boilerplate notations that need to be generated. Section A.1 showcases a flow and its corresponding BPMN code. To the best of our knowledge, there is a lack of well-established benchmarks to evaluate NL-driven workflow generation. With this paper, we contribute Flow-Bench , a high quality dataset designed specifically to support research in natural language-driven business process automation, that consists of realistic utterances and their corresponding BPMN representations 1 1 1 Flow-Bench dataset can be accessed at https://github.com/IBM/Flow-Bench . The availability of this dataset is intended to catalyze model improvements, benchmarking, and development of NLP techniques tailored to the BPA domain. We also contribute Flow-Gen , an approach that leverages LLMs to first translate natural language into an intermediate representation (IR) with Python syntax to precisely capture the logic of the intended business process. The Python IR provides several advantages. It takes advantage of the innate Python code generation capabilities of LLMs and it bridges the gap between unstructured natural language and formalized business process definition languages, facilitating easier verification and refinement of the generated logic. Subsequently, our approach translates this IR into specific target process definition languages, such as BPMN-compliant XML or DMN decision tables, ensuring compatibility with existing BPA solutions and tools. Further, the use of an IR makes it easy to catch errors early and to support multiple target BPA languages. We include the Python IR in the Flow-Bench dataset, along with the BPMN. 2 Flow-Bench dataset Workflows in Flow-Bench consist of sequences of API invocations, including conditionals and loops, similar to those found in commercial workflow automation platforms like IBM App Connect IBM App Connect 2025 and Zapier Zapier Apps 2025 . In addition to API calls, workflows may incorporate manual interventions, termed user tasks . These user tasks typically involve steps within a business process that require human action, such as managerial approvals, and thus, do not have corresponding APIs. To construct Flow-Bench we initially sourced realistic business workflows from pre-existing templates provided by commercial workflow automation platforms (specifically IBM App Connect and Zapier). These templates cover common enterprise use cases, including support ticket creation, task management, and marketing campaign automation. We carried out three high level steps to arrive at the final Flow-Bench dataset: (1) quality control, (2) manual labeling, and (3) data augmentation. Quality Control: First, we collected and manually examined workflows sourced from the automation platforms. We discarded or truncated workflows that were overly complex with multiple levels of nested conditions, in order to start with relatively small workflows that a user can reasonably describe in a few sentences. We removed event triggers from workflows and discarded workflows that consisted of only a single API call. We also discarded workflows that involved APIs without publicly available OpenAPI spec. Manual Labeling: We manually added or corrected user utterance to workflows. In some cases, we rephrased existing workflow descriptions to reflect an active user command to constructing the workflow as opposed to a passive description of the workflow. The workflows often only had a proprietary representation in the respective automation tool, so we manually crafted Python IR snippets and generated the corresponding BPMN specification. We also retrieved the OpenAPI specs for all activities in the workflows and established a common naming convention for API-based worfklow activities. In addition, we added clear descriptions for each API, tweaking the descriptions in the OpenAPI specs (if available). Data Augmentation: Once we had a set of high quality samples, we expanded the dataset in two ways. First we incorporated user tasks by either adding a new activity in the workflow, or by removing the corresponding API from the catalog. Then we added samples to reflect how users may incrementally build a workflow, by adding, deleting, or replacing activities in the workflow. For each new sample, we defined the workflow before and after the edit, and the corresponding user utterance. To mimic actual software development the incremental edits may apply anywhere in the current workflow, not always editing left to right. The final complete workflows in Flow-Bench are generated through incremental build steps categorized as add , delete , or replace . Build steps in Flow-Bench are kept as self-contained tests by including three elements: (1) Prior Sequence , representing the current state of the workflow before applying changes 2 2 2 This may be empty in the initial generation step. ; (2) Utterance , describing the modification to be performed in natural language (e.g, "Retrieve all issues from the Jira board."); and (3) Expected Sequence , indicating the resulting workflow after implementing the command specified by the utterance. Flow-Bench comprises 101 incremental build step tests structured according to this methodology. Figure 1 illustrates an example test case from Flow-Bench . Each build step is uniquely identified and provides a self-contained test scenario, including explicit BPMN representations of both the Prior Sequence and Expected Sequence . Additionally, each test is annotated with metadata describing the build step type ( add , delete , replace ), the control flow structure ( linear or conditional , where conditional encompasses both if-statements and loops), and the inclusion