StarFlow : Generating Structured Workflow Outputs From Sketch Images
Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian · 2025
Read StarFlow : Generating Structured Workflow Outputs From Sketch Images by Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21889.txt.
Introduction
Drawing a work plan on paper is fast, but turning that sketch into steps a computer can run is slow hand work. Patrice Bechard and co-authors present StarFlow in a 2025 paper, a system that reads a sketch or diagram and writes out the matching workflow. Their test bed mixes made-up diagrams, hand-marked ones, and real ones from work sites.
The team tuned several image-and-text models on this mixed set and ran checks to see where each one failed. Tuning helped a great deal, and the tuned models beat much larger general models on the task. Hard parts stay hard: loose freehand lines, odd diagram styles, and hidden logic are easy for a person to read and tough for a model. The paper maps both the gains and those limits.
Catalog record
| CREATOR | Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21889
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StarFlow : Generating Structured Workflow Outputs From Sketch Images Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian Abstract Workflows are a fundamental component of automation in enterprise platforms, enabling the orchestration of tasks, data processing, and system integrations. Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. To simplify this process, we explore the use of generative foundation models, particularly vision-language models (VLMs), to automatically generate structured workflows from visual inputs. Translating hand-drawn sketches or computer-generated diagrams into executable workflows is challenging due to the ambiguity of free-form drawings, variations in diagram styles, and the difficulty of inferring execution logic from visual elements. To address this, we introduce StarFlow , a framework for generating structured workflow outputs from sketches using vision-language models. We curate a diverse dataset of workflow diagrams—including synthetic, manually annotated, and real-world samples to enable robust training and evaluation. We finetune and benchmark multiple vision-language models, conducting a series of ablation studies to analyze the strengths and limitations of our approach. Our results show that finetuning significantly enhances structured workflow generation, outperforming large vision-language models on this task. All resources, including models, code, datasets, and evaluation metrics, are made available 1 1 1 https://servicenow.github.io/StarFlow/ . 1 Introduction Workflows play a crucial role in automating business processes, orchestrating data flows, and integrating enterprise applications. They enable organizations to streamline operations, reduce manual effort, and enforce business logic across complex systems MuleSoft, a Salesforce Company (2025) ; ServiceNow (2025) ; Microsoft (2025) . Despite their ubiquity, workflow creation remains a challenging task, often requiring users to manually configure processes through low-code platforms or visual programming environments. While these tools offer greater accessibility than traditional programming, they still demand a deep understanding of system logic, data dependencies, and execution rules. An intuitive alternative would be the ability to generate structured workflows directly from visual representations, such as hand-drawn sketches or diagrams, as portrayed in Figure 1 . However, this problem is inherently difficult due to the ambiguity of sketches, variations in diagramming conventions, and the complexity of extracting structured execution logic from visual elements. Figure 1: The task of sketch to workflow. Given an input image representing a business process, the task is to convert the logic of the diagram found in the image into a structured JSON output describing the execution logic of the workflow, including the appropriate trigger, actions, and inputs. In this work, we introduce StarFlow , a framework designed to generate structured workflow representations from sketch-based inputs using vision-language models (VLMs). Our approach involves curating a diverse dataset comprising synthetic, manually annotated, and real-world workflow diagrams, which we use to finetune multiple vision-language models. To evaluate the performance of our approach, we use a flow similarity metric that measures the structural fidelity of generated workflows based on the tree representation of the workflow and tree edit distance. Our results demonstrate that finetuning significantly enhances the ability of VLMs to generate structured workflows, outperforming general-purpose models on this specialized task. By addressing the limitations of existing workflow creation methods and demonstrating the effectiveness of vision-language models in this domain, our work represents a step toward making workflow automation more intuitive and accessible. Our key contributions are as follows: • We introduce StarFlow , a framework for converting hand-drawn and digital workflow sketches into structured representations, enabling seamless workflow automation. • We build a diverse dataset of workflow diagrams, spanning synthetic, human-annotated, and real-world samples, to enhance training and evaluation. To foster future research and reproducibility, we open-source all resources , including models and their implementations, code used for training, datasets, and evaluation metrics. 2 Related Work 2.1 Structured Output and Code Generation Language models trained on code ( Chen et al., 2021 ; Li et al., 2023 ; Roziere et al., 2023 ; Hui et al., 2024 ; Zhu et al., 2024 , e.g.) have seen significant advancements in recent years, improving various aspects of software development, including code generation Nijkamp et al. (2022) ; Jiang et al. (2024) ; Rodriguez et al. (2024a) , comprehension Feng et al. (2020) ; Lu et al. (2021) , and translation tasks Lachaux et al. (2020) ; Yan et al. (2023) . These models utilize large-scale source code datasets ( Kocetkov et al., 2022 , e.g.) to learn programming syntax and semantics, enabling them to produce functional and syntactically correct code snippets from natural language prompts. Evaluation of code generation models is notoriously difficult. Metrics such as the HumanEval benchmark Chen et al. (2021) aim to evaluate a model’s ability to generate functionally correct code solutions. Other metrics, such as CodeBLEU Ren et al. (2020) , extend the traditional BLEU score Papineni et al. (2002) by incorporating code-specific features such as syntax and data flow, offering a more nuanced evaluation of code generation quality. In this work, we draw inspiration from CodeBLEU and introduce a flow similarity metric based on tree representation and tree edit distance Zhang and Shasha (1989) . 2.2 Multimodal Large Language Models Vision-language models (VLMs) ( Alayrac et al., 2022 ; Liu et al., 2023 ; Agrawal et al., 2024 ; Wang et al., 2024 ; Dubey et al., 2024 , e.g.) have made significant strides in integrating visual and textual data, enabling more sophisticated multimodal understanding. These models excel at various tasks, including image captioning Lin et al. (2014) ; Vinyals et al. (2015) , visual question answering Antol et al. (2015) ; Hudson and Manning (2019) ; Yue et al. (2024) , and document understanding Rodriguez et al. (2024b) ; Tong et al. (2025) . One task that remains challenging for VLMs is to generate code or structured outputs based on a screenshot or diagram Liu et al. (2022) ; Shukla et al. (2023) ; Shi et al. (2025) ; Rodriguez et al. (2024b) ; Herrera-Camara and Hammond (2017) . For example, Shi et al. (2025) introduce a benchmark to assess the performance of VLMs on generating code to reproduce charts. Closely related to our work, Liu et al. (2022) propose a two-step process to generate code from a flowchart via two distinct models, one to extract the structure of the diagram, and the other to generate executable code from pseudocode. In this paper, we focus on generating structured workflows in JSON format from hand-drawn or computer generated sketches. 2.3 Workflow Generation Recent work on workflow generation from textual inputs has demonstrated significant advancements in the field of automated task planning and execution. Approaches relying on retrieval-augmented generation and task decomposition have been shown to be effective for solving this problem Béchard and Ayala (2024) ; Bassamzadeh and Methani (2024) ; Ayala and Béchard (2024) . Other notable efforts include Zeng et al. (2023) , who built models to generate workflows for specific applications, Fan et al. (2024) who develop a synthetic data pipeline used to train a workflow generator, and Cai et al. (2023) who built a graphical user interface allowing a user to build and edit a workflow with the assistance of an LLM. In this work, we focus on generating workflows from hand-drawn sketches and computer generated diagrams instead of