Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding

Junyi Ye, Ankan Dash, Wenpeng Yin, Guiling Wang · 2024

Read Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding by Junyi Ye, Ankan Dash, Wenpeng Yin, Guiling Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16420.txt.

Introduction

Junyi Ye, Ankan Dash, Wenpeng Yin, and Guiling Wang proposed TextFlow in twenty twenty-four, a two-stage method for answering questions about flowcharts. Flowcharts come as images, and image-aware AI models read them end to end. That leaves users little control and hides where errors arise, in seeing or in reasoning.

TextFlow first turns the chart image into text, in formats like Graphviz or Mermaid that the user can pick. A second stage, a text reasoner, answers questions from that text alone, and the text can even be run as a graph object with tools. Errors can now be traced to the seeing stage or the reasoning stage. Strong language models can be swapped into the second stage. Tests on two flowchart benchmarks gave top scores and steady results.

Catalog record

CREATORJunyi Ye, Ankan Dash, Wenpeng Yin, Guiling Wang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16420

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding Junyi Ye, Ankan Dash, Wenpeng Yin, Guiling Wang Abstract Flowcharts are typically presented as images, driving the trend of using vision-language models (VLMs) for end-to-end flowchart understanding. However, two key challenges arise: (i) Limited controllability —users have minimal influence over the downstream task, as they can only modify input images, while the training of VLMs is often out of reach for most researchers. (ii) Lack of explainability —it is difficult to trace VLM errors to specific causes, such as failures in visual encoding or reasoning. We propose TextFlow , addressing aforementioned issues with two stages: (i) Vision Textualizer —which generates textual representations from flowchart images; and (ii) Textual Reasoner —which performs question-answering based on the text representations. TextFlow offers three key advantages: (i) users can select the type of text representations (e.g., Graphviz , Mermaid , PlantUML ), or further convert them into executable graph object to call tools, enhancing performance and controllability ; (ii) it improves explainability by helping to attribute errors more clearly to visual or textual processing components; and (iii) it promotes the modularization of the solution, such as allowing advanced LLMs to be used in the Reasoner stage when VLMs underperform in end-to-end fashion. Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow ’s state-of-the-art performance as well as its robustness. All code is publicly available 1 1 1 GitHub: https://github.com/JunyiYe/TextFlow . 1 Introduction Flowcharts are extensively used to represent processes, algorithms, and workflows across a range of domains, including software engineering, business process modeling, and education. Accurate interpretation of flowcharts is essential for tasks such as automation, decision-making, and analysis. Given that flowcharts predominantly exist as images, the rise of large language models (LLMs) and large vision-language models (VLMs) has led to the use of VLMs for flowchart understanding in an end-to-end manner ( Tannert et al. 2023 ; Pan et al. 2024 ; Singh et al. 2024 ) . While end-to-end VLMs offer a straightforward approach to flowchart understanding, they exhibit two key limitations: (i) Limited controllability —users have minimal capacity to improve performance, as they can only manipulate input images, while training VLMs is resource-intensive and often inaccessible to most researchers. (ii) Lack of explainability —it is difficult to trace errors in VLM outputs to specific failures, whether in visual encoding, reasoning, or other stages. To overcome these challenges, we introduce TextFlow , a framework that decomposes flowchart understanding into two stages: (i) Vision Textualizer , which generates intermediate textual representations from flowchart images; and (ii) Textual Reasoner , which performs question-answering (QA) based on these text representations. This dual-stage framework provides three distinct advantages: (i) Controllability —users can flexibly choose the type of text representation (e.g., Graphviz , Mermaid , or PlantUML ) and convert them into executable graph objects for enhanced performance; (ii) Explainability —it improves error attribution by clarifying whether failures arise from visual or textual components; and (iii) Modularity —the framework allows for the use of more advanced LLMs in the Reasoner stage if VLMs are inadequate for end-to-end flowchart understanding, restricting the VLM to the Textualizer stage only. We evaluate TextFlow on the FlowVQA ( Singh et al. 2024 ) and FlowLearn ( Pan et al. 2024 ) datasets, combining various open-source and closed-source VLMs/LLMs for the Vision Textualizer and Textual Reasoner stages, respectively. Our experiments demonstrate that using Claude-3.5 ( Anthropic 2024 ) as the Textualizer , alongside either Claude-3.5 or GPT-4o ( OpenAI 2024 ) as the Reasoner , significantly outperforms end-to-end flowchart understanding using Claude-3.5 or GPT-4o alone. Further analysis reveals that (i) Graphviz is generally the most effective textual representation for flowcharts; (ii) TextFlow exhibits robustness across different task categories, flowchart sources, orientations, and sizes; and (iii) the Vision Textualizer stage is more often the source of errors than the Reasoner stage, particularly when using Claude-3.5 for both. Our contributions are threefold: • Our TextFlow presents a novel and effective approach to leverage VLMs/LLMs in flowchart understanding: instead of treating the task end-2-end, breaking it into subtasks appears to be easier. It not only achieves new state-of-the-art (82.74 vs. 76.61) but also potentially provides inspiration for other visual tasks. • We take the lead in introducing three textual formats— Graphviz , Mermaid , and PlantUML —for transforming flowcharts into structured text representations, facilitating the reasoning process. • Our detailed analysis provides key insights into solving the flowchart understanding problem, including identifying Graphviz as the most effective representation, conducting a fine-grained analysis across different flowchart sources and task categories, and quantitatively comparing the roles of Vision Textualizer and Textual Reasoner when using the same VLM. 2 Related Work End-2-End VQA for Flowchart/Diagrams For flowchart understanding, most prior work has focused on benchmark construction with different emphases. For example, FlowchartQA ( Tannert et al. 2023 ) emphasizes reasoning over geometric and topological features, FlowLearn ( Pan et al. 2024 ) focuses on synthetic and scientific flowcharts, IconQA ( Lu et al. 2021 ) targets abstract diagrams that are rich in semantics rather than natural images, and FlowVQA ( Singh et al. 2024 ) focuses on evaluating spatial reasoning, decision-making, and logical progression tasks. Most of these works adopt end-to-end systems. For instance, Lu et al. 2021 proposed Patch-TRM, which employs a pyramid cross-modal Transformer with diagram embeddings pre-trained on the Icon dataset, while Singh et al. 2024 and Pan et al. 2024 explored various VLMs for their respective tasks. Decomposing VQA into Multiple Steps Previous work in this area falls into two main categories: (i) Image preprocessing , where the decomposition involves processing flowchart images. For example, Pan et al. 2024 first identified visual components and applies OCR to the images. (ii) Question decomposition , where the focus is on breaking down questions into sub-reasoning steps. For instance, Cao & Jiang 2023 decomposed questions into sub-tasks and assigned them to suitable pre-trained models without adaptation. Khan et al. 2024 compared human-written and model-generated question decompositions, while Barezi & Kordjamshidi 2024 decomposed multi-hop questions by determining the modality required, using a captioner for visual sub-questions and LLMs for textual ones. IdealGPT ( You et al. 2023 ) employed LLMs to generate sub-questions, VLMs to provide sub-answers, and another LLM to reason and produce the final answer. Our approach differs from these by focusing not on preprocessing flowchart images or decomposing questions, but on decomposing the process into two distinct stages. The first stage generates a novel textual representation, and the second transforms the original visual QA task into a textual QA task. 3 Methodology Figure 1: Our dual-stage TextFlow vs. prior work. As illustrated in Figure 1 , the approach TextFlow consists of two main stages: (1) Vision Textualizer : A VLM converts the visual components of the flowchart into a high-quality text representation. (2) Textual Reasoner : An LLM or VLM based Textual Reasoner can use tools and interpret the text representations to answer questions related to the flowchart’s logic and structure. By decoupling these tasks, TextFlow enables specialized models to focus on their respective strengths: Vision Textualizer process the visual elements, while Textual Reasoner handle reasoning tasks, thus improving accuracy and flexibility. Next, we elaborate on Vision Textualizer (Section 3.1 ) and Textual Reasoner

Read the complete text