Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents
Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Vivek Gupta, Dinesh Manocha · 2025
Read Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents by Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Vivek Gupta, Dinesh Manocha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01344.txt.
Introduction
Language models asked about flowcharts often invent connections that are not there, which is dangerous in fields like health and logistics. Manan Suri and co-authors define a new task: fine-grained flowchart attribution, tracing exactly which parts of a chart support each claim in a model's answer. Verifiable grounding, they argue, is what makes flowchart answers trustworthy.
Their system, FlowPathAgent, cuts the chart into pieces, rebuilds it as a symbolic graph, and lets an agent walk the graph to produce attribution paths. They also release FlowExplainBench, a benchmark spanning many chart styles and question types. The agent cut visual hallucinations and beat strong baselines by 10 to 14 percent. The authors present attribution as the fix for shaky diagram reasoning.
Catalog record
| CREATOR | Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Vivek Gupta, Dinesh Manocha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Vivek Gupta, Dinesh Manocha Abstract Flowcharts are a critical tool for visualizing decision-making processes. However, their non-linear structure and complex visual-textual relationships make it challenging to interpret them using LLMs, as vision-language models frequently hallucinate nonexistent connections and decision paths when analyzing these diagrams. This leads to compromised reliability for automated flowchart processing in critical domains such as logistics, health, and engineering. We introduce the task of Fine-grained Flowchart Attribution , which traces specific components grounding a flowchart referring LLM response. Flowchart Attribution ensures the verifiability of LLM predictions and improves explainability by linking generated responses to the flowchart’s structure. We propose FlowPathAgent , a neurosymbolic agent that performs fine-grained post hoc attribution through graph-based reasoning. It first segments the flowchart, then converts it into a structured symbolic graph, and then employs an agentic approach to dynamically interact with the graph, to generate attribution paths. Additionally, we present FlowExplainBench , a novel benchmark for evaluating flowchart attributions across diverse styles, domains, and question types. Experimental results show that FlowPathAgent mitigates visual hallucinations in LLM answers over flowchart QA, outperforming strong baselines by 10-14% on our proposed FlowExplainBench dataset. 1 Introduction Figure 1: Attribution (represented by \bullet\!\!-\!\!\bullet\!\!-\!\!\bullet ) with FlowPathAgent ensures logical consistency in flowchart-based reasoning. FlowPathAgent uses a neurosymbolic approach to generate attribution paths ( ➊ & ➋ ) in the flowchart. This enhances interpretability and reliability in flowchart driven automated decision-making. Flowcharts are a fundamental tool for representing structured decision-making processes. Used across domains such as software engineering, business process modeling, and instructional design, flowcharts provide a visual roadmap of logical operations, guiding both human users and automated systems Charntaweekhun and Wangsiripitak 2006 ; Perols and Perols 2024 ; Zimmermann et al. 2024 ; Ensmenger 2016 . Their structured yet visual nature makes them an effective medium for conveying procedural logic. However, interpreting flowcharts accurately is challenging due to their nonlinear structures (branching and loop-based control flow), where meaning emerges from the interplay between textual content, visual arrangement, and logical dependencies. Ambiguities in flowchart interpretation arise from diverse notational conventions, implicit relationships, and misinferred steps, making precise attribution of information sources difficult Eppler et al. 2008 . Recent advancements in Vision Language Models (VLMs) have enabled substantial progress in flowchart processing Singh et al. 2024 . These models leverage both textual and visual information, allowing them to extract structural relationships, recognize decision nodes, and generate answers based on flowchart content. However, despite their capabilities, VLMs struggle with hallucination: the tendency to generate information that is not grounded in the input Huang et al. 2024 ; Guan et al. 2024 . In the context of flowcharts, hallucination can manifest as misidentifying decision nodes, producing incorrect logical pathways, or fabricating connections that do not exist in the original structure. This issue severely impacts the reliability of automated flowchart reasoning, particularly in high-stakes applications such as healthcare, software verification and process automation. Although VLMs have made significant progress in understanding flowcharts, prior work has mainly concentrated on flowchart parsing Arbaz et al. 2024 , conversion Shukla et al. 2023 ; Liu et al. 2022 , and question-answering Singh et al. 2024 ; Tannert et al. 2023 , while overlooking the critical aspect of fine-grained attribution. While existing attribution methods Huo et al. 2023 ; Chen et al. 2023 focus on textual grounding, attributing responses to visual-textual elements like flowcharts presents unique challenges. It involves not just text recognition, but also interpreting the interconnected decision nodes, hierarchical structures, and conditional pathways that define flowchart semantics. Attribution serves as a crucial mechanism for mitigating hallucination by explicitly tracing the paths in the flowchart that ground a particular response, enabling rigorous evaluation of the model’s fidelity to the flowchart’s logic, as illustrated in Fig 1 . Such fine-grained attribution is fundamental for ensuring reliability, particularly when these systems are deployed in domains where verifiable decision-making is crucial. Main Results. We introduce Flowchart Attribution task aimed at identifying the optimal path within a flowchart that grounds the model’s response. The optimal path aims to extract the most relevant sequence of nodes and edges that directly support the model’s reasoning, encompassing all the key decision points and actions involved in the prediction. To facilitate the evaluation of this task, we propose FlowExplainBench , a novel benchmark that features a diverse set of flowcharts with varying styles, domains, and question types. We introduce FlowPathAgent , a neurosymbolic agent specifically designed to perform fine-grained as a post-hoc flowchart attribution. Instead of relying solely on text-based or vision-based cues, FlowPathAgent integrates symbolic reasoning by using an agentic interface to interact with the flowchart as a graph object. FlowPathAgent begins with segmenting flowcharts into distinct components, followed by constructing symbolically operable flowchart representations. These graph-based representations have direct correspondence to visual regions of the flowchart, enabling the model to interoperate between the visual and symbolic representations. We leverage graph tools to extract and manipulate these representations, allowing for identification of relevant nodes and edges. Our methodology facilitates precise attribution of the model’s reasoning steps to specific decision points within the flowchart, providing accurate and interpretable explanations of the model’s output. Experimental results demonstrate that FlowPathAgent significantly outperforms strong baselines Lai et al. 2024 ; Peng et al. 2023 ; Yuan et al. 2025 by 10-14% on FlowExplainBench . Our main contributions 1 1 1 Code and data will be released on acceptance. are: • We introduce a new task of Fine-grained Flowchart Attribution where the goal is to identify the optimal path within a flowchart diagram that grounds the LLM text response. • FlowExplainBench - a novel evaluation benchmark consisting of 1k+ high quality attribution annotations over flowchart QA with diverse styles, domains, and question types. • FlowPathAgent - a neurosymbolic agent capable of performing fine-grained post-hoc attribution for flowchart QA. FlowPathAgent uses a VLM-based agentic approach to perform graph-based reasoning and symbolic manipulation to accurately trace the decision process within flowcharts. 2 Related Work 2.1 Flowchart Understanding Research in flowchart understanding has evolved from basic image processing to complex reasoning tasks. Modern deep learning approaches, such as FR-DETR Sun et al. 2022a , have significantly improved symbol and edge detection through end-to-end architectures that combine CNN backbones with multi-scale transformers. The emergence of LLMs has led to benchmarks like FlowchartQA Tannert et al. 2023 , FlowLearn Pan et al. 2024 , SCI-CQA Shen et al. 2024 , and FlowVQA Singh et al. 2024 , which assess geometric understanding, spatial reasoning, and logical capabilities of models for question-answering on flowchart images . Recent work like Ye et al. 2024 has begun exploring alternatives to end-to-end VLMs; Ye et al. 2024 introduced intermediate textual representations between visual processing and reasoning steps for Flowchart QA; Liu et al. 2022 ; Shukla et al. 2023 explored code generation from flowcharts. 2.2 Attribution in LLMs Large Language Models (LLMs) are challenged with factual accuracy Zhang et al. 2023 . While various solutions have emerged, including citation-aware training Gao et al. 2023 and tool augmentation Ye et al. 2023 , ensuring reliable attributions remains crucial. Three primary attribution strategies have emerged in literature: (1) Direct model-driven