FABLE : A Novel Data- F low A nalysis B enchmark on Procedural Text for L arge Language Model E valuation
Vishal Pallagani, Nitin Gupta, John Aydin, Biplav Srivastava · 2025
Read FABLE : A Novel Data- F low A nalysis B enchmark on Procedural Text for L arge Language Model E valuation by Vishal Pallagani, Nitin Gupta, John Aydin, Biplav Srivastava (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24258.txt.
Introduction
Vishal Pallagani, Nitin Gupta, John Aydin, and Biplav Srivastava test if word tools can track a flow in a 2025 paper. Flow is how a mark is born, bent, and kept in a step tale, the base of all plan craft.
Their test turns eight stock flow crafts to asks in plain text. The crafts run from which birth rules a use, to which marks may bear a germ. Three worlds bear them: fare lists, road plans, and made work plans. Near two and a half thousand asks fill the bank. A thought-deep tool led, but at a score of time past its kin. The plain and code tools scored near a coin toss. The authors give the bank as a first gauge of this core skill.
Catalog record
| CREATOR | Vishal Pallagani, Nitin Gupta, John Aydin, Biplav Srivastava |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24258
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FABLE : A Novel Data- F low A nalysis B enchmark on Procedural Text for L arge Language Model E valuation Vishal Pallagani, Nitin Gupta, John Aydin, Biplav Srivastava Abstract Understanding how data moves, transforms, and persists, known as data flow , is fundamental to reasoning in procedural tasks. Despite their fluency in natural and programming languages, large language models (LLMs), although increasingly being applied to decisions with procedural tasks, have not been systematically evaluated for their ability to perform data-flow reasoning. We introduce FABLE , an extensible benchmark designed to assess LLMs’ understanding of data flow using structured, procedural text. FABLE adapts eight classical data-flow analyses from software engineering: reaching definitions, very busy expressions, available expressions, live variable analysis, interval analysis, type-state analysis, taint analysis, and concurrency analysis. These analyses are instantiated across three real-world domains: cooking recipes, travel routes, and automated plans. The benchmark includes 2,400 question–answer pairs, with 100 examples for each domain-analysis combination. We evaluate three types of LLMs: a reasoning-focused model ( deepseek-r1:8b ), a general-purpose model ( llama3.1:8b ), and a code-specific model ( granite-code:8b ). Each model is tested using majority voting over five sampled completions per prompt. Results show that the reasoning model achieves higher accuracy, but at the cost of over 20x average slowdown in inference time compared to the other models. On the contrary, the general-purpose and code-specific models perform close to random chance. FABLE provides the first diagnostic benchmark to systematically evaluate data-flow reasoning and offers insights for developing models with stronger procedural understanding. 1 Introduction Procedural texts, such as recipes, travel routes, and plans, describe dynamic processes that unfold over time. Understanding these texts requires models to reason not only about individual statements but also about how information evolves throughout a sequence of actions. A core capability underlying this understanding is data-flow reasoning, which involves tracking how entities, variables, and their attributes are defined, modified, and propagated as a procedure progresses. In software engineering, data-flow analysis [ 3 , 4 , 10 , 17 ] plays a foundational role in verifying the correctness of programs by examining how variables change across control structures. This capability is equally essential for comprehending procedural language, where similar patterns of definition and modification occur in naturalistic contexts. For example, a model reading a recipe must track ingredient transformations and temporal dependencies to maintain a coherent understanding of the process. Despite this alignment between code and procedural text, current evaluation benchmarks do not fully test such reasoning capabilities in LLMs. In literature, researchers have introduced datasets that evaluate isolated aspects of procedural reasoning. Some focus on tracking entity states [ 21 ] , while others target causal or temporal relationships between steps [ 22 , 14 ] . These benchmarks have highlighted the importance of procedural understanding, but they also expose significant gaps. For instance, on the WIQA dataset [ 22 ] , transformer-based models reached only about 74% accuracy compared to 96% for humans. Similarly, on the temporal ordering benchmark TORQUE [ 14 ] , models lag by approximately 30% behind human performance. Moreover, existing benchmarks often isolate a single reasoning dimension, such as causality or time, and tend to use short, domain-specific descriptions. In contrast, real-world procedures involve multiple interdependent factors, including causality, temporal structure, and entity state evolution over extended sequences. In this paper, we introduce FABLE , a benchmark specifically designed to evaluate data-flow reasoning in LLMs through procedural texts. FABLE is inspired by program analysis techniques and extends them to natural language. It emphasizes comprehensive tracking of “where and how information flows” through a procedure, bridging the gap between causal effects, entity state changes, and temporal dependencies within the same benchmark. FABLE adapts eight classical data-flow analyses from software engineering, including reaching definitions, available expressions, very busy expressions, live variable analysis, interval analysis, type-state analysis, taint analysis, and concurrency analysis. These analyses are instantiated across three real-world domains: cooking recipes, travel routes, and automated plans. Each analysis challenges models to identify how information is introduced, propagated, and transformed over time. The benchmark contains 2,400 question-and-answer pairs, with 100 examples for each analysis within each domain. We evaluate three categories of large language models: a reasoning-focused model, a general-purpose model, and a code-specific model. Each model is tested using majority voting over five sampled completions per prompt. Our results show that the reasoning model achieves the highest accuracy but has over 20x average slowdown in inference time compared to the other two models. In contrast to the reasoning model, the general-purpose and code-specific models perform close to random chance. These findings reinforce the challenges of data-flow reasoning and demonstrate the utility of FABLE as a diagnostic benchmark. Our key contributions are as follows: (1) We introduce FABLE , the first benchmark to adapt eight classical data-flow analyses from software engineering to procedural text, enabling comprehensive evaluation of data-flow reasoning in LLMs; (2) We benchmark three categories of large language models and conduct a two-fold empirical analysis: (i) we evaluate to show that the dataset presents a substantial reasoning challenge, and (ii) we establish baseline performance and computational tradeoffs across reasoning-focused, general-purpose, and code-specific LLMs. 2 Background & Related Works 2.1 Procedural Text and the Need for Data-Flow Understanding We refine the definition of procedural text that is defined in prior literature [ 27 ] . A procedural text is a triple \mathcal{P}\;=\;(\,G,\;S,\;\preceq\,), where: G is a goal , expressed in natural language describing the final desired outcome, S=\{\,s_{1},\dots,s_{n}\} is a finite set of steps , each step s_{i} being a natural-language instruction describing some action or sub-procedure to perform, and \preceq is a (partial or total) order on S , indicating the permissible sequence(s) in which these steps can be executed to achieve G . Procedural texts are ubiquitous in the real world. They appear in domains as diverse as cooking, scientific experimentation, travel planning, and robotics. Their role is central to knowledge sharing, decision-making, automation, and task execution in both human and machine systems. Understanding procedural texts requires tracking how entities and their attributes evolve across ordered steps, how one step’s output serves as another’s input, and how the process collectively satisfies the intended goal [ 20 ] . This naturally calls for data-flow reasoning, a concept rooted in program analysis, where variable definitions and their propagation through execution paths are systematically tracked. Analogously, in natural language, such reasoning involves identifying where entities are introduced, how they are transformed, and which steps depend on prior states. Despite its importance, existing benchmarks only target isolated aspects of procedural reasoning, as discussed below. 2.2 Prior Relevant Benchmarks Early work on procedural text understanding focused on tracking how entities change state throughout a process. A comparison of FABLE with relevant prior benchmarks is presented in Table 1 . ProPara [ 20 ] , a seminal dataset of 488 formal science paragraphs, introduced this challenge by asking models to identify changes in entity properties (like location or existence) at each step of a described process (e.g., photosynthesis). OpenPI [ 23 ] later extended this idea to 810 real-world how-to guides from WikiHow. It required models to generate structured state-change tuples for each step, capturing transformations. In contrast, FABLE spans a broader range