The Path Not Taken: Duality in Reasoning about Program Execution
Eshgin Hasanov, Md Mahadi Hassan Sibat, Santu Karmaker, Aashish Yadavally · 2026
Read The Path Not Taken: Duality in Reasoning about Program Execution by Eshgin Hasanov, Md Mahadi Hassan Sibat, Santu Karmaker, Aashish Yadavally (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20917.txt.
Introduction
Hasanov and colleagues argue that judging code understanding by input-output prediction alone gives a narrow, contamination-prone picture. True grasp of execution, they say, has two sides: predicting what a program does for a given input, and inferring how the input must change to reach a chosen behavior. Their DexBench benchmark pairs 445 instances across both tasks, and tests of 13 language models show dual-path reasoning discriminates dynamic understanding robustly.
Catalog record
| CREATOR | Eshgin Hasanov, Md Mahadi Hassan Sibat, Santu Karmaker, Aashish Yadavally |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53497 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.20917
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Path Not Taken: Duality in Reasoning about Program Execution Eshgin Hasanov, Md Mahadi Hassan Sibat, Santu Karmaker, Aashish Yadavally Abstract Large language models (LLMs) have shown remarkable capabilities across diverse coding tasks. However, their adoption requires a true understanding of program execution rather than relying on surface-level patterns. Existing benchmarks primarily focus on predicting program properties tied to specific inputs ( e.g. , code coverage, program outputs). As a result, they provide a narrow view of dynamic code reasoning and are prone to data contamination. We argue that understanding program execution requires evaluating its inherent duality through two complementary reasoning tasks: ( i ) predicting a program’s observed behavior for a given input, and ( ii ) inferring how the input must be mutated toward a specific behavioral objective. Both tasks jointly probe a model’s causal understanding of execution flow. We instantiate this duality in DexBench , a benchmark comprising 445 paired instances, and evaluate 13 LLMs. Our results demonstrate that dual-path reasoning provides a robust and discriminative proxy for dynamic code understanding. 1 Introduction Large language models (LLMs) are increasingly being adopted in the field of software engineering (SE), from writing code and unit tests to finding and fixing bugs. However, their performance remains inconsistent: these models can solve complex coding problems 14 , yet still fail to reason about basic loops 8 . This can possibly be due to mimicking surface-level patterns without a true understanding of the program, which inherently limits their applicability in real-world settings. As a result, recent efforts have focused on a fine-grained evaluation of the reasoning capabilities of coding LLMs 18 ; 5 ; 17 , moving beyond end-to-end evaluations on downstream tasks such as program synthesis 6 ; 11 or repair 14 . In practice, such reasoning evaluations primarily rely on executability , a distinct feature of source code requiring accurate modeling of execution flow. These assess how well LLMs understand program behaviors during execution, including predicting code coverage 8 ; 5 , input-output mappings 9 ; 5 , as well as tracking intermediate updates to program state and subsequent execution paths 19 ; 5 . Nevertheless, current program execution-based benchmarks consider runtime behaviors under only a single test case, i.e. , along one execution path ( e.g. , predicting the statements \mathcal{S} in a program P that are executed for an input I ). However, a given program may traverse many different paths depending on its inputs. Therefore, single-test evaluations offer a narrow view of the models’ program understanding. Moreover, since benchmarks defined by fixed sets of input-output pairs can be memorized during training 24 , they are susceptible to data contamination, a growing concern in modern LLM evaluation. To address this gap, we introduce the concept of duality in reasoning about program execution . The key idea is based on the argument that two program paths share a common execution space and diverge only at branching points where the current program state determines diverging execution flows. We refer to the actual path taken for a given input as execution path and the alternative as counterfactual path . We operationalize our framework by designing a pair of complementary reasoning tasks: one predicting a program property observed along the execution path ( forward reasoning ); other determining the input required for counterfactual path to be taken ( backward reasoning ). Note that a program naturally exhibits n -ality (or “multiality”), characterized by many possible pairs of execution and counterfactual paths across its input domain. This property can be viewed as a composition of discrete dual pairs. By selecting one such pair for evaluation, we can isolate and evaluate the causal logic a model employs to reason about program execution. Accordingly, we posit that probing a model’s ability to understand both how the observed execution occurs and under what conditions ( e.g. , intermediate program state) the execution would instead flow through the counterfactual path provides a more robust evaluation of reasoning capability in coding LLMs. As an illustration, consider the target program in Figure 1 . The program accepts multiple paths, including \pi^{(1)}\!: 3\!\to\!...\!\to\!6\!\to\!8\!\to 9\to\!...\!\to\!6\!\to 8\to\!9\!\to 5\!\to\!12\!\to\!\mathbf{[END]} , \pi^{(2)}\!: 3\!\to\!...\!\to\!5\!\to\!6\!\to\!7\!\to\!\mathbf{[END]} , \pi^{(3)}\!: 3\!\to\!...\!\to\!6\!\to\!8\!\to\!...\!\to\!11\!\to\!5\!\to\!...\!\to\!\mathbf{[END]} , among others. Any pair \langle\pi^{(i)},\pi^{(j)}\rangle reflects this multiality. For the test input “ ua6hajq ”, execution follows the control-flow path \pi^{(1)} . Here, we designate \pi^{(1)} as the execution path \pi_{exec} and select \pi^{(2)} as the counterfactual path \pi_{cf} to illustrate the proposed duality (highlighted in Figure 1 in green and red, respectively) 1 1 1 In Section 3.2 , we describe the algorithm used to select the counterfactual path, and in Section 4.3 , we benchmark LLMs with different counterfactual path selection strategies. . In particular, both \pi_{exec} and \pi_{cf} share execution space 3\!\to\!...\!\to\!6 , diverging only at line 6 depending on the branch condition. When it evaluates to “ No ”, the execution proceeds along \pi_{exec} ; otherwise, \pi_{cf} is taken. By design, forward reasoning evaluates an LLM’s ability to predict program properties along \pi_{exec} for a given input I_{exec} , while backward reasoning evaluates whether it can infer the new input I_{cf} that I_{exec} should be mutated to such that execution now flows along \pi_{cf} instead. Notably, the novelty in the notion of duality lies in its focus on the causal relationships in program execution that induce divergence from \pi_{exec} to \pi_{cf} , rather than in reasoning about program properties along distinct paths in isolation 17 . Accordingly, we propose DexBench , a dual-path, program execution-based benchmark that jointly investigates both forward execution reasoning and backward counterfactual reasoning. DexBench is built from real-world programs that cover a range of complexity, extracted from widely used HumanEval 6 , CruxEval 11 , and PythonSaga 30 . We evaluate 9 open-source and 4 closed-source/proprietary LLMs, spanning both reasoning and general-purpose models across small- to large-scale model sizes. Our experiments reveal that: ( i ) strong performance on execution or counterfactual reasoning in isolation does not translate to success under joint evaluation, highlighting the limitations of single-path evaluation benchmarks. ( ii ) dual-path reasoning provides a reliable proxy for causal, state-aware reasoning of execution flow. ( iii ) parameter scaling and reasoning-focused post-training do not guarantee improvements in dual-path reasoning, as seen with some mid-sized and general-purpose models that outperformed larger or explicitly reasoning-focused counterparts. Together, our findings indicate that DexBench offers a systematic and extensible framework to evaluate dynamic code understanding in LLMs. All code and data are publicly available. 2 2 2 https://github.com/sail-ucf/dexbench Figure 1: DexBench exploits the shared execution space between execution and counterfactual paths to enable a robust evaluation of LLMs in program execution. 2 Duality of Reasoning We formalize the concept of duality in reasoning about program execution as follows. 2.1 (Forward) Execution Reasoning Consider a program P with test input I_{exec}\in\mathcal{I} . Executing P on I_{exec} induces an execution trace \tau_{exec}\!=\!\big(\langle s^{(1)},\sigma^{(1)}\rangle...\langle s^{(|\tau_{exec}|)},\sigma^{(|\tau_{exec}|)}\rangle\big)\in\mathcal{T}, where each s^{(t)} is a statement in P executed at step t , and \sigma^{(t)} is the corresponding program state, defined as a mapping from program variables to their values. Let \phi:\!\mathcal{T}\!\to\!\mathcal{O} denotes the observable program properties or runtime behaviors derived from execution traces ( e.g. , program output, code coverage, or a post-condition over variables). For program P