Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization

Naing Oo Lwin, Rajesh Kumar · 2026

Read Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization by Naing Oo Lwin, Rajesh Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09894.txt.

Introduction

Lwin and Kumar compare two ways of running language-model modernisation of old COBOL code into Python, a task where scarce expertise and strict correctness make mistakes expensive. In agentic setups the model itself steers each tool step, while deterministic setups follow a fixed execution plan. The study holds models, prompts, tools and source programs constant and varies only who controls execution, so orchestration is the sole variable.

Deterministic runs matched the agentic runs on functional accuracy while improving worst-case robustness and cutting run-to-run variability. They also used up to three and a half times fewer tokens, which lowers operating cost sharply. For structured workflows with explicit validation stages, the authors conclude, a fixed plan gives steadier and cheaper behaviour without giving up translation quality.

Catalog record

CREATORNaing Oo Lwin, Rajesh Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09894

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deterministic vs. LLM-Controlled Orchestration for COBOL-to-Python Modernization Naing Oo Lwin, Rajesh Kumar Abstract . Modernizing legacy COBOL systems remains difficult due to scarce expertise, large and long-lived codebases, and strict correctness requirements. Recent large language model (LLM)-based modernization systems increasingly rely on agentic workflows in which the model controls multi-step tool execution. However, it remains unclear whether delegating execution control to the LLM improves correctness, robustness, or efficiency in structured software engineering workflows. We present a controlled empirical study of deterministic and LLM-controlled orchestration for COBOL-to-Python modernization. Using a unified experimental framework, we hold the language models, prompts, tools, configurations, and source programs constant while varying only the execution control strategy. This isolates orchestration as the sole experimental variable. We evaluate both approaches using functional correctness, robustness across repeated stochastic runs, and computational efficiency. Across multiple models, deterministic orchestration achieves comparable computational accuracy to LLM-controlled orchestration while improving worst-case robustness and reducing performance variability across runs. Deterministic execution also reduces token consumption by up to 3.5 × 3.5\times , leading to substantially lower operational cost. These results suggest that, in structured modernization workflows with explicit validation stages, fixed execution policies provide more stable and cost-efficient behavior than fully agentic orchestration without reducing translation quality. 1. Introduction Legacy software systems written in COBOL remain important in domains such as finance and government, where they continue to support long-lived operational infrastructure. Maintaining and modernizing these systems has become increasingly difficult due to workforce attrition, limited learning resources, scarce documentation, and the complexity of decades-old codebases ( Ciborowska et al. 2021 ) . Modernization is therefore necessary not only to sustain these systems but also to integrate them with modern software ecosystems. Recent work has explored the use of large language models (LLMs) for code translation and modernization, including systems that combine tool use, iterative reasoning, and autonomous execution ( Schick et al. 2023 ; Yao et al. 2023 ) . Many of these systems adopt agentic workflows in which the model selects tools, determines execution order, and decides when to retry or terminate execution. While such approaches have shown promise in open-ended software engineering tasks, their effectiveness in structured modernization workflows remains unclear. Legacy modernization workflows are typically organized around explicit stages such as transformation, validation, and testing. Intermediate outputs can often be verified automatically using executable test suites and deterministic validation procedures. This raises a key question: does delegating execution control to the LLM improve structured modernization workflows compared to fixed execution policies? Existing work has primarily focused on generation quality or end-task performance. As a result, it remains difficult to isolate how orchestration strategy itself affects correctness, robustness, and computational cost. In this paper, we present a controlled empirical study comparing deterministic and LLM-controlled orchestration for COBOL-to-Python modernization. Using ATLAS (Autonomous Transpilation for Legacy Application Systems), we hold the language models, prompts, tools, configurations, and source programs constant while varying only the execution control strategy. This isolates orchestration as the sole experimental variable. In deterministic orchestration, tool ordering, validation stages, and retry behavior are fixed across runs. In contrast, LLM-controlled orchestration allows the model to dynamically determine execution flow based on intermediate outputs. This enables a direct comparison between fixed execution policies and model-driven control under identical experimental conditions. Our results show that both approaches achieve comparable functional correctness but differ substantially in reliability and efficiency. Deterministic orchestration improves worst-case robustness and reduces variation across runs. It also reduces token consumption by up to 3.5\times , leading to substantially lower operational cost. These findings suggest that, for structured modernization workflows with explicit validation stages, deterministic execution provides a more stable and cost-efficient alternative to fully agentic orchestration without reducing translation quality. Briefly, our contributions are as follows: • We present a controlled experimental framework for LLM-based code modernization that isolates execution control as the sole experimental variable, enabling direct comparison between deterministic and LLM-controlled orchestration 1 1 1 Code and artifacts are available at https://github.com/astrio-ai/forall . . • We show that deterministic orchestration achieves comparable functional correctness to LLM-controlled orchestration while improving worst-case robustness and reducing variation across repeated runs. • We provide a quantitative analysis of computational efficiency and show that deterministic orchestration reduces token consumption by up to 3.5\times , leading to substantially lower operational cost. The remainder of this paper is organized as follows. Section 2 reviews prior work on automated code translation and agentic LLM systems. Section 3 describes the orchestration strategies and the ATLAS framework. Section 4 presents the experimental design, dataset, and evaluation metrics. Section 5 reports the empirical results and analysis. Finally, Section 6 concludes and discusses implications for LLM-based software engineering. 2. Background 2.1. Automated Code Translation Recent work has explored the use of large language models (LLMs) for automated code translation and modernization, framing translation as a mapping from a source program to a semantically equivalent target representation ( Yang et al. 2024 ) . Transformer-based models and large-scale pretraining have enabled direct, single-shot translation between programming languages, including legacy languages, often without explicit parallel corpora ( Lachaux et al. 2020 ; Jimenez et al. 2024 ) . Recent studies also examine multilingual code generation and large-scale code translation. Several works analyze factors that influence translation outcomes, including prompt language and prompt design ( Aljagthami et al. 2025 ; Orlanski et al. 2023 ; Pan et al. 2023 ; Tao et al. 2024 ) . While these approaches demonstrate promising translation quality, they primarily operate as one-pass generators and do not explicitly model execution control, iterative validation, or workflow structure. As a result, they provide limited insight into how different orchestration strategies affect correctness, robustness under repeated execution, or computational efficiency. 2.2. Agentic and Tool-Orchestrated LLM Systems More recent systems introduce agentic behavior, enabling LLMs to invoke tools, perform multi-step reasoning, and iteratively refine outputs ( Guo et al. 2024 ) . In many agentic frameworks, the LLM acts as a central orchestrator that dynamically selects tools, orders operations, and determines retry or termination behavior at runtime ( Qiu et al. 2025 ; Yao et al. 2023 ) . Several works also propose orchestration and workflow engines for standardizing and evaluating agentic execution behavior ( Mazzolenis and Zhang 2025 ; Zhang et al. 2025 ) . These designs are well-suited to open-ended or exploratory tasks, but they implicitly treat execution control as a reasoning task delegated to the language model. Prior work has shown that coupling planning and execution can introduce significant variability in execution behavior, where small variations in model outputs produce divergent execution paths ( Valmeekam et al. 2023 ) . This complicates reproducibility, stability, and cost predictability. In contrast, structured software engineering workflows such as legacy code modernization are typically organized around fixed execution stages, explicit tool dependencies, and verifiable intermediate states ( Tufano et al. 2024 ) . However, existing agentic systems do not empirically isolate the impact of execution control strategy itself. As a result, it remains unclear whether LLM-controlled orchestration provides measurable benefits over deterministic control when other factors are held constant. Our work addresses this gap through a controlled comparison of deterministic and LLM-controlled orchestration. We isolate execution control as the sole experimental variable. 3. Methodology Figure 1. Overview of the ATLAS system architecture

Read the complete text