Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data

Chu-Cheng Lin, Daiyi Peng, Yifeng Lu, Ming Zhang, Eugene Ie · 2025

Read Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data by Chu-Cheng Lin, Daiyi Peng, Yifeng Lu, Ming Zhang, Eugene Ie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18244.txt.

Introduction

Chu-Cheng Lin and co-authors make multi-step language model workflows trainable without breaking their rules, in a 2025 paper. Pipelines that chain model calls are usually tuned by fiddling with prompts, a brittle craft. Structured tasks also demand outputs in exact formats, which prompt tweaks cannot guarantee. Their framework recasts the whole workflow as a typed probabilistic program.

Models with small adapters and fixed logic together form one joint distribution to train by gradient. Theory shows the training bias fades as the system learns to obey its types. On finance questions and symbolic math, scores jumped dramatically over prompt-tuning baselines. One math benchmark rose from near zero to a respectable score. Reliable, rule-abiding pipelines are the prize.

Catalog record

CREATORChu-Cheng Lin, Daiyi Peng, Yifeng Lu, Ming Zhang, Eugene Ie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18244

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Type-Compliant Adaptation Cascades: Adapting Programmatic LM Workflows to Data Chu-Cheng Lin, Daiyi Peng, Yifeng Lu, Ming Zhang, Eugene Ie Abstract Reliably composing Large Language Models (LLMs) for complex, multi-step workflows remains a significant challenge. The dominant paradigm — optimizing discrete prompts in a pipeline — is notoriously brittle and struggles to enforce the formal compliance required for structured tasks. We introduce Type-Compliant Adaptation Cascades ( tac s), a framework that recasts workflow adaptation as learning typed probabilistic programs. tac s treats the entire workflow, which is composed of parameter-efficiently adapted LLMs and deterministic logic, as an unnormalized joint distribution. This enables principled, gradient-based training even with latent intermediate structures. We provide theoretical justification for our tractable optimization objective, proving that the optimization bias vanishes as the model learns type compliance. Empirically, tac s significantly outperforms state-of-the-art prompt-optimization baselines. Gains are particularly pronounced on structured tasks, improving MGSM-SymPy from 57.1 % 57.1\% to 75.9 % 75.9\% for a 27B model, MGSM from 1.6 % 1.6\% to 27.3 % 27.3\% for a 7B model. tac s offers a robust and theoretically grounded paradigm for developing reliable, task-compliant LLM systems. \uselogo 1 Introduction Figure 1: A tac workflow that adapts LLMs to generate structured arithmetic expressions, in the form of a tac that consists of two adaptors and two deterministic Python functions. Annotations that are unavailable at training time are depicted as translucent and have dashed boxes. A main message of this work is that instead of defining the workflow imperatively as a fixed-parameter system, we treat an entire typed workflow as a single probabilistic program, whose parameters are lightweight PEFT modules, allowing end-to-end training with latent variables. Large Language Models (LLMs) are increasingly used as components in multi-step workflows that combine free-form reasoning, structured intermediate representations, and deterministic tooling. Popular frameworks ( e.g. , DSPy [ 1 ] , LangChain [ 2 ] ) expose convenient modular abstractions for composing such systems, but adaptation to task data still relies heavily on discrete prompt search. In practice, this creates two bottlenecks: (1) brittleness : discrete prompt optimization is unstable and sensitive to base-model capability, and (2) lack of type compliance : prompted LMs struggle to honor type and schema contracts required by structured inputs/outputs and tool interfaces, especially with smaller base models. We propose Type-Compliant Adaptation Cascades ( tac s), a probabilistic programming framework that treats an entire typed workflow as a single trainable model. An example of such workflow is shown in Figure 1 . A tac is a directed acyclic hypergraph whose nodes are typed [ 3 ] data containers and whose hyperedges are either (a) type-compliant LM adaptors parameterized by lightweight parameter-efficient fine-tuning (PEFT) modules [ 4 ] ( e.g. , LoRA [ 5 ] ) or (b) deterministic functions. By endowing the workflow with a probability model over its intermediate (possibly latent) typed variables, we can perform gradient-based end-to-end adaptation while enforcing type contracts at every step via parse and canon functions. We thus formalize typed LM workflow adaptation as a problem of learning unnormalized probabilistic programs, where each hyperedge assigns zero mass to type-invalid outputs; type compliance emerges as a partition function \mathcal{Z}_{{\bm{\theta}}}\leq 1 . To our knowledge, this perspective — and the associated MC-EM training with vanishing bias guarantees — has not been explored for LM adaptation. Why this paradigm. Unlike prior approaches that treated workflow adaptation as a prompt optimization problem, tac is a declarative, differentiable framework that avoids searching in a non-differentiable, combinatorial space, and its effectiveness is not bounded by to the base model’s intrinsic capacity. Moreover, tac s optimize continuous adapter parameters under explicit type constraints, enabling (1) robust compliance with structured interfaces and (2) principled learning with latent intermediate structures ( e.g. , rationales, expression trees, refinement steps). Contributions. • Framework . We formalize typed LM workflows as probabilistic programs: each learned hyperedge is an unnormalized conditional distribution that assigns zero mass to outputs violating type contracts. This yields a globally unnormalized model whose partition function \mathcal{Z}_{{\bm{\theta}}}\leq 1 measures type-compliant mass . • Theory . We introduce tac STaR, a Monte Carlo EM procedure that drops the intractable \nabla_{{\bm{\theta}}}\log\mathcal{Z}_{{\bm{\theta}}} term. We prove ( Theorem 1 ) that, under a standard well-specified assumption, maximizing the unnormalized objective recovers the MLE solution; and ( Theorem 2 ) that the gradient bias is bounded by a constant times (1-\mathcal{Z}_{{\bm{\theta}}}) . Thus, as training drives type compliance up (and thus \mathcal{Z}_{{\bm{\theta}}}\rightarrow 1 ), the approximation gap vanishes. • Practice . Across QA and structured generation tasks that require heavy reasoning (MGSM, MGSM-SymPy, FinQA, HotPotQA) and model sizes (7B, 27B), tac s consistently outperform strong DSPy prompt-optimization baselines. Gains are largest when (1) base models are smaller and (2) tasks require strict structure. For example, on MGSM-SymPy with a 27B model, tac s achieve \mathbf{75.9} vs. \mathbf{57.1} ; on FinQA (27B), \mathbf{34.0} vs. \mathbf{12.7} . With a 7B model, MGSM improves from \mathbf{1.6} to \mathbf{27.3} , and FinQA from \mathbf{0.7} to \mathbf{9.7} . Summary of results. (1) Gradient-based adaptation within typed workflows is markedly more effective than discrete prompt search for structured tasks. (2) Incorporating latent reasoning/refinement nodes and adapting them end-to-end improves over direct input-to-output adaptation. (3) Empirically, estimated type compliance mass \mathcal{Z}_{{\bm{\theta}}} rises rapidly during training and correlates with accuracy, supporting our theoretical justification for the unnormalized objective. Outlook. The typed, probabilistic view makes workflow design modular and data-driven. It also suggests a path toward automated generation of typed workflow graphs: first synthesize a candidate tac from a task specification, then adapt it to data with tac STaR. We view tac s as a foundation for reliable, type-compliant LM systems that move beyond prompt-only programming. 2 Background and Related Work The challenge of adapting LLMs to complex problems involving structured workflows and type constraints intersects with several lines of research, which we survey below. Programmatic LM workflows. A large body of work exposes LMs through programmed pipelines as typed or templated modules, with declarative constraints and optimizers, such as DSPy [ 1 , 6 ] , LMQL [ 7 ] , and LangChain [ 2 ] . These systems typically specify structure and then tune prompts or few-shot exemplars. They do not cast the entire workfrlow as a single probabilistic object with learnable continuous parameters, and a likelihood objective. While there have been proposals that optimized weights under such programmatic pipelines (such as BetterTogether [ 8 ] ), tac s differs fundamentally in its principled yet optimization-friendly probabilistic formulation, which enables both theoretically justified training methods ( Section 4.1 ) and advanced inference techniques ( Section 4.2 ). Probabilistic programming and structured prediction. Probabilistic programming languages tailored for machine learning, such as Edward [ 9 ] and Pyro [ 10 ] , combine differentiable components with stochastic control flow. On the other hand, classical structured prediction [ 11 , 12 ] provides tools for handling global constraints in unnormalized models. Our formulation connects these threads to LM workflows: each typed hyperedge is an unnormalized conditional whose type compliance functions as a partition function term \mathcal{Z}_{{\bm{\theta}}}\leq 1 . This distinct perspective allows us to train with a tractable objective, whos bias vanishes as type compliance rises. Problem-solving strategies and adapting for reasoning. Techniques like Chain-of-Thought (CoT)

Read the complete text