A Contractive Feedback Semantics for Reinforcement Learning
Zuyuan Zhang · 2026
Read A Contractive Feedback Semantics for Reinforcement Learning by Zuyuan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24759.txt.
Introduction
Zuyuan Zhang builds a contractive feedback semantics for discounted reinforcement learning. Standard accounts use Bellman equations for closed Markov processes. Here a one-step decision process acts as an open stochastic part, and long-horizon evaluation comes from a contractive feedback loop. Series and parallel wiring become composition and tensoring of Bellman maps, while feedback becomes a guarded trace with a unique fixed point. Local error stays controlled in nearby circuits, and abstractions preserve value up to distortion bounds. Specs lift through wiring and feedback as quantale-valued contracts.
Catalog record
| CREATOR | Zuyuan Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 132654 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24759
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Contractive Feedback Semantics for Reinforcement Learning Zuyuan Zhang Abstract Discounted reinforcement learning is usually presented through Bellman equations on closed Markov decision processes. This paper develops a compositional view: a one-step decision process is treated as an open stochastic component, and infinite-horizon policy evaluation is obtained by closing a contractive feedback loop. The resulting semantics assigns typed Bellman transformers to open components, interprets series and parallel wiring as composition and tensoring of transformers, and interprets feedback as an admissible guarded Banach trace realized by a unique fixed point. This perspective yields three theoretical consequences. First, approximate component equivalence is a contextual congruence for admitted well-typed guarded one-hole contexts: local operator error remains controlled after plugging the component into a surrounding circuit that uses the hole once and whose feedback nodes have certified uniform guardedness. Second, exact and approximate state abstractions become commuting or near-commuting coalgebraic diagrams, giving value-preservation and explicit sup-norm distortion bounds. Third, under monotone ω \omega -continuous contract-transformer semantics, safety, risk, and resource specifications can be represented as quantale-valued contracts, where local inductive bounds lift through wiring and feedback by least-fixed-point reasoning. Its central claim is not that all RL morphisms form a global traced monoidal category, but that discounted Bellman evaluation admits a contractive feedback semantics on the admissible class of guarded circuits. 1 Introduction Reinforcement learning (RL) is commonly formalized as dynamic programming on a closed Markov decision process: fix a policy, write down a Bellman operator, and study the fixed point of that operator ( 2 ; 23 ; 3 ; 29 ) . This view is effective, but it hides a structural fact. The one-step transition is not intrinsically closed. It is an open component with an input interface, an output interface, a reward channel, and a continuation. The infinite-horizon value appears only after the continuation is fed back into the component. In this paper we make this feedback operation explicit. The proposed viewpoint is simple but deliberately restricted: Bellman recursion is a guarded fixed-point trace . A policy closes the action interface of a one-step stochastic component; discounting makes the continuation map contractive; and policy evaluation is the Banach fixed point obtained by closing the remaining value wire. This converts a familiar analytic fact into a compositional semantics. Series wiring becomes composition of continuation transformers, parallel wiring becomes tensor/product semantics, and feedback becomes a guarded fixed-point trace in the sense of traced monoidal reasoning ( 15 ; 12 ; 26 ) and iteration/fixed-point semantics ( 4 ) , specialized here to contractive maps. We use “trace” in this guarded sense throughout: only maps whose feedback wire is uniformly contractive are traceable, and all trace laws are invoked only inside this admissible class. This shift matters because modern decision systems are not monolithic MDPs. They are assembled from modules: perception or representation maps, local controllers, reward shapers, safety monitors, belief-state filters, and off-policy evaluation components. Belief-state filtering is the standard semantic reduction for partially observed control ( 27 ; 16 ) , while off-policy evaluation is naturally phrased as a change of measure between trajectory laws ( 22 ; 14 ; 31 ) . A theory that only describes the final closed MDP has limited language for asking whether a local replacement is valid inside an arbitrary surrounding system. The semantic problem is therefore contextual: if two open components are close at their interface, when are they close after arbitrary composition and feedback? We answer this by treating open decision components as typed semantic objects. Each component induces a Bellman transformer from output continuations to input values. The transformer type records the direction in which value information flows. Once types are explicit, composition is no longer an informal engineering operation: it is ordinary function composition, with discount factors recording the depth at which each local reward or perturbation enters the feedback loop. This yields quantitative contextual equivalence bounds for the admitted class of well-typed linear one-hole contexts: the hole is used once, side components are fixed, tensoring does not copy the hole, and every feedback node satisfies a uniform guardedness certificate. The same semantics also clarifies abstraction. A deterministic representation map is a coalgebraic morphism when it commutes with the one-step transition and preserves rewards ( 25 ; 13 ) . In RL and planning, closely related abstraction, homomorphism, and bisimulation principles have been used to characterize when reduced models preserve decision-relevant behavior ( 11 ; 24 ; 20 ; 7 ; 8 ) . Exact commutation gives value preservation by an operator-intertwining proof. Approximate commutation gives an explicit sup-norm distortion bound controlled by reward mismatch and total-variation transition mismatch. Because the error is stated at the transformer level, it propagates through admitted larger decision circuits by the contextual congruence theorem. Finally, the framework separates metric fixed-point reasoning from order-theoretic contract reasoning. Discounted real-valued returns use Banach contraction. Safety, risk, and resource constraints are often better expressed as ordered guarantees. We therefore add a quantale-valued contract layer ( 19 ; 6 ; 32 ) . In this layer, a circuit is admitted when its open semantics is a monotone \omega -continuous contract transformer and its wiring obeys the compilation laws stated later. Under these assumptions, closed loops are interpreted as least fixed points, and local pre-fixed-point certificates lift compositionally through wiring and feedback. This layer connects the paper to classical lattice-theoretic and fixed-point semantics ( 30 ; 18 ; 28 ; 5 ) . The contribution is not a new learning algorithm and not an empirical claim. It is a theoretical reframing of RL semantics around contractive feedback. The main results are as follows. • Guarded traced Bellman semantics. We formulate open discounted decision components and prove that policy evaluation is the guarded Banach trace of an admissible feedback circuit; no global trace operation on arbitrary maps is assumed. • Typed compositionality. We prove that series and parallel wiring correspond to composition and tensoring of Bellman transformers, with explicit discount bookkeeping. • Contextual equivalence. We define an operator metric on invariant value balls and prove that approximate equivalence is preserved by admitted well-typed linear contexts generated by composition, tensoring, and guarded trace. • Coalgebraic abstraction. We prove exact value preservation under commuting homomorphisms and quantitative value distortion under approximate commutation. • Quantale contracts. We prove that inductive safety/risk/resource contracts lift through series, parallel, and feedback wiring whenever the contract-transformer semantics is monotone, \omega -continuous, and compatible with the stated wiring laws. • Extensions and sanity checks. We show how the same interfaces cover belief-state POMDP lifting, decomposable change of measure under explicit factorization assumptions, time-varying operator tracking, and minimal modular robustness examples. The paper is organized as follows. Section 2 gives the semantic principles. Section 3 fixes the categorical, metric, and order-theoretic interfaces. Section 4 proves Bellman-as-guarded-trace. Section 5 proves compositionality and contextual congruence. Section 6 develops exact and approximate abstraction. Section 7 gives the quantale contract layer. Sections 8 – 10 record extensions and explicit robustness examples. All proofs are collected in Appendix A . 2 Background and Motivation The paper is based on three semantic principles. Feedback principle. Infinite-horizon value is not merely a solution of an equation; it is the result of feeding a continuation output back into a one-step