Automating the Refinement of Reinforcement Learning Specifications

Tanmay Ambadkar, Đorđe Žikelić, Abhinav Verma · 2025

Read Automating the Refinement of Reinforcement Learning Specifications by Tanmay Ambadkar, Đorđe Žikelić, Abhinav Verma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01047.txt.

Introduction

Tanmay Ambadkar, Đorđe Žikelić, and Abhinav Verma automate the sharpening of learning specs in a 2025 paper. Logical specifications can guide reinforcement learning agents toward complex tasks by stating rules the behavior must satisfy. When a spec is under-specified, though, it leaves too much open, and agents may fail to learn useful policies.

The authors explore improving coarse-grained specs through an exploration-guided strategy. The agent's own exploration reveals which parts of the spec need refinement. Automating that loop could save the hand-tuning that now limits spec-based training. Better specs mean agents that learn intended behavior, not loopholes.

Catalog record

CREATORTanmay Ambadkar, Đorđe Žikelić, Abhinav Verma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-01047

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automating the Refinement of Reinforcement Learning Specifications Tanmay Ambadkar, Đorđe Žikelić, Abhinav Verma Abstract Logical specifications have been shown to help reinforcement learning algorithms in achieving complex tasks. However, when a task is under-specified, agents might fail to learn useful policies. In this work, we explore the possibility of improving coarse-grained logical specifications via an exploration-guided strategy. We propose AutoSpec , a framework that searches for a logical specification refinement whose satisfaction implies satisfaction of the original specification, but which provides additional guidance therefore making it easier for reinforcement learning algorithms to learn useful policies. AutoSpec is applicable to reinforcement learning tasks specified via the SpectRL specification logic. We exploit the compositional nature of specifications written in SpectRL, and design four refinement procedures that modify the abstract graph of the specification by either refining its existing edge specifications or by introducing new edge specifications. We prove that all four procedures maintain specification soundness, i.e. any trajectory satisfying the refined specification also satisfies the original. We then show how AutoSpec can be integrated with existing reinforcement learning algorithms for learning policies from logical specifications. Our experiments demonstrate that AutoSpec yields promising improvements in terms of the complexity of control tasks that can be solved, when refined logical specifications produced by AutoSpec are utilized. 1 Introduction Reinforcement Learning (RL) algorithms have made tremendous strides in recent years 33 ; 32 ; 24 ; 21 . However, most algorithms assume access to a scalar reward function that must be carefully engineered to make environments amenable to RL—a practice known as reward engineering 13 . This creates challenges in applying RL to new environments where useful reward functions are hard to construct. Furthermore, scalar Markovian rewards cannot provide sufficient feedback for certain tasks 1 ; 5 , leading to growing interest in non-Markovian reward functions 23 ; 18 ; 4 . To make non-Markovian rewards tractable, it is standard to represent them via logical specification formulas that capture the intended task. These approaches, known as specification-guided reinforcement learning 2 ; 22 ; 14 ; 17 ; 18 , derive reward functions from logical specifications. However, this creates two challenges: (i) providing specifications granular enough to guide RL algorithms, and (ii) defining accurate labeling functions mapping environment states to specification predicates. Users often create coarse specifications or labeling functions that, while logically correct, provide insufficient guidance for learning. Figure 1: Example of refinement by AutoSpec in a 9-rooms environment. The original MID-node region includes a trap state from which recovery is impossible. The refined specification excludes this trap, enabling the agent to learn a policy with higher satisfaction probability. We present AutoSpec , a framework for automatically refining coarse specifications without user intervention. We say that a logical specification is coarse (or under-specified ) if its predicate labelings or logical structure are too coarse to allow specification-guided RL algorithms to translate logical specifications into reward functions that allow for effective learning of RL policies. AutoSpec starts with an initial logical specification, translates it to a reward function, and attempts to learn a policy. If the learned policy’s performance is unsatisfactory, AutoSpec identifies which specification components cause learning failures and automatically refines both the specification formula and labeling function. The refined specification’s satisfaction implies the original’s satisfaction while providing additional structure for learning. This process repeats until a satisfactory policy is learned. AutoSpec works with SpectRL specifications; boolean and sequential combinations of reach-avoid tasks 17 . Any SpectRL specification decomposes into an abstract graph where edges specify reach-avoid tasks 18 . AutoSpec identifies problematic edges and applies targeted refinements: either modifying the labeling function for regions (Figure 1 ) or restructuring the graph to add alternative paths. These problematic edges are identified by employing an exploration-guided strategy that utilizes empirical trajectory data to identify edges in the abstract graph whose reach-avoid tasks (i.e. initial, target or unsafe regions) make it hard to learn a good RL policy. For instance, in Figure 1 , the initial MID region of the MID - GOAL edge in the abstract graph is under-specified, as it overlaps with a trap state. The trap state is not immediately obvious as there is a path from MID to GOAL . By analyzing explored traces, AutoSpec identifies problematic start states in MID and refines the region to exclude the trap (as shown in Figure 1 ), thereby refining the logical reach-avoid specification associated to the MID - GOAL edge. We prove that all refinements maintain soundness, where satisfaction of the refined specification implies satisfaction of the original. AutoSpec integrates with existing SpectRL-compatible algorithms as demonstrated with DiRL 18 and LSTS 31 . Our contributions: 1. A framework for automated refinement of logical RL specifications with four refinement procedures, all with formal soundness guarantees (Section 3 ). 2. Integration with existing specification-guided RL algorithms, enabling them to solve tasks with coarse specifications (Section 3 ). 3. Empirical demonstration that AutoSpec enables learning from specifications that existing methods cannot handle (Section 4 ). Related work. Recent years have seen substantial progress in solving RL tasks specified via logical specifications 2 ; 22 ; 14 ; 6 ; 9 ; 12 ; 11 ; 10 ; 17 ; 18 ; 36 . Many works consider different fragments of Linear Temporal Logic (LTL) or their variants for specifying RL tasks. 14 ; 6 consider tasks that can be specified using deterministic finite automata (DFA) and solve them by reward machines , which decompose these tasks and translate them into a reward function. The reward function can then be used to train existing RL algorithms. 22 considers a variant of LTL called TLTL for specifying tasks and propose a method for translating these specifications into continuous reward functions. 12 ; 11 ; 10 study the translation of tasks specified in LTL into reward functions. 3 examines the theoretical questions related to the translation of logical specifications into reward functions. 17 defines the specification language SpectRL, a finitary fragment of LTL and provides justification for using this language to define specifications for RL tasks. A compositional method that decomposes SpectRL specifications into an abstract graph and constructs a reward function for each abstract graph edge was proposed in 18 . The approach by 34 focuses on discovering optimal reward structures through environmental exploration and reward analysis. Compositional methods are further explored by 26 , who propose removing unfulfillable subtasks, and 25 , who introduce verification techniques to certify learned policies. 38 propose Claps , a compositional method for learning neural network policies with formal guarantees on the satisfaction of SpectRL specifications, thus advancing the applicability to safety-critical RL applications by utilizing prior methods for learning reach-avoid policies with formal guarantees 20 ; 37 ; 7 . Recent advancements also include LTL2Action 35 , which translates LTL specifications into sequences of tasks for RL agents. Other recent approaches, such as 28 and DeepLTL 15 , leverage goal-conditioned RL and automata-based architectures to solve complex LTL and \omega -regular tasks zero-shot or in multi-task settings. Trainify 16 employs counterexample-guided abstraction and refinement (CEGAR) to iteratively improve policies by addressing failure cases identified through counterexamples. However, these works primarily focus on learning policies for fixed, well-defined specifications. In contrast, AutoSpec studies the problem

Read the complete text