SymDQN: Symbolic Knowledge and Reasoning in Neural Network-based Reinforcement Learning

Ivo Amador, Nina Gierasimczuk · 2025

Read SymDQN: Symbolic Knowledge and Reasoning in Neural Network-based Reinforcement Learning by Ivo Amador, Nina Gierasimczuk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02654.txt.

Introduction

Ivo Amador and Nina Gierasimczuk present, in a 2025 paper, a learning system that lets a neural network take guidance from symbolic rules. Pure network learners can act in odd ways because they lack explicit reasoning. Their design adds rule-based modules to a dueling deep network learner.

The test world is a five-by-five grid where an agent meets shapes tied to rewards. Modules handle shape recognition and reward prediction, steering the agent's choices. A step-by-step study removed parts of the system to see what each module added. The agent's behavior stayed closer to what reasoning about the world would suggest. The work shows one way to blend logic with learning without losing the learner's flexibility.

Catalog record

CREATORIvo Amador, Nina Gierasimczuk
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02654

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SymDQN: Symbolic Knowledge and Reasoning in Neural Network-based Reinforcement Learning Ivo Amador, Nina Gierasimczuk Abstract We propose a learning architecture that allows symbolic control and guidance in reinforcement learning with deep neural networks. We introduce SymDQN , a novel modular approach that augments the existing Dueling Deep Q-Networks ( DuelDQN ) architecture with modules based on the neuro-symbolic framework of Logic Tensor Networks (LTNs). The modules guide action policy learning and allow reinforcement learning agents to display behavior consistent with reasoning about the environment. Our experiment is an ablation study performed on the modules. It is conducted in a reinforcement learning environment of a 5x5 grid navigated by an agent that encounters various shapes, each associated with a given reward. The underlying DuelDQN attempts to learn the optimal behavior of the agent in this environment, while the modules facilitate shape recognition and reward prediction. We show that our architecture significantly improves learning, both in terms of performance and the precision of the agent. The modularity of SymDQN allows reflecting on the intricacies and complexities of combining neural and symbolic approaches in reinforcement learning. 1 Introduction Despite its rapidly growing impact on society, Artificial Intelligence technologies are tormented by reliability issues, such as lack of interpretability, propagation of biases, difficulty in generalizing across domains, and susceptibility to adversarial attacks. A possible way towards more interpretable, controlled and guided algorithms leads through the field of neuro-symbolic AI, which explores new ways of integrating symbolic, logic-based knowledge in neural networks (NNs). In particular, the framework of Logic Tensor Networks (LTNs, for short) Serafini and d’Avila Garcez 2016 ; Badreddine et al. 2022 enhances learning by interpreting first-order logic formulas concretely on data used by NNs algorithms. Such formulas express properties of data and, given a fuzzy semantics, can be integrated into the loss function, thus guiding the learning process. In this paper, we apply LTNs to a reinforcement learning problem. By integrating LTNs in the training process, our learning agent uses logic to learn the structure of the environment, to predict how different objects in the environment interact with each other, and to guide its actions by performing elementary reasoning about rewards. We investigate how such integration affects learning performance of a robust, established and well-studied framework of Dueling Deep Q-Network ( DuelDQN , for short) Wang et al. 2016 . The structure of the paper is as follows. In Section 2 we briefly recall Logic Tensor Networks and elements of the underlying Real Logic. In Section 3 we introduce our methodology: the SymDQN architecture and its training process. We follow up with the presentation of the experiment in Section 4 . We discuss of the results in Section 5 . Section 6 concludes and outlines directions for future work. Related Work Since its conception, the framework of LTNs has been applied in various domains. In computer vision, LTNs were used to inject prior knowledge about object relationships and properties, improving interpretability and accuracy in object detection Donadello et al. 2017 . Their addition to convolutional neural networks improves the robustness on noisy data Manigrasso et al. 2021 . They enhance the accuracy in reasoning tasks in open-world and closed-world scenarios Wagner and d’Avila Garcez 2022 . In Bianchi and Hitzler 2019 , LTNs are leveraged for deductive reasoning tasks. Finally, in learning by reinforcement LTNs were used to integrate prior knowledge into reinforcement learning agents improving both the learning rate and robustness to environmental changes Badreddine and Spranger 2019 . The latter work is similar to ours in the choice of tools, but it differs in its methodology. In Badreddine and Spranger 2019 , LTN is a separate pre-training module which interacts with DuelDQN only by creating inputs. In contrast, our SymDQN integrates LTN in the training process (making it learn alongside DuelDQN ). Our work uses logic to adjust elements of a reinforcement learning framework. In that, it is related to reward shaping approaches, where the learner is given external symbolic information about the environment, e.g., in the form of linear time logic formulas (also known as restraining bolts) in Giacomo et al. 2019 or of an induced automaton in Furelos-Blanco et al. 2021 . In a way, the LTN approach is similar: logical formulas adjust the reinforcement learning process. However, our technique is a more distributed form of reward shaping. First, the formulas of Real Logic are used as guides to obtain a symbolic representation of the environment, then to predict immediate rewards from encountering the objects of the environment. Finally, a logical formula is used to help the learner align the q -values (the agent’s long term policy) with the predicted immediate rewards of symbolically represented objects. In other words, we restrain the reinforcement learner by expecting it to reason about its behavior as it learns, and we investigate the impact of this restriction on learning precision and performance. We will come back to this issue in Section 5.1 , after we have detailed all the components. 2 Real Logic Real Logic is the basis of the functioning of LTNs. In this section we provide a rudimentary introduction (for a full exposition consult Badreddine et al. 2022 ). Let us start with a simple example. Example 1 . Consider two datasets: a data set of humans (with two features: age and gender), and a dataset of pets (with three features: height, weight and color). Assume that Alice appears in the data set of humans (for instance as a five year old female), and Max and Mittens are listed in a dataset of pets. To be able to talk about Alice, Max and Mittens, we need a logical language that includes constants referring to objects (particular rows of the datasets). Note that such constants can be of different types —in our example humans consists of two, while pets are composed of three features. The signature of the language of Real Logic \mathcal{L} contains a set \mathcal{C} of constant symbols, a set \mathcal{F} of function symbols, a set \mathcal{P} of predicate symbols, and a set \mathcal{X} of variable symbols. Let \mathcal{D} be a non-empty set of domain symbols (that represent types). Domain symbols are used by functions \mathbf{D} , \mathbf{D_{in}} , and \mathbf{D_{out}} which for a given element of the signature output its type, in the following way. \mathbf{D}:\mathcal{X}\cup\mathcal{C}\rightarrow\mathcal{D} specifies the types for variables and constants; \mathbf{D_{in}}:\mathcal{F}\cup\mathcal{P}\rightarrow\mathcal{D}^{\ast} specifies the types of the sequence of arguments allowed by function and predicate symbols ( \mathcal{D}^{\ast} stands for the set of all finite sequences of elements from \mathcal{D} ); \mathbf{D_{out}}:\mathcal{F}\rightarrow\mathcal{D} specifies the type of the range of a function symbol. Example 2 . Continuing Example 1 , let the language of pet-ownership \mathcal{L}_{pets} have the signature consisting of the set of constants \mathcal{C}=\{\textsc{Alice},\textsc{Max},\textsc{Mit}\} , a set of function symbols \mathcal{F}=\{\textsc{owner}\} , a set of predicate symbols \mathcal{P}=\{\textsc{isOwner}\} , and two variable symbols \mathcal{X}=\{\textsc{pet},\textsc{person}\} . Further, we have two domain symbols, one for the domain of humans and one for pets, \mathcal{D}=\{H,P\} . Then, our domain functions can be defined in the following way. \mathbf{D}(\textsc{Alice})=H (Alice is a constant of type H ), \mathbf{D}(\textsc{Max})=\mathbf{D}(\textsc{Mit})=P (Max and Mittens are of type P ). Further, each dataset will have its own variable: \mathbf{D}(\textsc{pet})=P , \mathbf{D}(\textsc{person})=H . We

Read the complete text