Interpretable by Design: Query-Specific Neural Modules for Explainable Reinforcement Learning

Mehrdad Zakershahrak · 2025

Read Interpretable by Design: Query-Specific Neural Modules for Explainable Reinforcement Learning by Mehrdad Zakershahrak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08749.txt.

Introduction

Mehrdad Zakershahrak redesigns reinforcement learning agents as queryable knowledge bases rather than pure control machines. A trained agent silently learns a great deal about its world, including reachability, distances, and dynamics, but standard architectures bury that knowledge where nobody can ask for it.

The proposed architecture, QDIN, gives each query type its own specialized neural module: one for policy questions, others for reachability, paths, and comparisons. The striking finding is a decoupling between knowing and doing. Inference accuracy can reach near perfection, with reachability scores around ninety-nine percent, while control returns stay near thirty-one percent. Query-specialized designs beat both unified models and after-the-fact extraction, while keeping competitive control. The author frames this as a research agenda for interpretable, verifiable agents.

Catalog record

CREATORMehrdad Zakershahrak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08749

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interpretable by Design: Query-Specific Neural Modules for Explainable Reinforcement Learning Mehrdad Zakershahrak Abstract Reinforcement learning has traditionally focused on a singular objective: learning policies that select actions to maximize reward. We challenge this paradigm by asking: what if we explicitly architected RL systems as inference engines that can answer diverse queries about their environment? In deterministic settings, trained agents implicitly encode rich knowledge about reachability, distances, values, and dynamics—yet current architectures are not designed to expose this information efficiently. We introduce Query-Conditioned Deterministic Inference Networks (QDIN), a unified architecture that treats different types of queries (policy, reachability, paths, comparisons) as first-class citizens, with specialized neural modules optimized for each inference pattern. Our key empirical finding reveals a fundamental decoupling: inference accuracy can reach near-perfect levels (99% reachability IoU) even when control performance remains suboptimal (31% return), suggesting that the representations needed for accurate world knowledge differ from those required for optimal control. Experiments demonstrate that query-specialized architectures outperform both unified models and post-hoc extraction methods, while maintaining competitive control performance. This work establishes a research agenda for RL systems designed from inception as queryable knowledge bases, with implications for interpretability, verification, and human-AI collaboration. 1 Introduction Consider a robot navigating a warehouse. When asked ”Can you reach the loading dock from here?”, current RL systems would need to either execute a trajectory or run expensive planning algorithms, despite having learned representations that implicitly encode this information. This limitation stems from a fundamental design choice: we architect RL systems as action selectors, not as inference engines. 1.1 The Limitations of Action-Centric Design Modern reinforcement learning has achieved remarkable success by optimizing a single objective: mapping states to actions that maximize expected return [ 46 , 61 , 62 , 51 , 70 ] . This action-centric paradigm has driven decades of algorithmic development, from Q-learning to policy gradients to actor-critic methods [ 64 , 75 , 76 , 58 , 59 , 45 , 40 , 25 ] . Yet this narrow focus constrains both how we design RL architectures and how we deploy them in practice. A trained RL agent, especially in deterministic environments, possesses extensive implicit knowledge: • Reachability: Which states can be reached within a time budget? • Distances: What is the shortest path between arbitrary locations? • Comparisons: Which of multiple goals is more valuable or closer? • Counterfactuals: What would happen under alternative action sequences? Current architectures bury this knowledge within monolithic networks optimized solely for action selection. Extracting it requires either expensive post-hoc analysis [ 22 , 77 , 63 , 7 ] or training separate models [ 74 , 38 ] —neither approach is satisfactory for real-time deployment. 1.2 Toward Query-Driven Architectures We propose a paradigm shift: design RL architectures explicitly as query-driven inference systems . Rather than treating query-answering as an auxiliary capability bolted onto action-selection machinery, we architect networks with specialized modules optimized for different types of inference from the ground up. Policy execution becomes one query type among many, not the sole focus of optimization. This reconceptualization has profound implications. In applications requiring interpretability [ 54 , 1 , 23 , 8 ] , safety verification [ 4 , 18 , 2 , 53 ] , or human collaboration [ 14 , 3 , 83 ] , the ability to query an agent’s knowledge often matters more than optimal action selection. A warehouse robot that can accurately answer reachability queries provides more value than one with slightly better navigation but opaque reasoning. 1.3 Contributions We instantiate this vision through three contributions: 1. Architectural Innovation: We introduce QDIN (Query-Conditioned Deterministic Inference Networks), featuring specialized neural modules for different query types—convolutional decoders for spatial reachability, attention mechanisms for comparisons, and sequential models for path generation—unified through query-conditioned processing. 2. Empirical Discovery: We demonstrate a fundamental decoupling between inference accuracy and control performance. Our experiments reveal that reachability prediction can achieve near-ceiling accuracy (99% IoU) even when returns remain poor (31%), suggesting distinct representational requirements for knowledge versus control. 3. Research Agenda: We establish query-driven RL as a research direction, identifying key challenges in architecture design, training procedures, and evaluation metrics that prioritize queryability alongside traditional performance measures. 2 Related Work 2.1 From Models to Oracles Model-based reinforcement learning [ 57 , 26 , 27 , 33 , 32 , 13 ] learns environment dynamics primarily for planning. While these methods construct world models, they typically use them only to simulate trajectories for policy improvement. Value Iteration Networks [ 66 , 39 , 50 ] embed differentiable planning modules but still optimize for policy extraction. Our work extends this by treating the learned model as a queryable knowledge base with dedicated inference interfaces. The concept of agents as oracles has appeared in theoretical contexts [ 35 , 34 ] but has not driven architectural design. Recent work on world models [ 24 , 56 ] demonstrates that agents learn rich representations, yet these approaches still funnel knowledge through action selection rather than exposing it directly. 2.2 Interpretability and Explanation Post-hoc interpretability methods [ 22 , 47 , 30 , 79 , 81 , 80 ] attempt to explain trained policies through visualization, saliency maps, or model distillation. These approaches face a fundamental limitation: they try to extract information from architectures not designed for extraction. In contrast, we build interpretability into the architecture itself through query-specific modules. Explanation generation in RL [ 28 , 60 , 43 ] focuses on producing human understandable descriptions of agent behavior. While valuable, these methods typically operate on top of standard architectures. Our approach inverts this—the architecture itself is designed to answer explanatory queries efficiently. 2.3 Multi-Task and Meta-Learning Multi-task RL [ 55 , 67 , 29 , 52 ] trains agents on multiple reward functions, while meta-learning [ 17 , 15 , 72 , 49 ] enables rapid adaptation to new tasks. These paradigms share our use of shared representations but differ fundamentally in purpose. We don’t train on multiple tasks; we train on multiple query types about the same task, with architectural specialization for each query modality [ 21 , 20 , 19 ] . Hierarchical RL [ 37 , 69 , 9 , 65 , 48 , 78 ] decomposes problems into subtasks, creating natural query boundaries. While hierarchical methods could complement our approach, they typically focus on temporal abstraction for control rather than enabling diverse inference capabilities. 2.4 Graph Neural Networks and Structured Computation Graph neural networks for RL [ 82 , 73 , 36 , 11 ] provide relational inductive biases that could complement our query-specific architectures. While GNNs excel at relational reasoning, our approach differs by using specialized architectures matched to query structure rather than general message passing. Neural module networks [ 6 , 31 ] share our vision of compositional architectures but focus on visual question answering rather than RL inference. 2.5 Differentiable Programming Recent work on differentiable programming [ 44 , 5 ] provides tools for embedding optimization within neural networks. These techniques could enhance our path-finding and comparison modules, providing stronger inductive biases for structured queries. 3 Problem Formulation 3.1 Deterministic MDPs as Queryable Systems Consider a deterministic MDP \mathcal{M}=(\mathcal{S},\mathcal{A},T,r,\gamma,\rho_{0}) where the transition function T:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S} is deterministic. This determinism implies

Read the complete text