eXplainable AI for data driven control: an inverse optimal control approach

Federico Porcari, Donatello Materassi, Simone Formentin · 2025

Read eXplainable AI for data driven control: an inverse optimal control approach by Federico Porcari, Donatello Materassi, Simone Formentin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11446.txt.

Introduction

Federico Porcari, Donatello Materassi, and Simone Formentin wrote this research paper, published in 2025. It explains black-box controllers, systems that steer machines well but opaquely, by reverse-engineering the goal they seem to chase.

The method treats explanation as an inverse problem: find the cost function, the balance of tracking error against effort, that would make the observed controller optimal. Solving an inverse linear quadratic problem yields those hidden weights. Like the popular LIME technique, the explanation is local to a working region. Unlike plain sensitivity checks, it speaks the language of control, so engineers can judge whether the controller's trade-offs make sense. Worked examples show the inferred costs exposing odd or counterintuitive behavior. The authors pitch the approach as a transparency tool for data-driven control design.

Catalog record

CREATORFederico Porcari, Donatello Materassi, Simone Formentin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11446

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

eXplainable AI for data driven control: an inverse optimal control approach Federico Porcari, Donatello Materassi, Simone Formentin Abstract Understanding the behavior of black-box data-driven controllers is a key challenge in modern control design. In this work, we propose an eXplainable AI (XAI) methodology based on Inverse Optimal Control (IOC) to obtain local explanations for the behavior of a controller operating around a given region. Specifically, we extract the weights assigned to tracking errors and control effort in the implicit cost function that a black-box controller is optimizing, offering a more transparent and interpretable representation of the controller’s underlying objectives. This approach presents connections with well-established XAI techniques, such as Local Interpretable Model-agnostic Explanations (LIME) since it is still based on a local approximation of the control policy. However, rather being limited to a standard sensitivity analysis, the explanation provided by our method relies on the solution of an inverse Linear Quadratic (LQ) problem, offering a structured and more control-relevant perspective. Numerical examples demonstrate that the inferred cost function consistently provides a deeper understanding of the controller’s decision-making process, shedding light on otherwise counterintuitive or unexpected phenomena. I Introduction AI has been advancing rapidly, leading to its application across a wide array of domains, from healthcare and finance to manufacturing and transportation [ 1 , 2 , 3 , 4 , 5 ] . However, a major problem with AI-based algorithms is the lack of transparency in their decision-making processes [ 6 , 7 ] , which has spurred the development of a new research field called eXplainable Artificial Intelligence (XAI), specifically focused on creating methods to interpret how these systems make decisions [ 8 , 9 , 10 ] . Despite the growing interest for XAI methods, there is no clear consensus or formalization on what constitutes an “explanation.” Currently, the most common distinction in XAI methods is between global explanations, which seek to provide a broad understanding of the model’s behavior across the entire input space, and local explanations, which focus on analyzing the model’s decisions for a specific instance or trajectory, offering insights into why a particular outcome was produced in a given scenario [ 11 ] . Common global explanation techniques include approximations using decision trees [ 12 ] , rule-based models [ 13 ] , and Accumulated Local Effects (ALE) along with its variations [ 14 , 15 ] ; while prevalent local explanation methods include LIME (Local Interpretable Model-Agnostic Explanations) [ 7 ] , SHAP (SHapley Additive exPlanations) [ 16 , 17 ] , counterfactual explanations [ 18 , 19 ] , and saliency maps. Typically, local explanations are easier to obtain, as they require analyzing only a limited region of the input space rather than the entire model. Despite their localized scope, they remain highly valuable, particularly for debugging purposes and individual failure analysis. Additionally, local explanations are often easier to categorize, as they frequently belong to the class of feature attribution methods [ 16 ] . In these methods, the explanation takes the form of a vector, with each entry indicating the contribution of an individual input component to the output, offering a structured and direct way to assess input relevance. More recently, AI applications have also been extending into the design of control algorithms, where AI techniques are leveraged to improve the performance of self-operating machines such as drones, self-driving cars, and robotic systems [ 20 , 21 ] . The lack of transparency in AI-based control algorithms is especially problematic for autonomous systems, since it presents very specific challenges not satisfactorily addressed by current state-of-the-art XAI methods. Indeed, control systems typically operate through a dynamic process where decisions are not made in isolation; instead, they take decisions in real-time often solving constrained optimization problems over an extended time horizon, and operating within a feedback loop alongside the system under control. All these characteristics add increased complexity to understanding their behavior and driving increased research interest in developing dedicated methods for interpretable control [ 22 , 23 , 24 , 25 ] . A significant challenge in applying XAI methods to control problems is precisely that most existing XAI techniques are designed to explain static quantities, such as individual predictions in one-shot classification or regression tasks. Unlike static features, which have well-defined individual values, control systems operate over time, generating time series data that embody complex temporal dependencies and evolving patterns [ 26 , 27 ] . The main goal of this article is to develop a local explainability tool, that is specifically designed for control systems. Unlike standard XAI methods, which typically focus on estimating how much a single static feature influences the output, our approach aims to assess how individual temporal signals affect the system’s behavior over time. This is achieved by leveraging specific control-theoretical tools. Given a black-box controller and a particular trajectory to explain, we obtain a local explanation through a linearization procedure around the trajectory to be explained, followed by solving an inverse optimal control (IOC) problem. The cost function derived from the inverse optimal control process serves as the explanation, as its knowledge contains information about the relative importance of each input and output time series in terms of control effort and performance. This provides insights into what the black-box controller is effectively optimizing, shedding light on the implicit trade-offs guiding its decision-making process. Interestingly, this specific explanation tool can be encapsulated within a more general framework that enables a more formal definition of explanations. Under this framework, the core principle is to view an explanation as a form of reduced complexity model [ 28 ] . Like many existing XAI methods, our approach recognizes that explanations often rely on an underlying model, whether explicitly constructed or implicitly assumed. However, what sets our framework apart is the introduction of an explicit interpretability function , which provides a formal mechanism for extracting relevant metrics from the simplified model. This function dictates how the explanation should be constructed and evaluated, ensuring that the process is rigorous and systematic rather than ad hoc. As already mentioned, this article considers the weight costs obtained from of an inverse optimal control problem at the output of interpretability function. However, it is also discussed how alternative interpretability functions could be employed, establishing connections with other existing XAI methods and further generalizing the applicability of our framework. The article describes in Section II the main problem that we want to address in a formal way. Section III introduces different interpretability functions that can be applied to LTI systems. In particular, we emphasize the relevance of inverse optimal control methods in revealing the trade-offs that the black-box controller is effectively operating within its control policy. Section IV provides several numerical illustrations demonstrating how our approach can highlight phenomena in controlled systems that are not immediately intuitive. II Problem Statement We consider a discrete-time, time-invariant dynamical system described by: \displaystyle x(k+1) \displaystyle=f(x(k),u(k)), (1a) \displaystyle y(k) \displaystyle=g(x(k),u(k)), (1b) where f and g are unknown functions governing the system dynamics. Here, x(k) represents the system state at time step k , while u(k) and y(k) denote the input and output signals, respectively. We assume that u(k) is a controllable input, and we have access to experimental data

Read the complete text