Training Observable Control Policies to Expose Agent State Through Actions
Andres Enriquez Fernandez, John J. Bird · 2026
Read Training Observable Control Policies to Expose Agent State Through Actions by Andres Enriquez Fernandez, John J. Bird (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27609.txt.
Introduction
Autonomous agents often cannot radio their status to partners or controllers. Physical limits or mission rules may block communication entirely. Andres Enriquez Fernandez and John Bird ask whether actions alone can reveal an agent's state. An observer can watch what an agent does even when no messages arrive. Their work trains control policies that make this watching easier.
The team uses reinforcement learning to shape the policies. The training reward includes a score for how well an estimator can recover the state from actions. Policies with this reward become more observable. Simulations of the trained agent confirm the effect. In an aircraft tracking problem, one such policy was found. It improved state estimation a great deal. It also had minimal impact on normal task performance. The result suggests coordination may be possible from action watching alone.
Catalog record
| CREATOR | Andres Enriquez Fernandez, John J. Bird |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27609
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training Observable Control Policies to Expose Agent State Through Actions Estimating System State from the Actions of a Reinforcement Learning Agent, AIAA 2023-2657, Session: Autonomy V, Published Online:19 Jan 2023https://doi.org/10.2514/6.2023-2657, AIAA Scitech 2023 Forum, 23-27 January 2023, National Harbor, MD & Online Enabling Inter-Vehicle Coordination Through Observable Control Policies, AIAA 2024-0989, Session: Autonomy, Published Online: 4 Jan 2024https://doi.org/10.2514/6.2024-0989, AIAA Scitech 2024 Forum, 8-12 January 2024, Orlando, FL Andres Enriquez Fernandez, John J. Bird Abstract Physical or operational constraints often impose communications limitations on autonomous agents. Such limitations complicate monitoring or multiagent coordination. Even when strong communications are absent, some information may still be available. The remainder of the relevant agent state may be reconstructed via estimation. The actions taken by an agent are a potential source of information – as the agent interacts with the environment, these actions may be observed even in the absence of explicit communication. We investigate using actions to estimate the state of an agent, using reinforcement learning to develop policies which make the estimation problem more tractable. Policy observability is encouraged through the training reward and is analyzed using simulation of the trained agent. In an aircraft tracking problem a policy with enhanced observability is found that has minimal impact on nominal task performance. 1 Introduction Effective collaboration between cooperating agents (including autonomous to autonomous agents and human to autonomous agent coordination) typically depends on a mutual understanding of at least a subset of the states of each agent. This is often achieved via direct communication that enables complex behavior, coordination, environmental information sharing, and sub-task learning [ 15 , 4 ] . Reliability and availability of communications is not assured however. Communications limitations may appear at the channel level as delays, drops, and bandwidth limitations. Limited or absent direct communication can also occur because of physical or operational considerations. The environment may inhibit communication (e.g. underwater), the operation may be compromised by excessive communication, or the agents may not be equipped with common hardware or language for direct communication. Casting the problem as a partially observable Markov Decision Process (POMDP) offers one approach to resolve uncertainty introduced when states are not directly communicated. Under this formulation, the state vector of every agent includes the agent’s belief of the state of the other agents [ 18 ] , and the agents are rewarded for their collective actions [ 10 ] . This couples the decision-making model to the collective actions of all the agents. One implication of this structure is that changes to the task of even a single agent requires retraining all the agents, limiting practicality in a federated system. Another practical constraint is that the POMDP approach scales poorly as the dimension of joint action, state, and observation spaces grow exponentially with the number of agents [ 1 ] . When coordination is approached as a POMDP, an observer is implicitly developed in the agent through the training process [ 14 ] . An alternative which “decouples” the agents is to adopt the observer-controller structure common in control systems, explicitly formulating an estimator rather than training it implicitly through the expanded agent state. Here, the estimator assimilates a subset of the available information to infer the needed states of other agents, and enable coordination [ 13 ] . One option for the observations are the control actions taken by agents. These observations have the advantage that an observation model is already defined (the control policy), and that in some cases they may be visible through direct observation without explicit communication; in effect they may be “broadcast” for free. Informal examples of such implicit information sharing through the observation of actions occur in a number of contexts. For example, a ground vehicle rapidly braking or changing lanes can indicate an obstacle in the road, an aircraft observed to make a course change in the presence of nearby traffic has likely detected it visually, and an aircraft which deploys its undercarriage is likely near an airport. Enabling this implicit communication, and increasing its efficiency can enable coordination between autonomous agents in communications limited scenarios. It may also enhance the comfort and safety of mixed human-autonomous operations as human agents are more likely to understand what the agent is doing. To improve observability of an agent’s control policy, we formulate an estimator which uses as observations the actions chosen by a control policy. We train the policy to reward the estimator’s performance while the agent performs a target tracking task. The observable policy is compared to one trained only to perform the tracking task, comparing task-only performance, estimate quality, and a measure of observability. While one objective of this work is to enable coordination in communication-limited scenarios, we focus here on estimating the state of a single agent given only a limited stream of observations from that agent. If the state can be determined reliably, then collaborative control policies can be defined using the estimated states [ 23 , 20 ] . The result is a scenario in which agents can be trained and updated separately, provided their control policies can be shared with other agents prior to a task. Contributions of this work include: 1) we implement an estimator for an agent’s state that uses only its control policy output as observations, 2) we improve observability of control policies using reinforcement learning, 3) we demonstrate through singular value decomposition analysis of the observability matrix and through Monte Carlo simulation that the resulting policies improve estimator performance with minimal impact on the nominal task performance. Figure 1 illustrates our test problem – a control policy must keep a fixed-wing aircraft near a target point by controlling the aircraft bank angle. A second agent tries to determine the first agent’s position (x_{relative},y_{relative}) and velocity (\dot{x}_{relative},\dot{y}_{relative}) relative to the target point using as an observation only the steering command selected by the first agent. Figure 1 : The example problem used to explore training observable control policies. The estimator must determine the agent’s state (position and velocity relative to the target) with observation only of the steering command, \phi . 2 Agent Definition Decision-making agents can broadly be described with a control policy – a mapping that associates the agent’s state (which can include both physical state and the internal information state of the agent) with a control action to be executed. The agent has the objective of steering the system from some state \mathbf{s}_{k} , at step k , to a future state or set of states \left\{\mathbf{s}_{objective}\right\} . This model can describe a wide variety of systems, from classical control systems to human operators [ 8 ] . Control policies can be constructed via direct mathematical analysis or data-driven approaches, and can accommodate both deterministic and probabilistic agents and control approaches. We specifically focus on probabilistic agents, which associate states with distributions over actions: \pi:\mathbf{s}_{k}\rightarrow p(\mathbf{a})~\forall~\mathbf{a}\in\mathcal{A}(\mathbf{s}_{k}) where \pi is the agent’s policy function, \mathbf{a} is an action, p(\mathbf{a}) is the probability density function associated with the actions, and \mathcal{A}(\mathbf{s}_{k}) is the set of all actions which are permissible in state \mathbf{s}_{k} . The agent samples an action \tilde{\mathbf{a}}\sim p(\mathbf{a}) which is executed during each decision interval. Thus, for continuous states and actions the control is a