Policy-to-Language: Train LLMs to Explain Decisions with Flow-Matching Generated Rewards
Xinyi Yang, Liang Zeng, Heng Dong, Chao Yu, Xiaoran Wu, Huazhong Yang, Yu Wang, Milind Tambe, Tonghan Wang · 2025
Read Policy-to-Language: Train LLMs to Explain Decisions with Flow-Matching Generated Rewards by Xinyi Yang, Liang Zeng, Heng Dong, Chao Yu, Xiaoran Wu, Huazhong Yang, Yu Wang, Milind Tambe, Tonghan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12530.txt.
Introduction
Xinyi Yang and co-authors taught language models to explain agent decisions in two thousand twenty-five. Agents increasingly act beside people in shared spaces. Their choices need plain-language reasons to be trusted. The framework trains explainers with AI feedback rewards. A flow model generates the reward scores. It captures how human judgments vary and blur.
The flow attends to decision context and wording cues. Theory bounds its drift from true human rewards. Training used noisy proxy scores from other models. Human and model judges rated the explanations highly. Readers could act on the explained decisions better. The approach scales explanation training without armies of raters. Trust in deployed agents is the end goal.
Catalog record
| CREATOR | Xinyi Yang, Liang Zeng, Heng Dong, Chao Yu, Xiaoran Wu, Huazhong Yang, Yu Wang, Milind Tambe, Tonghan Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12530
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Policy-to-Language: Train LLMs to Explain Decisions with Flow-Matching Generated Rewards Xinyi Yang, Liang Zeng, Heng Dong, Chao Yu, Xiaoran Wu, Huazhong Yang, Yu Wang, Milind Tambe, Tonghan Wang Abstract As humans increasingly share environments with diverse agents powered by RL, LLMs, and beyond, the ability to explain their policies in natural language will be vital for reliable coexistence. In this paper, we build a model-agnostic explanation generator based on an LLM. The technical novelty is that the rewards for training this LLM are generated by a generative flow matching model. This model has a specially designed structure with a hidden layer merged with an LLM to harness the linguistic cues of explanations into generating appropriate rewards. Experiments on both RL and LLM tasks demonstrate that our method can generate dense and effective rewards while saving on expensive human feedback; it thus enables effective explanations and even improves the accuracy of the decisions in original tasks. 1 Introduction Intelligent agents, ranging from reinforcement learning (RL) agents ( Kiran et al. 2021 ; Zhao et al. 2021 ; Liu & Zhu 2024 ; Qiu et al. 2024 ) , large language models (LLMs) ( Yao et al. 2022 ; Shinn et al. 2024 ; Wang et al. 2023a ; Kang et al. 2020 ) , to robotic systems ( Ismail et al. 2018 ) , are becoming increasingly intertwined with daily lives ( Wang et al. 2024b ) . To foster a more transparent, safe, and aligned agent ecosystem, a promising avenue is communicating the reasoning behind actions or decisions generated by agent policies in natural language ( Lazaridou et al. 2016 ) . Such explanations would allow humans to understand the rationale behind specific decisions, offer meaningful feedback, and make corrections when necessary, ultimately fostering trust and enhancing the reliability of the intelligent ecosystem ( Cambria et al. 2023 ) . However, these agents vary significantly in their policies, including decision-making algorithms and internal structures ( Bulling 2014 ; Wang et al. 2024c ) . Model-specific explainable AI methods inevitably lag behind due to the per-task engineering efforts required to adapt from one agent to another ( Rai 2020 ) . This necessitates a model-agnostic explanation generator for agent policies capable of inferring the reasoning behind decisions made by these policies given the context in which these decisions occur (e.g., in an RL task, a decision is an RL action, and the context could be an MDP state). In this paper, we explore the feasibility of building such a policy-to-language model based on LLMs, leveraging their ability of capturing and manipulating intricate contextual and linguistic cues to provide interpretable and persuasive explanations for agent decisions. Given agent decisions and their context, it is not difficult for an LLM to generate plausible explanations by drawing on its world knowledge, but such explanations may lack effectiveness due to issues like model capacity or hallucination ( Ji et al. 2023 ; Ganguli et al. 2022 ) . To ensure trustworthy communication, our aim is for explanations to reflect the agent’s reasoning—verified via whether a third party can accurately infer the agent’s decisions. That is, we propose to train an Explanation LLM to generate reasoning solely based on the context, with rewards based on whether a third party can reconstruct agent decisions from explanations. Notably, we avoid providing actual decisions to the Explanation LLM , preventing direct disclosure and forcing the model to derive effective explanations by the analyzing the context. Figure 1: An overview of our method. (Left) We prompt an Explanation LLM to generate reasoning about an agent decision based on the context information. Our focus is on whether a third party can infer the actual decision from this explanation. (Middle) We employ a rectified flow model \varphi to generate a probability distribution \hat{p} over possible decisions, according to how likely they appear as a plausible outcome after each sentence of the explanation. Per-sentence rewards for training the Explanation LLM are the changes in the probability of the actual decision (highlighted in blue). (Right) The architecture and training of the rectified flow \varphi are based on a Guidance LLM . The Guidance LLM provides positive samples, where, with the context and explanation as input, it can produce a distribution p that assigns the highest probability to the actual decision. The rectified flow \varphi is trained to produce such distributions p , with a cross-attention layer in its middle that selectively leverages information from the Guidance LLM input, enabling generalization to negative samples. As for the third party that provides rewards, human feedback is accurate, but can be expensive to scale across tasks with varying requirements ( Liu et al. 2024b ; Lambert et al. 2024 ) . An alternative is to use a Guidance LLM as a surrogate feedback provider. However, as shown by previous work ( Yang et al. 2024b ) and confirmed in our experiments (ablation \mathtt{Ours} \mathtt{w/o} \mathtt{Flow} in Tab. 5 ), such feedback can be noisy and inefficient. To solve this, we propose using generative models ( Ho et al. 2020 ; Song et al. 2020 ; Liu et al. 2022 ) to generate feedback, given their proven ability in generating complex distributions from simple distributions. The architecture and training of this generative model are based on the Guidance LLM , and ultimately provides the desired third party rewards to the Explanation LLM . Concretely, we feed an explanation to the Guidance LLM sentence by sentence, and evaluate how likely the Guidance LLM is to consider each possible decision as a plausible outcome after each sentence addition. This effectively forms a distribution over all possible decisions. Some distributions are positive , assigning the highest probability to the actual decision. We adopt rectified flow ( Liu et al. 2022 ) for generative modeling and train it on these positive samples to reconstruct them from Gaussian noise distributions. Rectified flow is an efficient flow matching ( Chen et al. 2018 ; Lipman et al. 2023 ) method, characterized by robust and fast inference, requiring only a few steps to solve the associated ordinary differential equation and generate high-dimensional distributions ( Song & Ermon 2019 ) . Our goal is that, having learned from positive samples, the rectified flow model can generalize to some negative samples where the Guidance LLM makes errors. To facilitate this, we design a specialized rectified flow network architecture that is able to exploit linguistic and contextual cues of explanations and context. Specifically, we learn to transform the inputs of the rectified flow model to flow tokens on which we apply a cross-attention layer to selectively incorporate information from the hidden states of the Guidance LLM ’s last layer. Inputs to the Guidance LLM include decisions, their context, and explanations. The output of the cross-attention layer is processed by a fully connected network to produce the output of the rectified flow model. Compared to alternative methods, such as directly fine-tuning the Guidance LLM , our method is more lightweight while effectively improving the reliability of the feedback. We evaluate our method on both RL ( \mathtt{SMAC} , Samvelyan et al. 2019 ) and LLM tasks ( \mathtt{MMLU} by Hendrycks et al. 2020 , \mathtt{MathQA} by Amini et al.