TriEx: A Game-based Tri-View Framework for Explaining Internal Reasoning in Multi-Agent LLMs

Ziyi Wang, Chen Zhang, Wenjun Peng, Qi Wu, Xinyu Wang · 2026

Read TriEx: A Game-based Tri-View Framework for Explaining Internal Reasoning in Multi-Agent LLMs by Ziyi Wang, Chen Zhang, Wenjun Peng, Qi Wu, Xinyu Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20043.txt.

Introduction

Wang and four colleagues present TriEx, a tri-view framework for explaining internal reasoning in multi-agent language models. It aligns three artifacts: first-person self-reasoning tied to an action, second-person beliefs about opponents over time, and third-person oracle audits from environment signals. Explanations thus become checkable objects grounded in evidence, comparable across time and perspectives. In imperfect-information strategic games, TriEx exposes gaps between what agents say, believe, and do, showing explainability depends on interaction.

Catalog record

CREATORZiyi Wang, Chen Zhang, Wenjun Peng, Qi Wu, Xinyu Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE96701
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20043

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TriEx: A Game-based Tri-View Framework for Explaining Internal Reasoning in Multi-Agent LLMs Ziyi Wang, Chen Zhang, Wenjun Peng, Qi Wu, Xinyu Wang Abstract Explainability for Large Language Model (LLM) agents is especially challenging in interactive, partially observable settings, where decisions depend on evolving beliefs and other agents. We present TriEx , a tri-view explainability framework that instruments sequential decision making with aligned artifacts: (i) structured first-person self-reasoning bound to an action, (ii) explicit second-person belief states about opponents updated over time, and (iii) third-person oracle audits grounded in environment-derived reference signals. This design turns explanations from free-form narratives into evidence-anchored objects that can be compared and checked across time and perspectives. Using imperfect-information strategic games as a controlled testbed, we show that TriEx enables scalable analysis of explanation faithfulness, belief dynamics, and evaluator reliability, revealing systematic mismatches between what agents say, what they believe, and what they do. Our results highlight explainability as an interaction-dependent property and motivate multi-view, evidence-grounded evaluation for LLM agents. Code is available at https://github.com/Einsam1819/TriEx . 1 Introduction Large Language Models (LLMs) increasingly act as autonomous agents capable of multi-step planning, strategic interaction, and real-time decision-making ( 27 ; 13 ; 28 ) . Yet as these models are deployed in settings that involve uncertainty, hidden information, and interactions with other agents, understanding how they internally construct strategies and make decisions becomes both crucial and challenging ( 34 ) . Existing explainability methods for NLP, such as attribution or post-hoc natural-language rationales ( 23 ; 22 ; 30 ; 25 ) , have predominantly been developed for static, single-turn prediction tasks. While such techniques can offer insight into which input features influenced a particular output, they fall short of capturing interaction-dependent belief dynamics and multi-agent reasoning, and thus provide limited evidence about how models represent other agents, update beliefs over time, or translate internal states into strategic actions ( 6 ) . Recent empirical observations highlight the severity of this gap. For example, AlphaArena ( 1 ) places multiple LLMs inside a live cryptocurrency and stock trading environment to study their behavior under realistic financial pressures. Despite strong performance on standard reasoning benchmarks, state-of-the-art models exhibited unstable strategies in this interactive setting. Crucially, their natural-language explanations often diverged from the decisions they executed, suggesting post-hoc rationalization and limited faithfulness. These findings underscore that in complex multi-agent contexts, existing explanation techniques often fail to reveal the internal mechanisms that drive behavior. A central challenge these results point to is that we lack frameworks capable of exposing how LLMs model other agents, including their preferences, tendencies, and strategic profiles, and how such representations shape subsequent decisions. In human social interaction, individuals routinely construct and refine mental models of others over repeated encounters ( 2 ) . These models support predictions about behavior, enable coordination, and guide adaptive strategies ( 24 ) . Despite LLMs increasingly participating in multi-agent settings, it remains unclear whether they develop analogous structures, how such representations evolve through interaction, and how faithfully their natural-language explanations reflect these latent processes. We introduce the Tri-View Agent Explanation Framework (TriEx) , a methodology inspired by human social cognition for exposing complementary aspects of LLM internal decision-making. TriEx triangulates three trace-grounded perspectives: a first-person self-explanation tied to the executed action, a second-person opponent belief state updated through interaction, and a third-person oracle audit that checks cross-view consistency against objective reference signals and compares judgments across oracle models. By integrating these perspectives, TriEx provides a structured framework for examining how LLMs form and use internal representations during decision-making. Although poker serves as a testbed, the mechanisms TriEx targets, including belief updating, perspective integration, and cross-view consistency, are fundamental to many interactive, uncertain settings involving social reasoning. Strategic games thus serve as diagnostic environments rather than domain-specific applications. We view TriEx as an initial step toward systematic explainability for LLM agents in multi-agent settings. In summary, the contributions of this work are as follows: • We introduce TriEx , a tri-view explainability framework for interactive LLM agents that logs decision traces with structured self-explanations, explicit counterpart belief states, and trace-grounded third-person audits. • TriEx makes agent explainability testable by enabling cross-view checks of explanation faithfulness to actions, belief grounding and causal sensitivity under controlled interventions, and evaluator reliability across oracle models. • Using TriEx, we find three robust patterns: faithfulness degrades with decision complexity, counterpart beliefs are intervention-sensitive and shape decision behavior, and evaluator reliability is highest for categorical and rank-based judgments but unstable for absolute scores. 2 Related Work 2.1 Explainability of LLMs Explainability for LLMs spans methods from user-facing natural-language rationales to mechanism-level analyses of the computations underlying model behavior ( 34 ) . A common practice is to elicit self-explanations such as Chain of Thought (CoT) traces ( 30 ) and treat them as explanations of the decision process; however, substantial evidence shows that CoT rationales can be unfaithful, offering plausible narratives weakly tied to the features or computations responsible for the output ( 25 ; 14 ) . Prior work therefore proposes intervention- or self-consistency-based faithfulness evaluations, but these often emphasize output-level consistency and can be confounded by task accuracy ( 14 ; 19 ; 11 ; 3 ) . Beyond natural-language rationales, mechanistic interpretability explains LLM behavior through internal components and causal computations ( 26 ) . Techniques such as activation patching and causal tracing can localize computations underlying factual recall and support targeted model editing ( 17 ) . However, later analyses show that patching-based conclusions are sensitive to experimental design and may yield misleading artifacts, motivating stricter evaluation protocols ( 33 ; 15 ) . Recent work further advances mechanism-level explanations by identifying sparse, interpretable feature circuits and analyzing intermediate computation dynamics in in-context learning and failure cases ( 16 ; 9 ) . Despite these advances, much prior work focuses on static settings, leaving the evolution of explanations and internal beliefs under repeated interaction underexplored. We address this gap with TriEx, which instruments decision traces to jointly analyze actions, self-explanations, opponent beliefs, and third-person audits over time. 2.2 Games for LLM Evaluation Games and simulated environments have emerged as a complementary paradigm for evaluating LLM agents ( 31 ; 29 ; 10 ; 7 ; 20 ) . Most work, however, focuses on external performance or behavioral outcomes, offering limited insight into how models internally represent strategies, opponents, or uncertainty over time. By contrast, TriEx uses strategic games as controlled diagnostic environments where decision traces can be logged and audited against objective reference signals. 2.3 Social Cognition and Agent Modeling TriEx is inspired by findings in social cognition showing that humans form and refine mental models of others through repeated interaction ( 2 ; 8 ) . Related work on Theory of Mind argues that agents attribute latent mental states to predict others’ actions, and in strategic settings this can involve recursive reasoning about others’ policies ( 21 ; 4 ; 32 ) . At the same time, cognitive psychology highlights limits of introspective verbal reports, showing that coherent self explanations may be weakly connected to the actual determinants of behavior ( 18 ) . Together, these findings motivate TriEx’s separation of first-person self explanations, second-person opponent beliefs, and third-person audits, enabling

Read the complete text