Embodied Explainability and Ontological Obstacles CCS: Human-centered computing Interaction design theory, concepts and paradigms CCS: Human-centered computing Collaborative interaction CCS: Human-centered computing HCI theory, concepts and models CCS: Computing methodologies Philosophical/theoretical foundations of artificial intelligence

Marvin Pafla, Jesse Hoey, Kate Larson, Mark Hancock · 2026

Read Embodied Explainability and Ontological Obstacles CCS: Human-centered computing Interaction design theory, concepts and paradigms CCS: Human-centered computing Collaborative interaction CCS: Human-centered computing HCI theory, concepts and models CCS: Computing methodologies Philosophical/theoretical foundations of artificial intelligence by Marvin Pafla, Jesse Hoey, Kate Larson, Mark Hancock (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23840.txt.

Introduction

Pafla, Hoey, Larson, and Hancock argue that AI explainability has been framed the wrong way, in a 2026 argument paper on human computer interaction. Standard practice treats explanation as something extracted from a model's internals and displayed to users. The authors, drawing on Dourish and enactivist theories of cognition, contend that understanding is instead created in use, as people act on what a system lets them probe and change.

Looking inside large language models, they say, runs into ontological obstacles. Surrogate models import outside abstractions that users can mistake for the model's own reasons. Many products called explanations are therefore misnamed, which can deepen overreliance on wrong outputs. The authors reserve explainability for designs offering real affordances to probe and repair behavior.

Catalog record

CREATORMarvin Pafla, Jesse Hoey, Kate Larson, Mark Hancock
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.23840

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Embodied Explainability and Ontological Obstacles CCS: Human-centered computing Interaction design theory, concepts and paradigms CCS: Human-centered computing Collaborative interaction CCS: Human-centered computing HCI theory, concepts and models CCS: Computing methodologies Philosophical/theoretical foundations of artificial intelligence Marvin Pafla, Jesse Hoey, Kate Larson, Mark Hancock Abstract . Explainability is often framed as a property of an AI model, with explanations extracted from its internals and shown to users. In this argument paper, we instead provide an embodied account of explainability based on Dourish and enactivist cognition: understanding is created in use as people act on affordances in shared practice. Using demonstrations and conceptual analysis, we reveal ontological obstacles when “looking inside” large language models: surrogates import external abstractions that can be mistaken for the model’s, and focusing on internal reasoning misses that explainers participate in their own understanding. We discuss these obstacles in XAI practice, arguing that many explanations are misnamed, which skews their purpose and can increase overreliance. Finally, we highlight how embodied explanations reorganize sense-making by making what matters publicly available for action, and argue that explainability claims should be reserved for designs that provide affordances to probe, coordinate, and repair behaviour in situated practice. Why We Struggle to Explain the Answers of Large Language Models (LLMs) 1. Introduction Large language models (LLMs) increasingly mediate consequential work, and with that comes a renewed demand: they should be explainable . Across HCI and XAI, this demand is addressed in a variety of ways—for example by attaching post-hoc artifacts (e.g., feature attributions, natural-language rationales), building simplified surrogates, or “looking inside” via mechanistic interpretability. Despite their differences, these approaches share a common hope: that exposing the model’s relevant “reasons” will help people decide when to trust, contest, or override its outputs. This view of explainability is increasingly strained in active practice. Empirical studies show that XAI artifacts (e.g., saliency maps) can be persuasive without being helpful, and can even increase overreliance when they accompany incorrect outputs ( 64 ; 10 ; 36 ; 86 ; 96 ; 85 ; 52 ; 81 ) . In many chat-style LLM deployments, fluent rationales can mimic understanding while leaving users without reliable ways to probe what would change an output, diagnose failures, or coordinate responsibility ( 85 ; 64 ; 52 ) . This finding creates a practical dilemma: demand for explainability rises, but these artifacts often fail to provide the stable affordances needed for intervention and repair in situated activity ( 37 ; 47 ; 102 ; 79 ) . The stakes extend beyond usability: regulation and liability demand lay-understandable explanations and freedom from defects, yet many existing approaches often reveal defects without guaranteeing their absence and leave it unclear when an explanation is “good enough” ( 37 ; 46 ) . We argue that this mismatch is not only technical but conceptual. Many XAI methods implicitly treat the explanandum as fixed in advance, assuming that understanding follows from exposing the “right” internal structure ( 28 ; 100 ) . This framing inherits a familiar internalist (“Cartesian”) standpoint in which cognition is a matter of internal representations of a mind-independent world ( 100 ; 39 ) . It also aligns with a code model and theory-of-mind framing of communication: meaning is treated as private content to be encoded and decoded by inferring what is “in the head” of the agent ( 95 ) . But lived activity rarely fits this picture. What counts as relevant is not simply “there” to be read off and mapped to an internal rationale; it is grounded in practice through roles, tools, timing, and shared routines, and can shift abruptly under breakdown or changing stakes ( 30 ; 63 ; 27 ) . We therefore take embodied explainability as our starting point: a system is explainable insofar as it offers affordances people can use to coordinate, check, and repair what is going on ( 29 ; 30 ) . Such affordances are not fixed properties of an interface; they come into view through what people can actually do with it. In short, understanding is not merely delivered from the inside out; explainers participate in it . On this account, we run into a familiar tension: the very scale that makes LLMs powerful also makes them difficult to make understandable in use. Their performance comes from high-dimensional, distributed structure, while explainability in practice depends on a small set of stable factors people can manipulate, check, and share. Many XAI approaches—including mechanistic interpretability—respond by building surrogates : they compress the model into legible objects (saliency, rationales, features, circuits, graphs) that can be made intelligible. These surrogates can be valuable for analysis and debugging, but they introduce what we call ontological obstacles when treated as explanations for situated use: to be usable, one must impose abstractions that can look like “the model’s concepts” but are often supplied by the explainer. Separately, the fixation on “looking inside” can miss where the action is ( 29 ) —the embodied, social work through which people coordinate, test, and repair understanding at the point of use. If “looking inside” yields partial, abstracted structure that rarely functions as an everyday interactional resource, then explainability may not be settled by internal reasons alone. We therefore shift from internal reasons to how explanations become meaningful in public interaction, drawing on a pragmatic account of communication in which agents make intentions evident and others interpret that evidence against what is relevant in context ( 95 ; 49 ) . This framing encourages building systems that can join the public organization of the activity—and keeping that activity alive—rather than scaling context and adding explanations after the fact. Separate from HCI’s calls for accountability and transparency, we thus make an ontological point that connects understanding with the existence of interactional phenomena like conversations or queues. We develop the argument through conceptual analysis and demonstration-based reasoning: short scenarios that show what an explanation must accomplish to be usable in practice. In Section 2 , the hammer ( Section 2.1 ) motivates an embodied view of objects and understanding. In Section 3 , the clinic contrast ( Section 3 ) highlights the importance of embodiment in explainability. In Section 4 , we analyze “looking inside” LLMs as surrogate-building and argue that it runs into ontological obstacles. In Section 5 , we situate LLM explainability within XAI and synthesize empirical findings. In Section 6 , the queue breakdown ( Section 6 ) shifts the focus from internal reasons to public repair of shared sense-making. Finally, Section 7 frames explainability as a commitment to situated participation and accountable coupling, not an extra output channel layered onto general-purpose chat. This paper makes four contributions. First, we offer a unified account of embodied explainability that links affordances, the constitution of objects and context, and public meaning. Second, we reveal ontological obstacles when “explaining” LLMs via mechanistic interpretability and a focus on the model’s internals. Third, we frame the purpose of explanations in the larger context of XAI. Fourth, we explore public explanations : ostensive, norm-setting interventions that stabilize what is happening and what follows next by installing shared affordances for coordination and repair. 2. Embodied Cognition This section introduces the embodied-cognition lens underlying our account of explainability. We show

Read the complete text