Enhancing Explainability with Multimodal Context Representations for Smarter Robots DOI: 10.5281/zenodo.14930029 Conference: 3rd Workshop on Explainability in Human-Robot Collaboration at HRI ’25; March 3, 2025; Melbourne, Australia The 3rd Workshop on Explainability in Human-Robot Collaboration at
Anargh Viswanath, Lokesh Veeramacheneni, Hendrik Buschmeier · 2025
Read Enhancing Explainability with Multimodal Context Representations for Smarter Robots DOI: 10.5281/zenodo.14930029 Conference: 3rd Workshop on Explainability in Human-Robot Collaboration at HRI ’25; March 3, 2025; Melbourne, Australia The 3rd Workshop on Explainability in Human-Robot Collaboration at by Anargh Viswanath, Lokesh Veeramacheneni, Hendrik Buschmeier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16467.txt.
Introduction
Anargh Viswanath, Lokesh Veeramacheneni, and Hendrik Buschmeier want robots that can explain themselves by truly taking in the scene around a conversation, in this 2025 workshop paper. A robot that hears speech but ignores vision, or the reverse, will misunderstand its human partner and lose trust. The challenge is fusing the streams into one usable context.
Their framework builds a joint representation of speech and vision, then aligns the two in time. The test case is relevance: judging whether what the user just said matches what the robot currently sees. A multimodal joint representation module merges the inputs, while a temporal alignment module lines up words with the visual moments they refer to. The authors close by mapping how such context representations support different flavors of explainability in human-robot collaboration.
Catalog record
| CREATOR | Anargh Viswanath, Lokesh Veeramacheneni, Hendrik Buschmeier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16467
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Explainability with Multimodal Context Representations for Smarter Robots DOI: 10.5281/zenodo.14930029 Conference: 3rd Workshop on Explainability in Human-Robot Collaboration at HRI ’25; March 3, 2025; Melbourne, Australia The 3rd Workshop on Explainability in Human-Robot Collaboration at HRI ’25 CCS: Computing methodologies Cognitive robotics CCS: Human-centered computing HCI theory, concepts and models Anargh Viswanath, Lokesh Veeramacheneni, Hendrik Buschmeier Abstract . Artificial Intelligence (AI) has significantly advanced in recent years, driving innovation across various fields, especially in robotics. Even though robots can perform complex tasks with increasing autonomy, challenges remain in ensuring explainability and user-centered design for effective interaction. A key issue in Human-Robot Interaction (HRI) is enabling robots to effectively perceive and reason over multimodal inputs, such as audio and vision, to foster trust and seamless collaboration. In this paper, we propose a generalized and explainable multimodal framework for context representation, designed to improve the fusion of speech and vision modalities. We introduce a use case on assessing ‘Relevance’ between verbal utterances from the user and visual scene perception of the robot. We present our methodology with a Multimodal Joint Representation module and a Temporal Alignment module, which can allow robots to evaluate relevance by temporally aligning multimodal inputs. Finally, we discuss how the proposed framework for context representation can help with various aspects of explainability in HRI. 1. Introduction Figure 1 . Illustrating ‘Relevance’ metric for evaluating when and how well verbal utterances from users align with the robot’s visual perceptions. Here the verbal utterance ‘Put the blue cup in the dishwasher’ is of high relevance to the visual scene at time t_{n+c} , while the visual frame at time t_{n} has no relevance due to the absence of relevant objects. Similarly, the third visual scene with ‘yellow cup’ at time t_{n+b} is less relevant for the verbal utterance compared to the second visual scene with ‘dishwasher’ . Illustration showing five boxes, each at subsequent points in time ($t_N$, $t_{n+a}$, $t_{n+b}$, $t_{n+c}$ and containing a different view into the scene, roughly time-aligned with an audio prompt. Each box shows a different degree of relevance according to the relevance metric. Artificial Intelligence (AI) has seen rapid advancements in recent years, leading to its widespread adoption across various domains. Among these, the application of AI in robotics has brought transformative changes, enabling robots to perform complex tasks with greater autonomy and efficiency. However, despite these advancements, there are still significant challenges associated with them, particularly in terms of explainability and user-centered design. A fundamental issue in the field of Human-Robot Interaction (HRI) is enabling robots to perceive and understand their environment through multimodal inputs, such as audio and visual data. To interact effectively with humans, robots must be able to integrate and reason over multimodal information, ensuring that speech (verbal commands from the user) and vision (robot perception) are processed in a coherent and context-aware manner. Combining these inputs requires a structured approach to avoid information loss and create reasoning capabilities for fostering trust and long-term collaboration. This becomes particularly crucial in human-centered spaces, where robots are expected to comprehend routines, execute commands seamlessly, and exhibit reasoning capabilities when encountering uncertainties. In this paper, we propose a generalized and explainable multimodal framework for context representation, designed to improve the fusion of speech and vision modalities in HRI-based applications. To demonstrate its effectiveness, we introduce a use case focused on assessing ‘Relevance’ , a key metric that evaluates when and how well verbal utterances from users align with the robot’s visual perception. To illustrate our use case, consider a service robot deployed in a household setting for tidying and rearranging things. For completing these tasks, the robot can navigate autonomously and is equipped with sensors for continuously perceiving its environment within its field of view. Additionally, it can also receive verbal commands or utterances from humans through audio sensors. Suppose that while the robot is carrying out its routine cleaning, it receives a verbal command from the user ‘Put the blue cup in the dishwasher’ as shown in Fig. 1 . The user probably assumes that the robot will complete it as an additional task. However, in the current field of view as well as the previous visual scenes, it has encountered a ‘dishwasher’ but not a ‘blue cup’ . This necessitates the need for temporal alignment between the multimodal inputs to assess the relevance. In this paper, we propose a novel methodology consisting of a Multimodal Joint Representation module along with a Temporal Alignment module, which quantifies the relevance factor by temporally aligning multimodal representations. This will allow the robot to plan actions strategically and enhance its ability to bridge the gap between language and vision. Lastly, we explore how the proposed multimodal framework can contribute to explainability through two key aspects, namely Useful Representation and Meaningful Abstraction . These factors play a critical role in improving transparency, interpretability, and user experience in HRI. This work is ongoing, and in this paper, we outline our methodology for multimodal context representation, with a particular focus on the ‘Relevance’ problem. 2. Combining Vision and Speech Humans possess an extraordinary ability to effortlessly ground verbal expressions in the physical world. We are seamlessly linking language to real-world objects. We intuitively merge speech and vision modalities, enabling us to interpret verbal utterances within the context of visual scenes and vice versa ( Mondada 2016 ) . This innate ability supports social interaction along with collaborative tasks between humans ( Mondada 2014 ) . However, achieving similar capabilities in robots remains a significant challenge in the field of HRI ( Allgeuer et al. 2024 ) . Developing intelligent robots capable of long-term, meaningful interaction with humans, especially robots designed for collaborative tasks (e.g., service robots), requires overcoming this gap. In human-centred spaces, such as homes or healthcare settings, robots need to interact in a way that feels natural and sociable. This is essential for long-term and effective collaboration to build trust. For this, robots must possess certain capabilities, such as context awareness, multimodal knowledge representation, resolving references to objects in the physical world (grounding), and modality fusion for coherent reasoning, among others ( Allgeuer et al. 2024 ) . 2.1. Traditional Representations Traditionally, robots have relied on representations such as spatial representations (e.g. metric maps, topological maps, semantic maps, etc.) ( Landsiedel et al. 2017 ) , hierarchical structures (e.g. associated with behavior trees, state machines, etc.) ( Ghzouli et al. 2023 ) , and graphical models (e.g. knowledge graphs, scene graphs, etc.) ( Wilcock and Jokinen 2022 ) to represent world information ( Paulius and Sun 2019 ) . These structures help robots plan and perform physical tasks by breaking down complex actions into simpler, manageable steps. These approaches have proven effective for tasks like navigation and object manipulation, offering advantages in logical inferencing, failure-detection, and error diagnosis. However, these representations suffer with significant limitations. Firstly, these representations are often heavily vision-reliant, making it difficult to incorporate rich multimodal data, such as audio-video, into their representation space. This results in separated/disconnected processing of speech and vision, leading to information loss and inadequate reasoning capabilities for tasks that demand social interaction. Secondly, they might struggle in out-of-distribution scenarios, where the addition of new elements, such as objects