What Questions Should Robots Be Able to Answer? A Dataset of User Questions for Explainable Robotics DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-center
Lennart Wachowiak, Andrew Coles, Gerard Canal, Oya Celiktutan · 2025
Read What Questions Should Robots Be Able to Answer? A Dataset of User Questions for Explainable Robotics DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-center by Lennart Wachowiak, Andrew Coles, Gerard Canal, Oya Celiktutan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16435.txt.
Introduction
Lennart Wachowiak, Andrew Coles, Gerard Canal, and Oya Celiktutan built a 2025 data set of questions people ask home robots. As chat tools spread, robots must answer users well, not just act. The team showed 100 people videos and texts of robots doing home tasks. Each person wrote what they would ask the robot in that scene.
The set holds 1,893 questions in 12 groups and 70 subgroups in all. Most ask about task steps, skills, or how well the robot did. Users rate rare but hard what-if questions as most vital to answer. New users ask more plain fact questions than skilled users do. Builders can use the set to plan what robots should log and say.
Catalog record
| CREATOR | Lennart Wachowiak, Andrew Coles, Gerard Canal, Oya Celiktutan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16435
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Questions Should Robots Be Able to Answer? A Dataset of User Questions for Explainable Robotics DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing Empirical studies in interaction design CCS: Human-centered computing Interaction design process and methods CCS: Computing methodologies Natural language processing CCS: Information systems Information retrieval CCS: Computer systems organization Robotics Lennart Wachowiak, Andrew Coles, Gerard Canal, Oya Celiktutan Abstract . With the growing use of large language models and conversational interfaces in human–robot interaction, robots’ ability to answer user questions is more important than ever. We therefore introduce a dataset of 1,893 user questions for household robots, collected from 100 participants and organized into 12 categories and 70 subcategories. Most work in explainable robotics focuses on why-questions, such as “Why did you clean the bathroom but not the kitchen?”. In contrast, our dataset provides a wide variety of questions, from questions about simple execution details to questions about how the robot would act in hypothetical scenarios — thus giving roboticists valuable insights into what questions their robot needs to be able to answer. To collect the dataset, we created 15 video stimuli and 7 text stimuli, depicting robots performing varied household tasks. We then asked participants on Prolific what questions they would want to ask the robot in each portrayed situation. In the final dataset, the most frequent categories are questions about task execution details (22.5%), the robot’s capabilities (12.7%), and performance assessments (11.3%). Although questions about how robots would handle potentially difficult scenarios and ensure correct behavior are less frequent, users rank them as the most important for robots to be able to answer. Moreover, we find that users who identify as novices in robotics ask different questions than more experienced users. Novices are more likely to inquire about simple facts, such as what the robot did or the current state of the environment. As robots enter environments shared with humans and language becomes central to giving instructions and interaction, this dataset provides a valuable foundation for (i) identifying the information robots need to log and expose to conversational interfaces, (ii) benchmarking question-answering modules, and (iii) designing explanation strategies that align with user expectations. 1. Introduction In recent years, robots have moved beyond executing narrow tasks in defined spaces. They are now starting to be deployed in complex and dynamic environments shared with humans. Examples include robots in healthcare and assisted living ( Trainum et al. 2023 ) , or in the service industry, for instance, in restaurants ( Gonzalez-Aguirre et al. 2021 ) . When inhabiting the same space as people and executing tasks for or with them, natural language communication becomes essential. Language now also plays a central role in instructing robots, as in the case of vision–language–action models (VLAs). VLAs are large vision–language models fine-tuned to output robot actions given a language instruction ( Team et al. 2025 ) . Widely publicized VLA demos often feature household tasks, such as cleaning or tidying up — further areas with ample human contact. When interacting with such a robot, users might naturally have many questions for the robot. For example, users may want to understand how the robot works and what its decision-making process looks like. A robot’s ability to explain itself in such situations fosters safer interactions and is known to help users calibrate their trust levels appropriately ( Wang et al. 2016b ) . Due to the rise in text generation capabilities ( Achiam et al. 2023 ; Comanici et al. 2025 ) , giving natural language answers to user questions is easier than ever. However, for the answers to be truthful, they must be based on the robot’s true capabilities, underlying decision-making, and execution traces. If the robot’s question-answering module lacks this data, a large language model (LLM) might simply confabulate the answers ( Huang et al. 2025 ) . Without having relevant logs and access to run explainability methods, the LLM’s answers can devolve into statistically likely language patterns unrelated to reality. Therefore, roboticists need to know what data should be accessible to such a question-answering module. With this paper, we provide a novel large-scale dataset of questions users have for household robots. Our dataset enables roboticists to see what questions their robot might need to answer. Second, these questions will help roboticists understand what type of data needs to be collected by the robot during task execution and made accessible to the natural language interface. Third, the dataset and its question categories can serve as a strong foundation, grounded in real user needs, for benchmarking XAI and question-answering systems. Beyond providing the dataset, our paper addresses the following research questions in an exploratory manner: • RQ1 : What type of user questions should service robots be able to answer? • RQ2 : What type of questions do users consider especially important? • RQ3 : Do the types of questions and the importance the user places on the robot’s question-answering capabilities vary with the user’s robot experience and attitudes towards robots? Our main findings and contributions can be summarized as follows: • We provide a dataset of 1,893 user questions. We have hierarchically categorized the questions into 12 main categories and 70 subcategories ( RQ1 ). The resulting categories show a wide variety of question types, many of which are underexplored in the current explainable robotics literature. Beyond the causal why-questions often explored in XAI, the categories include questions about execution details, the robot’s capabilities, how and if it can handle potential difficulties, technical details of the robot, the robot’s source of knowledge, and many more. • We find that users assign different levels of importance to different question categories ( RQ2 ). However, a unifying theme is that questions about safety, failure, and potential issues rank highly. Questions about how the robot would deal with hypothetical scenarios and how it ensures its actions are correct rank highest. In contrast, the why-questions popular in XAI literature have the second-lowest average importance score. • We find that users with more positive attitudes towards robots also think it is more important for robots to be able to answer their questions ( RQ3 ). Additionally, we find that the types of questions asked differ based on a person’s level of experience with robots. Novices used a larger allocation of their questions to ask about execution details and the state of the environment. In contrast, more experienced users were more likely to ask questions about how the robot would deal with potentially difficult situations or how it makes decisions. 2. Related Work 2.1. Explainable AI/Robotics Explainability has become a core topic in the robotics ( Anjomshoae et al. 2019 ; Setchi et al. 2020 ) , AI planning ( Fox et al. 2017 ) , and machine learning ( Adadi and Berrada 2018 ) literature. By providing the reasons for the robot’s actions and decisions, explanations are supposed to: • enable users to understand the robot’s decision-making and be better at predicting future behavior ( Wachowiak et al. 2023 ; Doshi-Velez and Kim 2017 ) , • allow users to calibrate their trust levels appropriately ( Fischer