Evaluating an Immersive Analytics Application at an Enterprise Business Intelligence Customer Conference
Matthew Brehmer † University of Waterloo, Waterloo ON Canada, Ginger Gloystein Tableau, Seattle WA USA, Bailiang Zhou Tableau, Seattle WA USA, Abby Gray Cedar AI, Seattle WA USA, Sruthi Pillai Tableau, Seattle WA USA, Ben Medina † Adobe, Seattle WA USA, Vidya Setlur Tableau, Palo Alto CA USA · 2025
Read Evaluating an Immersive Analytics Application at an Enterprise Business Intelligence Customer Conference by Matthew Brehmer † University of Waterloo, Waterloo ON Canada, Ginger Gloystein Tableau, Seattle WA USA, Bailiang Zhou Tableau, Seattle WA USA, Abby Gray Cedar AI, Seattle WA USA, Sruthi Pillai Tableau, Seattle WA USA, Ben Medina † Adobe, Seattle WA USA, Vidya Setlur Tableau, Palo Alto CA USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15152.txt.
Introduction
Matthew Brehmer and co-authors evaluate a headset data app live at a business conference in a 2025 paper. Immersive analytics promises new ways to explore data in 3D, but testing such tools with real enterprise users is rare. A customer conference offered a crowd of working analysts in one place.
Twenty-two participants tried Tableau for visionOS, a headset version of the popular dashboard tool. Feedback split between delight at the novel experience and doubts about fitting it into daily analysis work. Usability findings covered the 3D interface and its gestures. The authors argue that judging such tools needs fresh methods mixing numbers with interviews, since novelty warps first impressions. The paper shares an enterprise view on evaluating immersive tools.
Catalog record
| CREATOR | Matthew Brehmer † University of Waterloo, Waterloo ON Canada, Ginger Gloystein Tableau, Seattle WA USA, Bailiang Zhou Tableau, Seattle WA USA, Abby Gray Cedar AI, Seattle WA USA, Sruthi Pillai Tableau, Seattle WA USA, Ben Medina † Adobe, Seattle WA USA, Vidya Setlur Tableau, Palo Alto CA USA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15152
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating an Immersive Analytics Application at an Enterprise Business Intelligence Customer Conference Matthew Brehmer † University of Waterloo, Waterloo ON Canada, Ginger Gloystein Tableau, Seattle WA USA, Bailiang Zhou Tableau, Seattle WA USA, Abby Gray Cedar AI, Seattle WA USA, Sruthi Pillai Tableau, Seattle WA USA, Ben Medina † Adobe, Seattle WA USA, Vidya Setlur Tableau, Palo Alto CA USA Abstract We reflect on an evaluation of an immersive analytics application (Tableau for visionOS) conducted at a large enterprise business intelligence (BI) conference. Conducting a study in such a context offered an opportunistic setting to gather diverse feedback. However, this setting also highlighted the challenge of evaluating usability while also assessing potential utility, as feedback straddled between the novelty of the experience and the practicality of the application in participants’ analytical workflows. This formative evaluation with 22 participants allowed us to gather insights with respect to the usability of Tableau for visionOS, along with broader perspectives on the potential for head-mounted displays (HMDs) to promote new ways to engage with BI data. Our experience suggests a need for new evaluation considerations that integrate qualitative and quantitative measures and account for unique interaction patterns with 3D representations and interfaces accessible via an HMD. Overall, we contribute an enterprise perspective on evaluation methodologies for immersive analytics. \onlineid 1004 \vgtccategory Research \vgtcpapertype evaluation keywords Immersive analytics, Business intelligence, evaluation, Apple Vision Pro. 1 Introduction The emergence of head-mounted displays (HMDs) has transformed the landscape of interactive experiences. Devices such as Apple’s Vision Pro [ 1 ] , HTC’s Vive [ 7 ] , Meta’s Quest [ 13 ] , and Microsoft’s HoloLens [ 14 ] have moved beyond niche gaming and entertainment platforms to become promising platforms for knowledge work, as demonstrated by the growing interest in immersive analytics [ 19 ] . They have the potential to manipulate virtual data objects in novel ways, thereby resulting in a deeper level of immersion [ 4 ] . The potential of immersive analytics elicits two major evaluation objectives. First, we must better understand how to design applications that are both engaging and useful , particularly in the domain of enterprise BI, where until recently there have been few commercial application offerings for HMDs and a general unfamiliarity with these devices. Second, we need to identify best practices for maintaining information clarity and usability given the dynamic conditions typical of immersive environments. The primary contribution of our work is a reflection on a recent evaluation of an HMD-based immersive analytics experience for the BI domain. At a large enterprise business intelligence conference, we conducted an opportunistic formative evaluation of Tableau for visionOS, a video-see-through (VST) augmented reality (AR) BI application built for the Apple Vision Pro. Based on observations and conversations with 22 participants, we arrived at insights relating to the novelty, usability, and potential utility of our application. While participants found the interaction paradigms in our immersive analytics experience to be novel, there was a notable dichotomy in how expert BI practitioners interacted with Tableau for visionOS relative to novices; the former adapted quickly to an immersive analytics workflow, while novices struggled and required additional guidance. In reflecting on our observations, we identified a tension between usability and the elicitation of use cases for immersive BI tasks, highlighting the challenges inherent in technology-driven innovation and understanding its ecological validity. This tension is particularly pronounced where novel interaction modalities such as gaze and gesture-based controls could be both exciting and perplexing for users. Our application introduced interfaces that often do not align with users’ existing mental models or analytics workflows, suggesting a possible learning curve that can impact application learnability and usability. We further acknowledge a tension between participants’ excitement with respect to a novel technology medium and their ability to articulate or envision the applicability of this technology in a real BI context, particularly given the difference between their work environment and our study setting. We distill our reflection into a set of recommendations for confronting and mitigating these tensions in future evaluations of immersive analytics applications and experiences that go beyond considerations of usability and performance on well-defined tasks. 2 Related Work Evaluations that consider both usability and utility are crucial for understanding the full potential of immersive analytics. Friedl-Knirsch et al.’s recent systematic literature review of user evaluation in immersive analytics summarizes the methodologies used in prior work and identifies underrepresented study designs [ 6 ] . Their survey considers a wide range of evaluation goals, from prototype evaluation to foundational research, and highlights the need for a shared evaluation framework to further research in immersive analytics. This dual focus on usability and utility is essential because it ensures that the developed technologies are not only user-friendly but also practically beneficial in real-world scenarios. This echoes a grand challenge for immersive analytics identified by Ens et al. [ 4 ] calling for a balanced consideration of usability and utility when evaluating applications aiming to support complex data analysis tasks, rigorously testing novel user interface design and interaction techniques while also reflecting on the practical applications of these designs and techniques. Similarly, Billinghurst et al. [ 3 ] evaluate the educational benefits of immersive analytics applications, demonstrating that while these tools can significantly enhance learning experiences through immersive interactions, their utility depends on the alignment with educational objectives and curriculum needs. Our work adds to this research with a reflection on usability and the elicitation of use cases following a formative evaluation of our immersive analytics application with participants recruited opportunistically at a large enterprise BI conference. Our participant pool and their domain expertise are notable, as Friedl-Knirsch et al [ 6 ] found that the most prevalent source of participants in evaluations of immersive analytics experiences are students and staff at post-secondary institutions. While BI professionals have been the focus demographic of prior formative studies (e.g., [ 10 , 21 ] ), these studies either predate the emergence of immersive analytics or focus on issues relating to desktop-based dashboard applications. 3 A Formative Evaluation of Tableau for visionOS Figure 1: A person wearing an Apple Vision Pro interacts with Tableau for visionOS. In this instance, a 3D globe and a bar chart display global CO_{2} emissions data, while a related dashboard appears in a separate web browser window. Tableau for visionOS. The design of Tableau for visionOS as a video-see-through (VST) augmented reality (AR) application followed the general interface design guidelines recommended for Apple’s Vision Pro [ 2 ] . However, as these guidelines are not specific to interacting with data, we also incorporated best practices for the design of interactive visual analytics applications [ 5 , 12 ] , such as displaying different visual representations for combinations of data field types and allowing for interactive exploration by applying filters, changing aggregation levels, and switching between alternate representations. Tableau for visionOS leveraged the Vision Pro’s spatial computing capabilities for visualizing data in space in proximity to contextually relevant interface elements. Tableau for visionOS featured interactive 2D bar and line charts, as well as 3D globe models for displaying geospatial data ( Figure 1 ). Tableau for visionOS incorporated eye tracking for dwell-based selection as well as hand gestures such as pinching, swiping,