Towards Interactive Multimodal Representation of ML Functions for Human Understanding of ML ISBN: 978-1-4503-XXXX-X/18/06 Conference: ; ; 2997 CCS: Human-centered computing Visualization techniques CCS: Computing methodologies Reinforcement learning CCS: Human-centered computing Haptic devices CCS: Human-centered computing Virtual reality
Bokang Wang, Yingxuan Liao, Leah Lee, Jack Wesson, Anlan Yang, Ruizi Wang, Yigang Wen · 2026
Read Towards Interactive Multimodal Representation of ML Functions for Human Understanding of ML ISBN: 978-1-4503-XXXX-X/18/06 Conference: ; ; 2997 CCS: Human-centered computing Visualization techniques CCS: Computing methodologies Reinforcement learning CCS: Human-centered computing Haptic devices CCS: Human-centered computing Virtual reality by Bokang Wang, Yingxuan Liao, Leah Lee, Jack Wesson, Anlan Yang, Ruizi Wang, Yigang Wen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00357.txt.
Introduction
Bokang Wang and 6 co-authors try to make machine learning less scary through hands-on displays. Public views on AI, they note, are clouded by confusion, at a time when daily life leans on these systems more and more. Their hope is that trying things hands-on can spark curiosity, and that curiosity can start a cycle of learning.
The team built three trial displays of machine learning functions, each using hand-picked, open data sets. The trials were built to expose what drawing people in looks like for newcomers, young people included, and people from fields far from tech. Instead of lectures, the work studies what helps newcomers stay curious about new tools. Its stated aim is to shift views away from fear of the unknown and toward calm trust.
Catalog record
| CREATOR | Bokang Wang, Yingxuan Liao, Leah Lee, Jack Wesson, Anlan Yang, Ruizi Wang, Yigang Wen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00357
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Interactive Multimodal Representation of ML Functions for Human Understanding of ML ISBN: 978-1-4503-XXXX-X/18/06 Conference: ; ; 2997 CCS: Human-centered computing Visualization techniques CCS: Computing methodologies Reinforcement learning CCS: Human-centered computing Haptic devices CCS: Human-centered computing Virtual reality Bokang Wang, Yingxuan Liao, Leah Lee, Jack Wesson, Anlan Yang, Ruizi Wang, Yigang Wen Abstract . Attitudes about artificial intelligence and machine learning are recent victims of endemic misunderstanding; given our increasing reliance on these technologies, the need for widespread understanding and confidence in their use is paramount ( 1 ) . To this end, our work seeks to increase understanding in these typically inaccessible topics through interactive visualizations ( 3 ) , thereby garnering curiosity in the hopes of kickstarting a cycle of understanding leading to further pursuit of knowledge. We hope this will cyclically shift global attitudes away from the intimidation of the unknown currently plaguing ML. This work explores best practices for supporting curiosity in new technologies, to inspire attitudinal paradigm-shifts. Over three, distinct visualizations of machine learning data, we created prototypes with carefully selected, highly-transparent datasets, to examine the success factors of engagement required for more informed attitudes on ML less dictated by the fear of the unknown. By employing interactive visualizations, we can captivate the interest of teenagers and individuals from diverse fields, encouraging them to explore the fascinating world of machine learning. Figure 1. K-Means clustering deconstructs user paintings to show clustering in 3D and VR K-Means clustering deconstructs user paintings to show clustering in 3D and VR 1. Introduction and Motivation This is a product development project to discover how 3D data visualization and interactions can be used to represent various features of machine learning algorithms in an engaging environment. Our research was motivated by the “black box” perceptions of ML in average audiences, and those who find it intimidating from lack of knowledge are our target audience. We developed three visualizations: 1. analyzing RGB datasets using proximity clustering algorithms on image’s pixel colors, 2. exploring novelty in interaction using haptic feedback to deconstruct songs, and 3. shaping reinforced learning experiences using relatable metaphors in narrative storytelling. Through our development, we discovered that engaging experiences foster a greater inclination to explore the “unknown”, building a cyclical relationship between the pursuit of learning and resultant engagement. 2. Isochromatic Deconstruction 2.1. Introduction In this prototype, we embarked on an endeavor to delve into the intricate depths of visual data by employing the K-means clustering algorithm to dissect RGB datasets. Our primary aim was to deconstruct paintings into isochromatic layers, unraveling the underlying color complexities that constitute these artistic masterpieces. Leveraging the RGB values as an XYZ dataset, we orchestrated the clustering of images into layers meticulously organized based on color proximity. This innovative approach transcends the conventional two-dimensional realm, propelling the visualization process into the captivating realms of a three-dimensional space. By seamlessly integrating Unity 3D as our development platform, we crafted an immersive environment where the boundaries between reality and digital art blur. The culmination of our efforts materialized in the form of a virtual space, brought to life through the Oculus Quest 2 Head Mounted Display and trackers. This amalgamation of technology allowed us to immerse ourselves and our audience in a multisensory journey, where every pixel of the painting pulsated with vibrancy and depth. Through this transformative experience, we not only visualized static images but also breathed life into them, fostering a profound connection between art and observer. 2.2. Result Numerous elements contributed to the success of our approach. Firstly, the visualization of direct and unequivocal data played a pivotal role. The inherent clarity of a 2D image transformed into a lucid pixel dataset, facilitating a seamless understanding of the underlying information and making a deep impression on the viewer’s mind. This clarity not only enhanced comprehension but also heightened the overall impact of the experience, ensuring that every detail of the artwork was vividly appreciated. Furthermore, the incorporation of customization features proved instrumental in fostering a sense of personal investment among participants. By empowering users to select their preferred images for analysis, we created a more intimate and relatable interaction, wherein individuals felt a deeper connection to the visualized data. This personalization not only enriched the experience but also instilled a sense of ownership, thereby enhancing engagement levels. Moreover, the integration of virtual reality (VR) technology served as a catalyst for immersion, propelling users into an alternate dimension where the boundaries between reality and digital art dissolved. The immersive nature of the VR variant provided users with an array of sensory stimuli, captivating their attention and imbuing the experience with a heightened sense of realism. Consequently, participants were not merely passive observers but active participants in a visually captivating journey, thereby elevating the overall impact and memorability of the experience. 3. Haptic Music Figure 2. Musical subcomponent analysis used the Fast Fourier transform algorithm to experience music haptically A human hand over the haptics device 3.1. Introduction Our approach involved intricately mapping the beats, notes, and accents of a song onto distinct sensations dispersed across the hand, thereby forging an unprecedented connection between auditory stimuli and tactile feedback. We want not only visualizing the music but also making the music touchable. This dynamic fusion of sensory modalities not only enabled us to craft mesmerizing visual metaphors on screen but also synchronized sound and haptics to heighten the visceral experience of ’feeling’ music. Firstly, our methodology entails employing the fast Fourier transform algorithm to meticulously dissect the beats, notes, and intensity of the music. This algorithm enables us to extract intricate details from the audio, facilitating a comprehensive understanding of its rhythmic and tonal nuances. Once we have obtained the essential beats, notes, and intensity data, our next step involves translating this information into a visually captivating representation on the screen. Through intuitive visualization techniques, users will be able to perceive the underlying musical structure and dynamics with clarity and precision. Furthermore, we aim to enhance the user experience by integrating the STRATOS Ultrahaptics ( 5 ) technology. Developed by Leap Motion, STRATOS Ultrahaptics harnesses the power of sonic vibration to generate tactile sensations without physical contact. By synchronizing with Leap Motion’s hand-tracking capabilities, this innovative device can imbue users with haptic feedback, enabling them to feel the music in a profoundly immersive manner. 3.2. Steps In our approach to handling the notes within the music, we adopt a systematic mapping strategy to ensure efficient and intuitive interaction. Given that there are 12 distinct types of notes in total, we allocate each note to specific fingers, enabling users to navigate through the musical landscape with ease and precision. Thumb: This versatile digit is assigned a range of notes, including C, C#, A, and A#. By utilizing the thumb, users can effortlessly access these fundamental notes, laying the foundation for their musical exploration. Index Finger: Positioned next in line, the index finger is entrusted with the notes D, D#, and B. Middle Finger: Singularly dedicated to the note E. Ring Finger: With the responsibility of representing the notes F and F#, the ring finger assumes a vital role in our mapping strategy. Little Finger: Completing our comprehensive mapping scheme, the little finger is designated to