Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models Conference: The 37th Annual ACM Symposium on User Interface Software and Technology; October 13–16, 2024; Pittsburgh, PA, USA The 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24),
Ada Yi Zhao, Aditya Gunturu, Ellen Yi-Luen Do, Ryo Suzuki · 2025
Read Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models Conference: The 37th Annual ACM Symposium on User Interface Software and Technology; October 13–16, 2024; Pittsburgh, PA, USA The 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24), by Ada Yi Zhao, Aditya Gunturu, Ellen Yi-Luen Do, Ryo Suzuki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03547.txt.
Description
Following written steps for a hands on task can be slow when the work is in front of you and the guide is on paper. Ada Yi Zhao and three co-authors present Guided Reality in 2025, a system that writes the steps and paints visual cues right onto the real scene through augmented reality glasses. Large language models draft the steps, and vision models find where each cue belongs.
The team sorted visual cues into five kinds drawn from user manuals, and the system picks a kind based on the step at hand. Sixteen users tried the system on real tasks and in open use. Four instructors also weighed in on how it could fit their training work.
Catalog record
| CREATOR | Ada Yi Zhao, Aditya Gunturu, Ellen Yi-Luen Do, Ryo Suzuki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03547