ChatAR: Conversation Support using Large Language Model and Augmented Reality
Yuichiro Fujimoto Ryukoku University · 2025
Read ChatAR: Conversation Support using Large Language Model and Augmented Reality by Yuichiro Fujimoto Ryukoku University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16008.txt.
Introduction
Yuichiro Fujimoto, writing alone in 2025, builds a headset system that whispers helpful facts into a conversation as it happens. ChatAR pairs an augmented reality display with a large language model. When the chat hits an unfamiliar keyword, the model drafts a short note, formats it, and floats it before the wearer's eyes. The aim is to close knowledge gaps that stall talk between people.
A catch: reading floating text moves the eyes, and partners notice. Fujimoto designed the display layout to keep eye movements natural. In one experiment, partners were far less likely to spot the reading. In a second, the wearer's share of speech balanced out and partners rated the talk more exciting. The work treats conversation itself as the interface to support, quietly.
Catalog record
| CREATOR | Yuichiro Fujimoto Ryukoku University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-16008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ChatAR: Conversation Support using Large Language Model and Augmented Reality Yuichiro Fujimoto Ryukoku University Abstract Engaging in smooth conversations with others is a crucial social skill. However, differences in knowledge between conversation participants can sometimes hinder effective communication. To tackle this issue, this study proposes a real-time support system that integrates head-mounted display (HMD)-based augmented reality (AR) technology with large language models (LLMs). This system facilitates conversation by recognizing keywords during dialogue, generating relevant information using the LLM, reformatting it, and presenting it to the user via the HMD. A significant issue with this system is that the user’s eye movements may reveal to the conversation partner that they are reading the displayed text. This study also proposes a method for presenting information that takes into account appropriate eye movements during conversation. Two experiments were conducted to evaluate the effectiveness of the proposed system. The first experiment revealed that the proposed information presentation method reduces the likelihood of the conversation partner noticing that the user is reading the displayed text. The second experiment demonstrated that the proposed method led to a more balanced speech ratio between the user and the conversation partner, as well as a increase in the perceived excitement of the conversation. \onlineid 0 \vgtccategory Research \teaser ChatAR – Conversation Support Using a Large Language Model and Augmented Reality. ChatAR automatically detects keywords from conversational speech and generates relevant information in real time. This information is displayed on the HMD in a way that minimizes disruption to the user’s eye movements while ensuring effective communication. keywords Augmented reality, conversation support, large language model, user interface 1 Introduction Face-to-face conversation forms the foundation of social life. Through conversations with others, we not only share information but also build relationships and contribute to the smooth functioning of society. In particular, casual conversations without a specific purpose often play a crucial role in building relationships [ 9 ] . This is attributed to human characteristics, such as the tendency to build trusting relationships and rapport more easily by sharing personal information and understanding others’ values [ 18 ] . However, many individuals feel uncomfortable or even fearful when engaging in casual conversations with others. It is estimated that over one million people in Japan experience social withdrawal due to this issue [ 7 ] . Differences in the ability of both parties to engage in face-to-face conversations, gaps in background knowledge, and a lack of consideration for the other’s perspective can hinder the maintenance of smooth conversations [ 9 , 17 ] . This study focuses on addressing the issue of knowledge gaps between two individuals. In conversations, it is essential to comprehend the other person’s speech, integrate it with one’s own knowledge, and respond appropriately. If the response evokes empathy or interest in the other person, the conversation is likely to progress smoothly and positively impact the relationship. On the other hand, if the responses are consistently lacking in substance, the conversation partner may develop a somewhat negative impression. For example, consider a scenario where John and Adam, who have just met at a buffet party, are engaged in conversation. Suppose John initiates a conversation with Adam by saying, “My hobby is watching movies.” If Adam responds by saying, “Oh, I don’t watch movies, so I don’t know anything about them,” the conversation is likely to end there. John, having disclosed personal information (i.e., his hobby), may feel somewhat disappointed and find it challenging to continue the conversation with Adam. Even without any knowledge of movies, there are several ways to expand the conversation; however, this requires conversational skills. Moreover, having broader general knowledge is often more advantageous than having limited knowledge. Conventional general and medical approaches to addressing this issue typically involve conversation practice or social skills training, both of which require a long time to achieve improvement. To address this issue, this study proposes “chatAR,” a chat support system that provides continuous intervention during face-to-face conversations by leveraging augmented reality (AR) and large language model (LLM) technologies, which have advanced remarkably in recent years. LLM-based dialogue technology, including OpenAI’s chatGPT [ 23 ] and Google’s Gemini [ 11 ] , has achieved smooth conversations between people and systems at a truly practical level. Regarding the combined use of AR and (multimodal) LLMs, research on authoring tool applications has recently emerged (e.g., [ 3 , 30 ] ). However, their application in the context of conversation support remains limited. By recognizing the topic of a person-to-person conversation and inputting the extracted keywords into LLM-based dialogue technology, it becomes possible to generate relevant information and appropriate responses. Furthermore, by integrating AR technology with a head-mounted display (HMD), the generated responses can be presented at an appropriate position within the user’s field of view. As of 2025, wearing an AR-HMD on a daily basis is not yet common. However, devices such as the Xreal One [ 35 ] and Meta Orion [ 19 ] , which offer improved comfort for everyday use, are beginning to emerge. The proposed system is designed with the assumption that, in the near future, lighter AR-HMDs will replace smartphones and be used continuously in daily life, including during face-to-face conversations. On the other hand, the use of such a system may raise concerns that LLM will significantly intervene in the content of conversations, which could potentially change the meaning of human conversations and have a negative impact on users. Therefore, this study takes these concerns into consideration and limits LLM intervention to a minimum, by occasionally providing users with conversation options. Another important consideration when using the proposed method is that “the system must not hinder smooth conversation.” For example, if the text generated by ChatGPT is simply displayed horizontally in front of the user via the HMD, the user would need to move their eyes repeatedly from side to side to read it, which could be disruptive. The unnatural movement of the user’s eyes may immediately make the conversational partner aware that the user is referring to something. While this is not necessarily undesirable, behaviors that clearly indicate a lack of focus on the conversation should be avoided [ 6 ] . Similarly, if the system requires user operation, it is preferable that the conversation partner remains unaware of this interaction. This study also proposes an information presentation method designed to make it difficult for the conversational partner to detect when the user is checking displayed information, which is incorporated into the LLM-AR conversation support system. Furthermore, two experiments were conducted to evaluate both the effectiveness of the proposed information presentation method and the overall conversation support system. To summarize, the contributions of this study are as follows: • This study proposes a conversation support framework that fully integrates LLM and AR technologies, along with a prototype implementation. This framework has the potential to be utilized in everyday conversations in the near future. • An information presentation method is introduced that minimizes the likelihood of the conversational partner noticing when the user is referencing displayed information within the conversation support system. • Two experiments demonstrated that the proposed system enhances the subjective excitement of conversations while reducing the conversational partner’s awareness of the user’s information