On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks CCS: Human-centered computing Human computer interaction (HCI)

Ting Bi 1 , Chenghang Ye 2 , Zheyu Yang 2 , Ziyi Zhou 2 , Cui Tang 2 , Jun Zhang 1 , Zui Tao 1 , Kailong Wang 1 , Liting Zhou 3 , Yang Yang 2 , Tianlong Yu 1 1 Huazhong University of Science and Technology 2 Hubei University 3 Dublin City University · 2025

Read On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks CCS: Human-centered computing Human computer interaction (HCI) by Ting Bi 1 , Chenghang Ye 2 , Zheyu Yang 2 , Ziyi Zhou 2 , Cui Tang 2 , Jun Zhang 1 , Zui Tao 1 , Kailong Wang 1 , Liting Zhou 3 , Yang Yang 2 , Tianlong Yu 1 1 Huazhong University of Science and Technology 2 Hubei University 3 Dublin City University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13209.txt.

Introduction

Ting Bi and co-authors test if smart glasses plus a smart text tool can run a con in a 2025 paper. Glasses that paint the world can feed a text tool sight, sound, and place, a rich diet for a trickster.

Their frame runs three steps: it brews a fake scene from what the glasses catch, pulls facts that fit a set role, and then works the mark in loops of talk. In a cleared trial, sixty hands tried set scenes three ways. With the full frame, most hands fell for a mail con, and most took the trickster's call as a friend. Some found the play a bit stiff. The authors give the work as a warn to build guards for the glass age.

Catalog record

CREATORTing Bi 1 , Chenghang Ye 2 , Zheyu Yang 2 , Ziyi Zhou 2 , Cui Tang 2 , Jun Zhang 1 , Zui Tao 1 , Kailong Wang 1 , Liting Zhou 3 , Yang Yang 2 , Tianlong Yu 1 1 Huazhong University of Science and Technology 2 Hubei University 3 Dublin City University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13209

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Feasibility of Using MultiModal LLMs to Execute AR Social Engineering Attacks CCS: Human-centered computing Human computer interaction (HCI) Ting Bi 1 , Chenghang Ye 2 , Zheyu Yang 2 , Ziyi Zhou 2 , Cui Tang 2 , Jun Zhang 1 , Zui Tao 1 , Kailong Wang 1 , Liting Zhou 3 , Yang Yang 2 , Tianlong Yu 1 1 Huazhong University of Science and Technology 2 Hubei University 3 Dublin City University Abstract . Augmented Reality (AR) and Multimodal Large Language Models (LLMs) are rapidly evolving, providing unprecedented capabilities for human-computer interaction. However, their integration introduces a new attack surface for social engineering. In this paper, we systematically investigate the feasibility of orchestrating AR-driven Social Engineering attacks using Multimodal LLM for the first time, via our proposed SEAR framework, which operates through three key phases: (1) AR-based social context synthesis, which fuses Multimodal inputs (visual, auditory and environmental cues); (2) role-based Multimodal RAG (Retrieval-Augmented Generation), which dynamically retrieves and integrates contextual data while preserving character differentiation; and (3) ReInteract social engineering agents, which execute adaptive multiphase attack strategies through inference interaction loops. To verify SEAR, we conducted an IRB-approved study with 60 participants in three experimental configurations (unassisted, AR+LLM, and full SEAR pipeline) compiling a new dataset of 180 annotated conversations in simulated social scenarios. Our results show that SEAR is highly effective at eliciting high-risk behaviors (e.g., 93.3% of participants susceptible to email phishing). The framework was particularly effective in building trust, with 85% of targets willing to accept an attacker’s call after an interaction. Also, we identified notable limitations such as “occasionally artificial” due to perceived authenticity gaps. This work provides proof-of-concept for AR-LLM driven social engineering attacks and insights for developing defensive countermeasures against next-generation augmented reality threats. 1. Introduction The rapid development of Augmented Reality (AR) and Large Language Models (LLMs) is revolutionizing human-computer interaction, enabling immersive experiences that blend digital overlays with real-world environments. AR systems, equipped with Multimodal sensors like RGB-D cameras and microphones, capture rich contextual data (e.g., facial or vocal information), while LLMs analyze and generate human-like dialogue with remarkable adaptability. While this synergy enables transformative applications, it also introduces unprecedented risks: the integration of AR’s real-time environmental perception and LLMs’ adaptive reasoning creates a potent vector for next-generation social engineering attacks ( Choo 2025 ) . Traditional social engineering techniques, such as phishing emails or identity theft ( Ho et al. 2019 ; Bilge et al. 2009 ; Roy et al. 2024 ; Timko et al. 2025 ) , rely on static deception strategies ( Burda et al. 2024 ; Vadrevu and Perdisci 2019 ; Yang et al. 2023 ; Ulqinaku et al. 2021 ) . In contrast, the fusion of AR’s environmental perception and LLMs’ generative capabilities will introduce a potential paradigm shift- allowing the attackers to craft highly personalized and adaptive attacks. For instance, AR sensors can infer a victim’s emotional status during a conversation ( Xu et al. 2025 ) , while LLMs can generate strategical dialogue (e.g., gradual trust-building) to exploit the reduced vigilance. Despite the enthusiasm for AR-LLM social applications ( Yang et al. 2025 ; Jansen and Fischbach 2020 ; Fuste and Schmandt 2017 ; Hirskyj-Douglas et al. 2020 ) and the growing awareness of AR privacy risks ( Chen et al. 2018 ; Lehman et al. 2022 ; Deng et al. 2023 ) and LLM-enabled phishing ( Falade 2023a ) , no prior work systematically examines their potential for orchestrated Social Engineering (SE) attacks. This gap leaves critical questions unresolved: Can AR sensory data (sight or sound of the target) be weaponized to support physical SE interactions (e.g., private conversations)? Can Multimodal LLMs enable hyper-personalization and bypass human cognitive defenses? How do LLM-supported adaptive SE strategies (e.g., gradual rapport-building) compare to traditional static approaches (e.g., scripted phishing) in eliciting compliance? To address these key questions, we propose SEAR (Social Engineering Augmented Reality), the first framework investigating the feasibility of using MultiModal LLMs to execute AR Social Engineering attacks. SEAR operates through three phases: (1) AR-Based Social Context Synthesis, which captures and fuses visual and auditory data to construct social context; (2) Role-Based Multimodal RAG, which retrieves social data (e.g., Instagram images) to build personal social profiles. (3) ReInteract Agents, which executes adaptive SE attack strategies (e.g., trust-building) through iterative feedback loops, refining suggestions based on target responses. The main contributions of this paper are as follows: • Proof-of-Concept: Demonstrates the viability of AR-LLM in boosting Social Engineering efficacy, demonstrating their personalization advantages. • SEAR framework: Designs an AR-driven pipeline integrating Multimodal LLMs and social agents to execute Social Engineering attacks. • Threat Analysis on IRB-dataset: Builds an open-source IRB-dataset of 180 annotated AR-mediated social interactions among 60 participants, with detailed analysis on their subjective experiences. • Foundation for Future Defense: Provides the dataset, toolkit and analysis to catalyze research into detecting and defending AR-driven Social Engineering attacks. This study was approved by the IRB. All human-related data were collected under rigorous ethical guidelines, anonymized prior to analysis, and handled in strict accordance with data protection protocols. No personally identifying information is disclosed in this study. The study adhered to all applicable legal and ethical standards for research involving human subjects. Section 2 reviews AR/LLM security studies and identifies critical gaps. Section 3 introduces SEAR’s system design. Section 4 introduces the dataset collection methodology. Section 5 describes the experimental setup and results. Section 6 concludes the paper. 2. Related Work Social Engineering Attacks: Traditional social engineering attacks rely on exploiting human psychological weaknesses, such as fake identities, phishing emails, and predefined scenarios to trick victims into disclosing sensitive information. In the study by Krombholz et al. ( Krombholz et al. 2015 ) , traditional social engineering attacks are broadly categorized into physical approaches, social approaches, reverse social engineering, technical approaches, and socio-technical approaches. Exploiting curiosity and interest ( Granger 2001 ) is an important method used by attackers to increase the chances of success, and they usually try to establish a relationship with the potential victim. However, with the development of large language models, generative AI provides attackers with increasingly powerful tools. For example, according to Falade et al.’s research ( Falade 2023b ) , FraudGPT is a zero-threshold tool that can automatically compose convincing phishing emails. Microsoft’s VALL-E ( Wang et al. 2023 ) , an AI-based voice simulator that replicates the user’s voice, is also a powerful tool that attackers can use to scam. AI systems can adapt their phishing methods based on massive data on the internet. This adaptive capability enables them to evolve increasingly sophisticated phishing strategies. AR Privacy: The immersive capabilities of augmented reality (AR) systems introduce profound privacy risks, as exemplified by devices like Ray-Ban Stories ( Iqbal and Campbell 2023 ) —smart glasses indistinguishable from conventional eyewear that enable covert photo, video, and audio capture in public spaces. Prior research highlights vulnerabilities such as password theft via AR-assisted stereoscopic scene reconstruction ( Chen et al. 2018 ) , side-channel attacks extracting private interaction data ( Zhang et al. 2023 ) , and malicious applications conducting hidden vision operations ( Lehman et al. 2022 ) . However, these studies overlook AR’s potential for orchestrated

Read the complete text