How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study

Junran Wang, Xinjie Shen, Zehao Jin, Pan Li · 2026

Read How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study by Junran Wang, Xinjie Shen, Zehao Jin, Pan Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05340.txt.

Introduction

Junran Wang and three co-authors test whether vision-language models notice privacy risks when acting as embodied assistants in homes and hospitals, where they can see and handle sensitive objects. Text-only benchmarks cannot capture that physical setting. Their ImmersedPrivacy framework instead places models in a simulated three-dimensional environment built in Unity, with sound and vision together.

Three tiers of tasks grow harder: spotting sensitive items in cluttered scenes, adapting as social context shifts, and resolving clashes between direct commands and inferred privacy limits. All twelve leading models tested showed consistent deficits, decaying as clutter grew. No model passed a modest accuracy mark when context shifted, and conflict handling was weakest of all.

Catalog record

CREATORJunran Wang, Xinjie Shen, Zehao Jin, Pan Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05340

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study Junran Wang, Xinjie Shen, Zehao Jin, Pan Li Abstract As Vision-Language Models (VLMs) are increasingly deployed as autonomous cognitive cores for embodied assistants, evaluating their privacy awareness in physical environments becomes critical. Unlike digital chatbots, these agents operate in intimate spaces, such as homes and hospitals, where they possess the physical agency to observe and manipulate privacy-sensitive information and artifacts. However, current benchmarks remain limited to unimodal, text-based representations that cannot capture the demands of real-world settings. To bridge this gap, we present ImmersedPrivacy , an interactive audio-visual evaluation framework that simulates realistic physical environments using a Unity-based simulator. ImmersedPrivacy evaluates physically grounded privacy awareness across three progressive tiers that test a model’s ability to identify sensitive items in cluttered scenes, adapt to shifting social contexts, and resolve conflicts between explicit commands and inferred privacy constraints. Our evaluation of 12 state-of-the-art models reveals consistent deficits. In cluttered scenes, all models exhibit monotonic performance decay as scene complexity grows due to perceptual deficit. When social context shifts, no model exceed 65 % 65\% selection accuracy. Under conflicting commands, the best model gemini-3.1-pro perfectly balances task completion and privacy preservation in only 51 % 51\% of cases. These findings reveal that current VLMs in the physical world suffer from perceptual fragility and fail to let their knowledge of privacy cues govern their situated behavior. Our code and data is available at https://github.com/immersed-privacy/immersed-privacy . † † footnotetext: Project page: https://immersed-privacy.github.io Figure 1 : Overview of ImmersedPrivacy . Our evaluation uses image, video, and audio modalities to simulate how VLMs perceive physical environments, social states, and observation histories. It is organized into three progressive tiers: Perceptual Sensitivity Grounding, Dynamic Socio-Contextual Adaptation, and History-Conditioned Inference. 1 Introduction The trajectory of modern artificial intelligence reflects a remarkable evolution from digital conversationalists [ 1 , 39 ] to physically embodied assistants [ 11 , 7 , 6 , 14 , 5 ] , with Vision-Language Models (VLMs) increasingly positioned as the major cognitive core of these systems [ 7 , 20 ] . As these models extend beyond virtual interactions to operate in personal physical environments such as homes, offices, and hospitals, they promise unprecedented levels of proactive assistance [ 32 , 33 , 15 , 27 ] , and the access to the nuances of human life. In these scenarios, the VLM is often the sole arbiter of what constitutes a privacy violation. This physically grounded autonomy introduces profound and underexplored privacy risks [ 41 , 8 , 28 , 44 ] . While a digital model’s alignment dictates what it should or should not generate in natural language [ 25 , 34 ] , an embodied VLM’s alignment must govern what it is permitted to observe, infer, and manipulate in the physical world [ 46 , 45 , 19 , 9 ] . Previous efforts to measure physical-world privacy awareness have highlighted this critical vulnerability but remain fundamentally constrained by their unimodal, static simulation environments. Existing benchmarks frequently rely on structured text representations, such as Planning Domain Definition Language [ 35 ] , or purely descriptive natural language to simulate physical states and evaluate language models [ 25 , 16 , 34 , 17 , 26 ] . While valuable for testing high-level logical alignment without perception, this text-based approach creates a significant gap between simulated evaluations and performance in the physical world. In reality, physical environments convey privacy-relevant information through visual [ 38 , 40 ] and acoustic channels rather than text: a sensitive document is one object among many on a cluttered desk, and the social state of a room is signaled by ambient sound rather than a written label. Moreover, such information is spatially distributed and often cannot be fully captured from a single ideal viewpoint, making active exploration an inherent part of situated perception [ 36 , 31 , 3 , 10 ] . An embodied VLM must therefore perceive its surroundings through multiple modalities, actively seek informative viewpoints, and integrate these observations to recognize that, for instance, a whispered conversation implies a social context requiring discretion. Prior work on privacy in embodied settings identifies three challenges that text-based benchmarks cannot adequately capture. First, certain physical artifacts are inherently privacy-sensitive [ 12 ] , yet recognizing them requires parsing visual clutters rather than reading a textual object list [ 35 , 25 ] . Second, privacy extends beyond shielding sensitive objects: the appropriateness of an action depends on the social atmosphere of a shared space [ 29 , 23 , 4 ] , which is conveyed through ambient visual and acoustic cues rather than explicit labels. Third, real-world privacy constraints are not static rules but dynamic constructs implicitly negotiated through interaction [ 30 , 2 , 37 ] ; a trustworthy agent must extract unstated boundaries from past multimodal observations and uphold them even when an explicit command conflicts. To bridge the gap between text-simulated scenarios and this multimodal, interactive reality, we present ImmersedPrivacy , a visual-audio evaluation framework built upon a highly customized Unity-based simulator [ 32 ] . Our evaluation is structured into three progressive tiers, each targeting one of the above challenges: Tier 1: Perceptual Sensitivity Grounding: Evaluates the VLM’s ability to parse complex, cluttered environments and isolate inherently sensitive items. Rather than passively receiving a list of objects, the model must visually explore a physical space and demonstrate a foundational awareness of privacy risks without explicit textual hints, testing its core visual perception and sensitivity bias. Tier 2: Dynamic Socio-Contextual Adaptation: Evaluates the VLM’s capacity to uphold socio-contextual norms [ 29 , 4 ] in shifting physical environments. Models must proactively interpret multi-view images and audio cues (e.g., distinguishing the sound of an empty office from an active meeting) and modulate their physical actions to respect the social norms of the space. Tier 3: History-Conditioned Inferential Adherence: Evaluates the VLM’s ability to manage internal conflicts between explicit tasks and implicit privacy constraints inferred from prior observations [ 30 , 2 , 37 ] . The model observes a sequence of interactions that establish an unspoken privacy boundary. When subsequently given a benign but conflicting task command, the VLM must draw on its observation history to selectively filter or modify its execution, preventing a privacy violation. By forcing VLMs to navigate the intersection of active multimodal perception, shifting social contexts, and history-conditioned task execution, our evaluation exposes critical vulnerabilities in current state-of-the-art models. Our empirical results demonstrate a startling gap between linguistic knowledge and physical execution: while models may know a document is private in text, most fail to recognize it visually in over 40\% of heavily cluttered environments, and ignore implicit social boundaries (e.g., a person on call) in nearly half of the tested scenarios. Moreover, models’ sensitivity identification decays monotonically with visual clutter, and privacy awareness can be masked by perceptual limitations, failures that text-based evaluations, which bypasses perception, cannot reveal. 2 Framework Table 1: Diagnostic coverage of each tier. ● = directly measured; ◐ = partially attributable; ○ = not targeted. Percep. Aware. Align. Tier 1 Single-Turn ● ● ○ Tier 1 Multi-Turn ◐ ● ○ Tier 2 ◐ ◐ ●

Read the complete text