CapRecover : A Cross-Modality Feature Inversion Attack Framework on Vision Language Models Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland. Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31,
Kedong Xiu, Sai Qian Zhang · 2025
Read CapRecover : A Cross-Modality Feature Inversion Attack Framework on Vision Language Models Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland. Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, by Kedong Xiu, Sai Qian Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22828.txt.
Introduction
Kedong Xiu and Sai Qian Zhang demonstrate an attack that steals the meaning of a user's photo from a vision-language model without ever reconstructing the photo, in this 2025 paper. Many such systems split the work: a visual encoder on the user's device compresses the image into features, and a cloud model reads those features. Anyone intercepting the features might hope to rebuild the image, but earlier methods produced only blur.
CapRecover skips the picture and goes straight for its semantics, recovering labels and captions from the intercepted features alone. It named the subject of CIFAR-10 images with up to ninety-three percent accuracy and wrote fluent captions from features of a standard network. Deeper layers leaked far more meaning than shallow ones. The authors also propose a defense: scramble each layer's features with noise that the next layer quietly removes, blinding eavesdroppers while the model works on.
Catalog record
| CREATOR | Kedong Xiu, Sai Qian Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CapRecover : A Cross-Modality Feature Inversion Attack Framework on Vision Language Models Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland. Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755203 ISBN: 979-8-4007-2035-2/2025/10 2584 CCS: Security and privacy CCS: Computing methodologies Artificial intelligence Kedong Xiu, Sai Qian Zhang Abstract . As Vision-Language Models (VLMs) become increasingly integrated into user-facing applications, they are often deployed in split DNN configurations, where the visual encoder (e.g., ResNet or ViT) runs on user-side devices and only intermediate features are transmitted to the cloud for downstream processing. While this setup reduces communication overhead, the intermediate data features containing sensitive information can also expose users to privacy risks. Prior work has attempted to reconstruct images from these features to infer semantics, but such approaches often produce blurry images that obscure semantic details. In contrast, the potential to directly recover high-level semantic content — such as image labels or captions — via a cross-modality inversion attack remains largely unexplored. To address this gap, we propose CapRecover , a general cross-modality feature inversion framework that directly decodes semantic information from intermediate features without requiring image reconstruction. Additionally, CapRecover can be used to reverse engineer traditional neural networks for computer vision tasks, such as ViT, ResNet, and others. We evaluate CapRecover across multiple widely used datasets and victim models. Our results demonstrate that CapRecover can accurately recover both image labels and captions without reconstructing a single pixel. Specifically, it achieves up to 92.71% Top-1 accuracy on the CIFAR-10 dataset for label recovery, and generates fluent and relevant captions from ResNet50’s intermediate features on COCO2017 dataset, with ROUGE-L scores up to 0.52. Furthermore, an in-depth analysis of ResNet-based models reveals that deeper convolutional layers encode significantly more semantic information, whereas shallow layers contribute minimally to semantic leakage. Furthermore, we propose a straightforward and effective protection approach that adds random noise to the intermediate image features at each middle layer and subsequently removes the noise in the following layer. Our experiments indicate that this approach effectively prevents information leakage without additional training costs. 1. Introduction The rapid advancement of Vision-Language Models (VLMs) has fundamentally reshaped the landscape of multimodal AI, positioning these models as the cornerstone of modern user-facing assistants. Unlike traditional Large Language Models (LLMs), VLMs seamlessly integrate image understanding with natural language processing, enabling comprehensive interpretations of real-world data. By harnessing vast amounts of textual and visual information, VLMs have achieved impressive results in tasks such as image captioning (e.g., GPT-4o ( OpenAI 2024 ) ), text-to-image generation (e.g., Stable Diffusion ( AI [n. d.] ) ), and optical character recognition. The success of architectures like CLIP ( Radford et al. 2021 ) and BLIP2 ( Li et al. 2023b ) underscores their potential to drive significant innovations in both research and practical applications. Despite these advances, VLMs are not without vulnerabilities. Recent research has predominantly focused on security threats such as prompt jailbreaking—where attackers manipulate models to produce harmful outputs—and prompt-stealing attacks that extract sensitive user prompts from generated images ( Gong et al. 2023 ; Luo et al. 2024 ; Shayegani et al. 2023 ; Shen et al. 2024 ) . However, one critical dimension remains underexplored: the leakage of sensitive information through intermediate feature representations. As illustrated in Figure 1 , an adversary who gains access to intermediate image features from the victim model (for instance, from a local device) could reconstruct the original image caption, potentially exposing private user data. This issue is especially relevant in the split DNN computing paradigm ( Mudvari et al. 2024 ; Zhang et al. 2024 ; He et al. 2024 ; Lu et al. 2024 ) , where a large model is divided into multiple blocks tailored to the computational capabilities of edge devices. In this setup, user data is initially processed on the edge device using the first layers of the model, and intermediate results are transmitted to a remote server for processing by later layers. This data transfer poses a risk, as it can be intercepted and exploited by attackers to reconstruct user inputs. Consequently, understanding and mitigating the leakage of intermediate image features is imperative for protecting user data and maintaining the integrity of VLM-driven services. Figure 1. Illustration of the cross-modality feature inversion attack scenario. In the depicted attack scenario, the adversary steals the intermediate image features from the visual model. Leveraging these stolen features, the adversary employs CapRecover to reconstruct the image caption/label, potentially revealing sensitive or private information. Prior work ( Xu et al. 2024 ; Zhu et al. 2025 ; He et al. 2019 ) has explored reconstructing images from intermediate features to further infer their semantic content. However, these methods are indirect and often suffer from preserving fine-grained semantics with poor visual fidelity (e.g., blurriness, missing textures), which could consequently limit their performance. Moreover, some attackers may primarily focus on the semantic meaning in the target image, e.g., what is happening and who is involved. This raises a critical yet underexplored question: Is it possible—and potentially more effective—for an attacker to directly recover high-level semantic information, such as image labels or captions, from intermediate features, without reconstructing the image at all? This new form of feature inversion attack shifts the adversary’s focus from pixel-level recovery to semantic reconstruction, and more directly threatens user privacy in practical scenarios. 1.1. Our work and contributions In this paper, we take a fundamentally different approach: instead of reconstructing the image, we directly recover/reconstruct the image’s semantic content from the leaked intermediate image features. We introduce CapRecover , a generic cross-modality feature inversion framework that exposes a critical vulnerability in VLMs: the capacity to reconstruct textual descriptions from intermediate image features. CapRecover bridges intermediate visual features with a pre-trained language model, bypassing image reconstruction entirely. By learning a lightweight projection layer between the vision and language domains, CapRecover enables accurate and fluent semantic recovery from commonly used encoders such as ResNet and ViT. To understand the privacy implications of this attack, we consider a threat model where the attacker passively observes the intermediate visual features sent from a user’s device to the cloud in a split-VLM pipeline. The attacker has no access to the original image or the language module of the VLM, and aims to infer semantic content directly from the encoder output. While a full description is provided in Sec. 2 , we note here that this threat model aligns with realistic deployment scenarios in edge-cloud systems and highlights a previously underestimated attack surface. We evaluate CapRecover on multiple datasets and VLM architectures across two key tasks: image classification and image captioning. Our experiments show that CapRecover can recover labels and captions with high fidelity—even without reconstructing a single pixel. We further analyze how semantic leakage varies across encoder depths and propose a simple, training-free defense mechanism that reduces leakage via reversible noise injection. We summarize our contributions as follows: • A general adversarial framework. We propose CapRecover , the first generic cross-modality feature inversion framework that directly reconstructs semantic information from the leaked intermediate image features, without requiring pixel-level image reconstruction. By leveraging a feature-to-text alignment mechanism, CapRecover effectively