Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Search

Bo Ma 1 , 2 Wei Qi Yan 1 Jinsong Wu 3 , 4 · 2026

Read Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Search by Bo Ma 1 , 2 Wei Qi Yan 1 Jinsong Wu 3 , 4 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13728.txt.

Introduction

Bo Ma, Wei Qi Yan, and Jinsong Wu propose Bodhi VLM, a framework that models privacy alignment for hierarchical visual representations. Privacy methods often add noise to such representations, and it is hard to model how the noise lines up with a declared budget. Bodhi VLM links sensitive concepts to layer-wise groups via clustering. Bottom-up and top-down searches locate sensitive feature regions across scales. An Expectation-Maximization Privacy Assessment module compares the fitted distribution to a reference and gives a budget-alignment signal.

Catalog record

CREATORBo Ma 1 , 2 Wei Qi Yan 1 Jinsong Wu 3 , 4
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55170
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13728

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Search Bo Ma 1 , 2 Wei Qi Yan 1 Jinsong Wu 3 , 4 Abstract Learning systems that preserve privacy often inject noise into hierarchical visual representations; a central challenge is to model how such perturbations align with a declared privacy budget in a way that is interpretable and applicable across vision backbones and vision–language models (VLMs). We propose Bodhi VLM , a privacy-alignment modeling framework for hierarchical neural representations : it (1) links sensitive concepts to layer-wise grouping via NCP and MDAV-based clustering; (2) locates sensitive feature regions using bottom-up (BUA) and top-down (TDA) strategies over multi-scale representations (e.g., feature pyramids or vision-encoder layers); and (3) uses an Expectation-Maximization Privacy Assessment (EMPA) module to produce an interpretable budget-alignment signal by comparing the fitted sensitive-feature distribution to an evaluator-specified reference (e.g., Laplace or Gaussian with scale c / ϵ c/\epsilon ). The output is reference-relative and is not a formal differential-privacy estimator. We formalize BUA/TDA over hierarchical feature structures and validate the framework on object detectors (YOLO, PPDPTS, DETR) and on the visual encoders of VLMs (CLIP, LLaVA, BLIP). BUA and TDA yield comparable deviation trends; EMPA provides a stable alignment signal under the reported setups. We compare with generic discrepancy baselines (Chi-square, K-L, MMD) and with task-relevant baselines (MomentReg, NoiseMLE, Wass-1). Results are reported as mean ± \pm std over multiple seeds with confidence intervals in the supplementary materials. This work contributes a learnable, interpretable modeling perspective for privacy-aligned hierarchical representations rather than a post hoc audit only. Source code: Bodhi-VLM GitHub repository I Introduction Privacy-preserving learning systems often inject noise into hierarchical visual representations (e.g., feature pyramids in detectors or vision-encoder layers in VLMs). A central question for neural network modeling is how to characterize and interpret the alignment between observed layer-wise perturbations and a declared privacy budget \epsilon under an evaluator-specified reference mechanism (e.g., Laplace or Gaussian with scale c/\epsilon ). We propose Bodhi VLM , a privacy-alignment modeling framework for hierarchical visual representations that: (1) models sensitive feature structure in a layer-wise manner via bottom-up (BUA) and top-down (TDA) grouping over multi-scale features; and (2) produces an interpretable budget-alignment signal via an Expectation-Maximization Privacy Assessment (EMPA) module that compares the fitted sensitive-feature distribution to the reference. The output is reference-relative and does not constitute formal differential privacy certification or estimation of effective \epsilon . BUA and TDA traverse hierarchical feature structures (e.g., feature pyramid networks in detectors, or the visual encoder layers in CLIP [ 23 ] and LLaVA [ 18 ] ), producing paired sensitive and non-sensitive feature groups per layer using NCP and MDAV-based clustering; EMPA then uses these groups in an EM-like procedure to output a discrepancy signal. The framework does not modify the underlying architecture but provides a learnable, interpretable view of how privacy-related noise is distributed across layers and how it aligns with a declared budget. Contributions: (1) We formalize BUA and TDA for sensitive feature search over FPN and VLM vision backbones using NCP and MDAV-based clustering. (2) We introduce EMPA as a feature-level budget-alignment (discrepancy) estimator and integrate it into Bodhi VLM. (3) We show how Bodhi VLM applies to the vision towers of representative VLMs (Section III-H ); extending the assessment to full cross-modal fusion and language-decoding pathways remains future work. (4) We report experiments on MOT20 and COCO and design an experimental protocol for comparison (BUA vs. TDA, EMPA vs. Chi-square, K-L, MMD, and task-relevant baselines MomentReg, NoiseMLE, Wass-1). (5) We compare Bodhi VLM with representative VLM privacy methods (ViP, DP-Cap, DP-MTV, VisShield) and with distribution-level and task-specific baselines (Section II-D ); we demonstrate the framework on CLIP, LLaVA, and BLIP vision encoder features (Section IV-B6 ). Scope and relevance to TNNLS. The core contribution of this work is modeling for hierarchical neural representations: we propose a formalization of layer-wise sensitive-feature structure (BUA/TDA), a grouping mechanism (NCP/MDAV) over multi-scale visual features, and an interpretable budget-alignment estimator (EMPA). This is a contribution to learning systems and representation structure , not only to applied privacy evaluation. The problem is situated within neural network architectures (vision backbones and VLM encoders) and is of interest to the TNNLS readership because it addresses how to model and interpret privacy-related perturbations in hierarchical representations produced by learned vision and vision–language models. II Related Work II-A Differential Privacy and Privacy Auditing Differential privacy (DP) [ 10 ] provides a rigorous notion of privacy loss via a budget \epsilon . Assessing whether a trained model satisfies a given \epsilon has been studied from both theoretical and empirical angles. One-line or few-run privacy auditing [ 21 , 14 ] aims to estimate the effective \epsilon from model behavior; our EMPA layer provides a complementary, feature-level assessment that consumes the outputs of BUA/TDA. Hitaj et al. [ 12 ] use deep generative models to study information leakage in collaborative learning. Agrawal and Aggarwal [ 2 ] quantify privacy in data mining. DP for deep learning [ 1 ] and for federated settings [ 20 ] are standard baselines; Bodhi VLM does not replace training-time DP but evaluates whether the observed noise aligns with the declared budget. II-B Hierarchical and Multimodal Representations Feature pyramid networks (FPN) [ 16 ] offer multi-scale representations used in object detection; we leverage their hierarchical structure for BUA and TDA. Vision-language models (VLMs) combine a visual encoder with a language model: CLIP [ 23 ] aligns image and text embeddings via contrastive learning; LLaVA [ 18 ] projects visual features into the LLM space for instruction following; BLIP [ 15 ] unifies encoding and decoding for vision-language understanding and generation. Recent VLMs such as Qwen2-VL [ 27 ] and InternVL [ 6 ] scale to 4K resolution and support video understanding, expanding the scope of multimodal tasks. These architectures expose layer-wise visual features (ViT blocks or CNN stages), to which BUA and TDA can be applied without modifying the model. II-C Privacy in Vision and Vision-Language Models Privacy for VLMs has gained attention with several distinct approaches. ViP [ 29 ] trains differentially private vision transformers via masked autoencoders and DP-SGD on LAION400M ( \epsilon{=}8 ), achieving 55.7% ImageNet linear probing. DP-Cap [ 24 ] improves upon ViP using image captioning for multimodal DP representation learning, reaching 65.8% ImageNet-1K under \epsilon{=}8 on LAION-2B. DP-MTV [ 22 ] enables many-shot multimodal in-context learning with (\varepsilon,\delta) -DP by aggregating demonstrations into compact task vectors, preserving 50% accuracy on VizWiz at \varepsilon{=}1.0 . VisShield [ 5 ] adopts a de-identification paradigm: it uses instruction-tuned VLMs to locate and mask Protected Health Information (PHI) in medical images via OCR. Membership inference on VLMs [ 13 ] reveals that instruction-tuning data is vulnerable to attacks achieving AUC >0.8 with few samples. Bodhi VLM complements these works: whereas ViP/DP-Cap/DP-MTV focus on training with DP and VisShield on de-identification, Bodhi VLM assesses whether the observed privacy noise aligns with a declared budget, providing a feature-level audit tool that can be applied to any VLM vision tower. II-D External Comparison Methods Used in This Work Our core innovation is hierarchical sensitive-feature modeling , grouping mechanisms (BUA/TDA with NCP/MDAV), and budget-alignment estimation

Read the complete text