Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models

Haeun Jang, Hwan Chang, Hwanhee Lee · 2026

Read Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models by Haeun Jang, Hwan Chang, Hwanhee Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03926.txt.

Introduction

Haeun Jang, Hwan Chang, and Hwanhee Lee introduce Doc-PP, a benchmark testing whether vision-language models obey user-set disclosure policies when answering questions about documents. Real deployments face dynamic rules about what may be revealed in which context. Safety research has mostly studied implicit social norms or text-only settings, leaving multimodal documents, with their mix of images and text, unexamined.

The benchmark is built from real-world reports requiring reasoning across visual and textual elements under strict non-disclosure policies. Evaluation exposed a systemic reasoning-induced safety gap: models often leak information when the answer must be inferred by combining pieces across modalities. Supplying extracted text improved perception yet made leakage easier. For companies deploying document assistants over sensitive files, the benchmark names a failure mode that standard safety tests miss entirely.

Catalog record

CREATORHaeun Jang, Hwan Chang, Hwanhee Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03926

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models Haeun Jang, Hwan Chang, Hwanhee Lee Abstract The deployment of Large Vision-Language Models (LVLMs) for real-world document question answering is often constrained by dynamic, user-defined policies that dictate information disclosure based on context. While ensuring adherence to these explicit constraints is critical, existing safety research primarily focuses on implicit social norms or text-only settings, overlooking the complexities of multimodal documents. In this paper, we introduce Doc-PP ( Doc ument P olicy P reservation Benchmark), a novel benchmark constructed from real-world reports requiring reasoning across heterogeneous visual and textual elements under strict non-disclosure policies. Our evaluation highlights a systemic Reasoning-Induced Safety Gap : models frequently leak sensitive information when answers must be inferred through complex synthesis or aggregated across modalities, effectively circumventing existing safety constraints. Furthermore, we identify that providing extracted text improves perception but inadvertently facilitates leakage. To address these vulnerabilities, we propose DVA ( D ecompose– V erify– A ggregation), a structural inference framework that decouples reasoning from policy verification. Experimental results demonstrate that DVA significantly outperforms standard prompting defenses, offering a robust baseline for policy-compliant document understanding. 1 1 1 https://github.com/hwanchang00/doc-pp 1 Introduction Figure 1: An example of multimodal policy preservation failure. The LVLM leaks protected information by reasoning across visual and textual evidence, violating an explicit non-disclosure policy. Red text indicates the leaked information. The advancement of Large Vision-Language Models (LVLMs) has enabled their widespread adoption for question answering over complex multimodal documents ( 1 ; 11 ) . In real-world deployments, however, such documents are often accompanied by user-defined policies that specify what information may or may not be disclosed depending on the audience or usage context ( 8 ) . For example, a quarterly financial report may be safely shared with external stakeholders only under the constraint that certain region-specific revenue figures remain confidential. These restrictions are externally imposed and vary across users, organizations, or access scenarios ( 10 ) . Given the dynamic nature of these constraints, it is operationally infeasible to manually mask sensitive regions for every policy update. As a result, an ideal LVLM-based document QA system should be capable of conditionally adhering to these dynamic policies, selectively withholding sensitive information while still providing useful responses. Despite this practical need, we find that even advanced LVLMs, such as GPT-5.2, fail to preserve user-defined policies when answering queries that require cross-modal reasoning. As illustrated in Figure 1 , even when the model is explicitly instructed not to disclose "Middle East revenue," it extracts a percentage from a pie chart, identifies the total revenue from the text, and calculates the protected information through implicit reasoning. However, prior research on the safety of LVLM ( 2 ) has primarily focused on static, implicit norms rather than dynamic, user-specified constraints. In the text-only domain, CoPriva ( 4 ) begins to study policy preservation under user-defined constraints, but are limited to textual inputs and do not account for heterogeneous visual components common in real-world documents. Moreover, their queries typically target localized text spans, rather than requiring reasoning over information distributed across multiple parts of a document. In this paper, we introduce Doc-PP ( Doc ument P olicy P reservation Benchmark), a benchmark designed to evaluate user-defined policy preservation in document-level question answering. We construct Doc-PP from real-world PDF documents spanning diverse document types, including financial and industry reports, where sensitive information is embedded across different sections and visual components. Each document is paired with explicit non-disclosure policies and queries that require reasoning over information distributed across multiple spans of the document. Answering these questions often necessitates integrating evidence from heterogeneous sources such as textual descriptions, tables, charts, and figures, reflecting realistic document understanding scenarios. Our evaluation of various LVLMs to Doc-PP reveals several key findings that highlight a systemic multimodal reasoning gap. First, we observe a pronounced disparity between explicit and implicit queries: while models often comply with policies when sensitive information is explicitly requested, leakage rates increase sharply when answers must be inferred through reasoning. Second, we find that providing models with OCR-extracted text—despite improving perceptual clarity—frequently exacerbates information leakage. Finally, we find that policy adherence degrades significantly in multi-modal evidence settings, where models must integrate information across text and visuals, indicating that cross-modal alignment often bypasses existing safety mechanisms. To mitigate these vulnerabilities, we explore a range of intervention strategies. We find that standard prompting-based defenses, including Chain-of-Thought prompting and post-hoc output revision, offer limited protection, as they fail to intercept the intermediate reasoning steps that lead to policy violations. In response, we propose DVA ( D ecompose– V erify– A ggregation), a simple structural inference framework that explicitly separates reasoning from policy verification. Our results demonstrate that DVA substantially reduces leakage across document types and query settings, providing a practical baseline for improving policy preservation in multimodal document QA systems. 2 Related Work As large language models (LLMs) ( 17 ) have advanced, they are increasingly deployed in settings where rich context is provided at inference time, often including personal, organizational, or situational information. This has motivated a growing body of work on how models handle private or sensitive information conditioned on the surrounding context. Prior studies such as ConfAIde ( 12 ) and PrivacyLens ( 19 ) investigate whether LLMs respect contextual privacy expectations, drawing on the theory of Contextual Integrity ( 15 ) to evaluate the appropriateness of information flow under different social roles, recipients, and interaction settings. Recently, this line of inquiry has expanded to the multimodal domain. VLM-GEOPRIVACY 2 investigates whether LVLMs can respect privacy in geolocation tasks by interpreting visual cues. However, these works predominantly focus on implicit norms of information disclosure—relying on the model to deduce what is appropriate based on broad social conventions or assumed privacy expectations—rather than enforcing specific directives provided by the user. In parallel, CoPriva 4 shifts the focus to explicit, user-defined constraints, benchmarking model adherence to specified non-disclosure policies in text-only environments. Our work extends this frontier into the multimodal document domain. Unlike prior text-based benchmarks, Doc-PP requires models to perform cross-modal integration to identify and prevent policy violations, particularly in scenarios where sensitive information is dispersed across complex visual-linguistic dependencies. Figure 2: Overview of the three-stage Doc-PP benchmark construction pipeline: (1) Policy Construction, where confidential targets are generated and filtered; (2) Query Construction, producing reasoning-heavy queries; and (3) Evaluation, using a checklist-based framework to measure leakage and faithfulness. 3 Doc-PP: Dataset Construction 3.1 Task Definition and Setup We introduce Policy-Preserving Multimodal Document QA, a task designed to evaluate whether LVLMs can fulfill information requests from documents containing both textual and visual elements while strictly adhering to explicit non-disclosure policies. Since confidential information can be encoded in either modality, this task requires sophisticated cross-modal reasoning (e.g., combining a chart’s percentage with a total value from the text) to identify and redact sensitive content. We define an evaluation instance as a triplet (D,P,Q) : • Document D : A document segment containing integrated textual and visual elements. To analyze how models perceive sensitive information across different formats, we consider two input conditions: (i) D^{ocr} , providing OCR-parsed document content, including text, images, and tables as separate modalities, and (ii) D^{img} , providing the images (PNG)

Read the complete text