To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification DOI: XXXXXXX.XXXXXXX Conference: First Workshop on Artificial Intelligence & Open Government at the 21st International Conference on Artificial Intelligence and Law; June 8, 2026; Singapore The First Workshop on Artificial Intelligence & Open Government at the 21st International Conference on Artificial Intelligence and Law, June 8, 2026, Singapore CCS: Computing methodologies Information extraction CCS: Information systems Clustering and classification
Maik Larooij, David Graus · 2026
Read To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification DOI: XXXXXXX.XXXXXXX Conference: First Workshop on Artificial Intelligence & Open Government at the 21st International Conference on Artificial Intelligence and Law; June 8, 2026; Singapore The First Workshop on Artificial Intelligence & Open Government at the 21st International Conference on Artificial Intelligence and Law, June 8, 2026, Singapore CCS: Computing methodologies Information extraction CCS: Information systems Clustering and classification by Maik Larooij, David Graus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10211.txt.
Introduction
Larooij and Graus tackle sensitivity sorting for the deliberative process privilege, the FOIA Exemption 5 rule letting states redact pre-decisional deliberations before releasing files. Transparency laws in the United States, the United Kingdom and the Netherlands grant citizens document-request rights. But sending uncleared files to third-party clouds is often legally or politically barred. The authors therefore classify with a small local model, Qwen3.5 9B, runnable on consumer-grade hardware.
Eight variants of language-model sentence sorting are compared with well-known prompting methods. Chain-of-thought prompting combined with few-shot prompting with error-based cases beats earlier sorting models on recall and F2 score. It closely approaches a widely used paid model, Gemini 2.5 Flash. A further analysis finds internal sentences carry more opinion-expressing verbs and more first-person phrasing. Internal tone is marked above all by mixes of multiple cues rather than any single marker.
Catalog record
| CREATOR | Maik Larooij, David Graus |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10211
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification DOI: XXXXXXX.XXXXXXX Conference: First Workshop on Artificial Intelligence & Open Government at the 21st International Conference on Artificial Intelligence and Law; June 8, 2026; Singapore The First Workshop on Artificial Intelligence & Open Government at the 21st International Conference on Artificial Intelligence and Law, June 8, 2026, Singapore CCS: Computing methodologies Information extraction CCS: Information systems Clustering and classification Maik Larooij, David Graus Abstract . Government transparency laws, like the Freedom of Information (FOIA) acts in the United States and United Kingdom, and the Woo (Open Government Act) in the Netherlands, grant citizens the right to directly request documents from the government. As these documents might contain sensitive information, such as personal information or threats to national security, the laws allow governments to redact sensitive parts of the documents prior to release. We build on prior research to perform automatic sensitivity classification for the FOIA Exemption 5 deliberative process privilege using Large Language Models (LLMs). However, processing documents not yet cleared for review via third-party cloud APIs is often legally or politically untenable. Therefore, in this work, we perform sensitivity classification with a small, local model, deployable on consumer-grade hardware (Qwen3.5 9B). We compare eight variants of applying LLMs for sentence classification, using well-known prompting techniques, and find that a combination of Chain-of-Thought prompting and few-shot prompting with error-based examples outperforms classification models of earlier work in terms of recall and F2 score. This method also closely approaches the performance of a widely-used, cost-efficient commercial model (Gemini 2.5 Flash). In an additional analysis, we find that sentences that are predicted as deliberative contain more verbs that indicate the expression of opinions, and are more often phrased in in first-person. Above all, deliberativeness seems characterized by the presence of a combination of multiple indicators, in particular the combination of first-person words with a verb for expressing opinion. 1. Introduction Several countries have implemented laws that grant citizens the right to request and obtain information from the government in the form of documents. Examples include the Freedom of Information Acts (FOIA) in the United States and United Kingdom, and the ‘Wet open overheid’ (Woo, Open Government Act) in the Netherlands. Due to the sensitive nature of these documents, the laws allow government agencies to redact parts of the documents prior to releasing them, for example when they contain personal information, threats to national security, or trade secrets. This requires officials to manually review all documents before publication, which is a time-consuming and error-prone task. The process of redacting sensitive information was identified as one of the main drivers behind substantial delays in response time ( 7 ; 25 ) . Prior research has already focused on automatic classification of FOIA exemption grounds, for example on personal information (FOIA UK Section 40) and international relations (FOIA UK Section 27) ( 11 ; 3 ; 12 ; 17 ; 9 ) , and the deliberative process privilege (FOIA US Exemption 5) ( 2 ) . These papers rely on training classifiers (such as Logistic Regression, SVM or BERT) for sentence classification. And while these papers show solid results, there are also some downsides to these methods. For example, the results in ( 2 ) show that performance fluctuates when the model is trained on documents of a different custodian of records or reviewer, suggesting that the models are sensitive to their training data and do not generalize well to out-of-distribution data, a common scenario in real-world cases. Furthermore, while sensitivity classification improves sensitivity reviewing speed and accuracy ( 14 ) , providing useful explanations to the predictions, which the proposed models do not support, would further improve this process. Large Language Models (LLMs) have the potential to address these shortcomings. LLMs can make ‘out-of-the-box’ predictions, making the training phase obsolete and therefore allowing to avoid the generalization problems. Also, LLMs are able to generate plausible (but not necessarily correct!) explanations about predictions. For this reason, 1 already leveraged GPT 3.5 for predicting whether certain paragraphs of text should be exempt from public release based on FOIA Exemption 5 (deliberative process privilege). The LLM did not perform better than traditional text classification methods, but the explanations seemed ‘not unreasonable’ and ‘helpful’. We build on the research of 1 , and subsequent work of 5 who augmented the dataset of 1 with sentence-level annotations, in three ways: (1) We test several well-established prompting techniques (Chain-of-Thought, few-shot, multi-agent), to see whether they improve classification performance over simple zero-shot prompting; (2) We compare the performance of a commercial, closed-source model (Gemini 2.5 Flash) to a smaller, local open-weights model (Qwen3.5-9B); (3) We analyze the presence of syntactical indicators, like verbs for expressing opinions, first-person words and future temporal words in sentences that are unanimously predicted deliberative or non-deliberative by all prompting variants. The second contribution is of particular importance in the open government domain: governments typically want to process documents on their own infrastructures, especially when documents may contain sensitive information. This hinders the use of larger, closed commercial models. In terms of evaluation, we argue that recall is more important than precision, as the impact of ’forgetting’ to redact sensitive information can be much larger than suggesting to redact too much. 2. Background 2.1. Deliberative Process Privilege (FOIA US Exemption 5 The U.S. Code of Federal Regulations, Title 32, § 1662.22 1 1 1 From the code of federal regulations: https://www.ecfr.gov/current/title-32/subtitle-B/chapter-XVI/part-1662/section-1662.22 provides a clear explanation of deliberative process privilege: "This privilege protects the decision-making processes of government agencies. Information is protected under this privilege if it is predecisional and deliberative. The purpose of the privilege is to prevent injury to the quality of the agency decision-making process by encouraging open and frank internal discussions, by avoiding premature disclosure of decisions not yet adopted, and by avoiding the public confusion that might result from disclosing reasons that were not in fact the ultimate grounds for an agency’s decision." From this explanation we infer that, for a record to be protected by this privilege, it must be both predecisional and deliberative . A record is predecisional if it is created before a final decision to assist a decision-maker. Its primary characteristic is that it precedes an agency’s action. It excludes "final opinions" or documents that justify and explain a decision that has already been made or a matter that has already been officially settled. A record is deliberative if it reflects internal decision-making processes, such as personal recommendations, legal advice, or policy opinions. The content must be the personal view of the author, rather than an already established policy of the agency. Examples include draft document or documents that are recommendatory in nature. While not exactly the same, other jurisdictions know similar exemption grounds. In the UK, Section 35 protects Ministerial communications and the formulation or development of government policy. Article 5.2 of the Dutch Woo protects deliberative policy views held by civil servants in the course of policy formulation. 2.2. Sensitivity Classification and Applications There have already been numerous efforts in predicting whether text should be exempt according to Freedom of Information laws. 10 showed the first SVM classifier to classify whether exemptions