An LLM-enabled semantic-centric framework to consume privacy policies

Rui Zhao, Vladyslav Melnychuk, Jun Zhao, Jesse Wright, Nigel Shadbolt · 2025

Read An LLM-enabled semantic-centric framework to consume privacy policies by Rui Zhao, Vladyslav Melnychuk, Jun Zhao, Jesse Wright, Nigel Shadbolt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01716.txt.

Introduction

Rui Zhao leads four co-authors on a system that turns long privacy policies into clear, machine readable maps of data practice. Few people read these policies, yet they state how sites collect and share personal data. Past tools could check a formal policy but could not build one at scale. The team uses large language models to pull key facts from policy text.

The pipeline builds a knowledge graph tied to a shared privacy word list and covers the top one hundred popular websites. Experts marked up test data so the team could score how well each model did. The graphs can then feed formal rule formats for apps that need consent checks. All data and code are shared for others to reuse and improve.

Catalog record

CREATORRui Zhao, Vladyslav Melnychuk, Jun Zhao, Jesse Wright, Nigel Shadbolt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01716

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An LLM-enabled semantic-centric framework to consume privacy policies Rui Zhao, Vladyslav Melnychuk, Jun Zhao, Jesse Wright, Nigel Shadbolt Abstract In modern times, people have numerous online accounts, but they rarely read the Terms of Service or Privacy Policy of those sites, despite claiming otherwise, due to the practical difficulty in comprehending them. The mist of data privacy practices forms a major barrier for user-centred Web approaches such as Solid, and for data sharing and reusing in an agentic world. Existing research proposed methods for using formal languages and reasoning for verifying the compliance of a specified policy, as a potential cure for ignoring privacy policies. However, a critical gap remains in the creation or acquisition of such formal policies at scale. We present a semantic-centric approach for using state-of-the-art Natural Language Processing (NLP) tools, namely large language models (LLM), to automatically identify key information about privacy practices from privacy policies, and construct Pr 2 ​ Graph \textit{Pr}^{2}\textit{Graph} , knowledge graph with grounding from Data Privacy Vocabulary (DPV) for privacy practices, to support downstream tasks. Along with the pipeline, the Pr 2 ​ Graph \textit{Pr}^{2}\textit{Graph} for the top-100 popular websites is also released as a public resource, by using the pipeline for analysis. We also demonstrate how the Pr 2 ​ Graph \textit{Pr}^{2}\textit{Graph} can be used to support downstream tasks by constructing formal policy representations such as Open Digital Right Language (ODRL) or perennial semantic Data Terms of Use (psDToU). To evaluate the technology capability, we enriched the Policy-IE dataset by employing legal experts to create custom annotations. We benchmarked the performance of different large language models for our pipeline and verified their capabilities. Overall, the pipeline and relevant resources shed light on the possibility of large-scale analysis of online services’ privacy practices, as a promising direction to audit the Web and the Internet. We release all datasets and source code as public resources to facilitate reuse and improvement. 1 Introduction Collectively, we are subscribing to an ever-increasing number of online services - each of which has us sign custom “Terms of Service” or “Privacy Policies” to enable the collection and use of our data. Despite the privacy and legal implications, less than 7% of users read these agreements, making them “the biggest lie on the Internet” [ 25 ] . A major cause for that is information overload, as the average privacy policy requires a 29-minute read time. Current regulations, including the General Data Protection Regulation (GDPR) [ 9 ] and the Digital Service Act (DSA) [ 10 ] , exacerbate the issue, requiring companies to collect more permissions from users without providing technical or legal standards for facilitating users, thus more burdens for the users. As a reaction to the online privacy challenges, decentralized personal data stores [ 14 ] have emerged as an alternative pattern: users hold data in their own storage and manage data access, while applications/services make requests to use the data. In particular, Solid (Social Linked Data) [ 31 ] pioneers using semantic technologies for data representation, for maximizing data interoperability and data reusing. However, less discussion has been focused on how to support users to make decisions regarding the data access and usage requests from the decentralized applications, thus still relying on privacy policies. This poses challenges for Solid towards an ethical Web ecosystem, where user-centric data governance, cross-boundary data sharing and intelligent autonomous agents all require well-understood data usage repercussions during (automated or manual) decision making. Research on usage control [ 7 , 32 ] , especially ODRL [ 1 ] and psDToU [ 42 ] building on semantic technologies, provides promising solutions to this issue, where formal policies and automated reasoning are employed to check the compliance of data usage, thus further facilitating decision-making. Work [ 42 ] has also demonstrated the possibility of integrating a policy engine into Solid protocol, as well as supporting continuous (“perennial”) policy checking for downstream derived data. In addition, research such as DPV [ 19 ] also created standard interoperable vocabularies for expressing concepts in formal languages. But an important question is not answered by such research: where do the formal policies come from ? Indeed, it would not be difficult to convert between different formal representations if a mapping is provided. Yet the problem lies in the source of information: at best, services only provide natural-language policies, rather than formal policies. In this paper, we pioneer a software pipeline for automatically converting natural-language privacy policies into formal knowledge of the privacy practices in them, as a knowledge graph called \textit{Pr}^{2}\textit{Graph} , for the sake of interoperability and extensibility provided by RDF [ 37 ] . The pipeline takes advantage of advancements of large-language models (LLMs) [ 40 , 17 , 34 ] , for both their remarkable natural-language-understanding performance, and their wide availability – off-the-shelf tools allow usage by people with modest technical abilities. We argue the necessity of a KG-centric view, instead of the general view on complementing LLM/NLP with KG [ 26 ] , in this context: unambiguous and precise meanings of policy terms are a necessity, while LLMs can only provide a probabilistic interpretation; issues such as hallucination [ 18 ] also reduce the appropriateness of LLMs; appropriate prompting engineering [ 24 , 30 ] is still necessary in the forseeable future when new queries are needed. On the other hand, KGs are extensible by design; through technologies like OWL [ 38 ] , KGs provide the needed formality and interoperability. Their white-box nature also supports auditing of information, before downstream tasks consume them, or to inform changes. Along with the pipeline, we also provide the \textit{Pr}^{2}\textit{Graph} for the top-100 most-visited websites, as an open and accessible information source, benefiting users with less technical or resource advantages; we also showcase a valuable downstream task for converting \textit{Pr}^{2}\textit{Graph} into formal policies of psDToU and ODRL powered by semantic technologies. In addition, we constructed a dataset by using human annotators, extending the Policy-IE dataset [ 2 ] , to form a high-quality basis for evaluating NLP pipeline’s accuracy – despite being phenomenal, LLMs’ capability in understanding privacy policies is not yet well-understood (Sec 2.1 ). We used this dataset to benchmark the performance of several state-of-the-art LLMs in different settings, verifying the capability of the models and thus the pipeline. Together, they form a complete story in providing, evaluating and demonstrating the utility of a semantic-centric NLP pipeline for privacy policy analysis and usage control, providing a valuable resource for filling in both practical and theoretical gaps of existing practices. Resource Location Due to the nature of these resources, they are located separately for audiences with different targets: 1. NLP pipeline and evaluation code: https://github.com/renyuneyun/pp-analyzer 2. Privacy policy annotations: https://doi.org/10.5281/zenodo.15392162 3. \textit{Pr}^{2}\textit{Graph} for top-100 most-visited websites 1 1 1 It also contains the formal policy in psDToU and ODRL as a demonstration of downstream tasks : https://doi.org/10.5281/zenodo.15408913 2 Background and Related Work 2.1 Privacy Policy Processing Privacy policies have been a research topic across several themes, with diverse methods and goals. Here, we present a review of the research on automatically identifying and re-presenting its main information, and those supporting users’ decision-making, due to their close link to this work. Different initiatives attempt to

Read the complete text