Addressing Labelled Data Scarcity: Taxonomy-Agnostic Annotation of PII Values in HTTP Traffic using LLMs

Anonymous Authors, 1 st Thomas Cory, 2 nd Axel Küpper · 2026

Read Addressing Labelled Data Scarcity: Taxonomy-Agnostic Annotation of PII Values in HTTP Traffic using LLMs by Anonymous Authors, 1 st Thomas Cory, 2 nd Axel Küpper (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06305.txt.

Introduction

Thomas Cory and Axel Küpper use language models to label personal data leaking through web and mobile app traffic. Privacy audits scan outbound HTTP messages for personally identifiable information, but learning based detectors need scarce hand labeled traffic. They also bind tightly to one fixed label taxonomy, so they transfer poorly when definitions of personal data shift across domains or over time.

Their pipeline supplies the taxonomy at runtime instead of baking it into training. Deterministic preprocessing feeds label level classification, targeted value labeling, and output checking stages run by language models. A tool creates synthetic traffic with validated labels, avoiding sensitive real captures for evaluation and prompting examples. Across three label sets of varied scope, the pipeline detected personal data types well and extracted the matching values.

Catalog record

CREATORAnonymous Authors, 1 st Thomas Cory, 2 nd Axel Küpper
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06305

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Addressing Labelled Data Scarcity: Taxonomy-Agnostic Annotation of PII Values in HTTP Traffic using LLMs Anonymous Authors, 1 st Thomas Cory, 2 nd Axel Küpper Abstract Automated privacy audits of web and mobile applications often analyse outbound HTTP traffic to detect Personally Identifiable Information (PII) leakage. However, existing learning-based detectors typically depend on scarce, manually labelled traffic and are tightly coupled to fixed label taxonomies, limiting transferability across domains and evolving definitions of PII. This paper investigates whether Large Language Models (LLMs) can support taxonomy-agnostic annotation of explicitly transmitted PII values in HTTP message bodies when the taxonomy is provided at runtime. We introduce a multi-stage LLM-based pipeline that combines deterministic pre-processing with label-level classification, targeted instance-level value annotation, and output validation. To enable controlled evaluation and exemplar-based prompting without relying on sensitive real-user captures, we further propose an LLM-based generator for synthetic HTTP traffic with manually validated, taxonomy-derived PII annotations. We evaluate the approach across three taxonomies spanning different PII domains and granularity levels. Results show that the pipeline accurately detects PII types and extracts corresponding values for concrete PII taxonomies. Overall, our findings position LLMs as a promising foundation for flexible, taxonomy-agnostic traffic annotation and for creating labelled data under evolving privacy taxonomies. I Introduction Privacy regulations, such as the GDPR [ 5 ] and CCPA [ 23 ] , introduce comprehensive legal frameworks regulating the collection and processing of Personally Identifiable Information (PII). However, assessing whether the data and privacy practices of web and mobile applications are compliant with these regulatory frameworks is challenging and often involves time-consuming manual input. The last decade has seen repeated attempts to reduce the manual overhead of such privacy audits through the development of automated approaches. Automated analysis of applications’ network traffic plays a key role in these efforts, as network traffic represents the primary vector for PII leakage, i.e. the transfer of PII out of the control sphere of the data subject [ 17 ] . Early attempts employed keyword-based heuristics to detect PII values in transmissions, but were inherently limited due to the variability of PII values. Later approaches employed machine-learning techniques such as Support Vector Machines (SVMs) or Bidirectional Encoder Representations from Transformers (BERT) [ 6 ] to detect PII values with greater flexibility. However, these approaches still have a key drawback: they require training on high-quality labelled data, which traditionally requires careful manual effort to create. Furthermore, they are locked to the label taxonomy and type of data they were trained on, greatly limiting their transferability, as even small changes to either generally negatively impact their performance substantially. This is particularly problematic for the task of PII detection, as the definition of an exhaustive taxonomy of PII types is challenging. Instead, existing taxonomies focus on specific domains or granularity levels and are rarely fully compatible with each other [ 25 ] . This challenge is reflected in the PII detection and de-identification literature more broadly: recent work reports little uniformity in how researchers and corpus creators classify PII, with taxonomies differing markedly in granularity and semantic scope, which in turn affects both detection and labelling behaviour [ 24 ] . Furthermore, the task of manually labelling and curating ground-truth datasets is highly labour-intensive and needs to be repeated for every update made to a label taxonomy [ 25 , 4 ] . The result is a widespread scarcity of high-quality datasets of labelled network traffic with up-to-date label taxonomies and limited flexibility of existing PII detectors. Generative models, particularly Large Language Models (LLMs), represent a paradigm shift in this regard: rather than requiring careful training and attunement to specific label taxonomies and input types, they are generalists that can be prompted to perform a variety of tasks without requiring retraining [ 28 ] . Prior work has demonstrated their applicability for a wide range of language processing and specifically Named Entity Recognition (NER) and annotation tasks [ 26 , 4 ] , motivating us to evaluate their suitability for taxonomy-agnostic annotation of PII values in HTTP traffic. In contrast to prior network-traffic PII detectors that are either heuristic-driven or trained and evaluated on fixed label sets, our work evaluates whether LLMs can mitigate labelled-data scarcity in network-traffic privacy auditing by enabling taxonomy-agnostic, value-level annotation of PII in HTTP messages and by supporting controlled generation of labelled synthetic traffic. We address the following research questions: RQ1: To what extent can an LLM-based pipeline accurately identify and annotate explicitly transmitted PII values in HTTP message payloads when the label taxonomy is provided at runtime? RQ2: How do key architectural choices affect label-level and instance-level annotation quality? RQ3: To what extent can synthetic HTTP messages with ground-truth PII annotations support evaluation and exemplar-based prompting, and how robust is performance across taxonomies that differ in domain focus and granularity? In addressing these research questions, our contributions are threefold: C1: We present a multi-stage LLM-based approach for the highly granular identification and annotation of PII entities in HTTP message payloads, capable of switching label taxonomies at runtime. C2: We complement the presented annotation approach with an LLM-based approach for the generation of synthetic HTTP traffic with pre-labelled PII values, reducing the need for labour-intensive manual annotation and curation of ground-truth datasets for shifting label taxonomies. C3: We evaluate the presented annotation approach on three PII label taxonomies covering different domains and granularity levels to assess the performance, suitability, and flexibility of LLM-based PII value annotation. II Related Work A substantial body of prior work has studied privacy leakage by analysing outbound network traffic from mobile applications. Early systems relied on rule- and keyword-based heuristics to identify sensitive values such as device identifiers or locations within HTTP requests, often via VPN-based interception on unrooted devices [ 21 , 9 , 14 ] . While effective for well-structured payloads and known formats, such heuristics are brittle under variable value representations, application-layer encoding, and evolving schemas. Later systems therefore introduced learning-based PII detectors. ReCon [ 17 ] interposes on network traffic and uses machine learning to detect and extract PII values across platforms and apps, while PrivacyProxy [ 22 ] identifies likely PII through app-specific signatures and crowd aggregation, reducing raw-traffic exposure through hashed representations. Ren et al. [ 16 ] further use ReCon longitudinally, showing that PII leakage patterns change across app versions and that transport-layer protection is adopted slowly. More recently, Kohli et al. [ 8 ] compare tabular models over structured packet features with document-classification models over raw packet content, including transformer-based architectures for mobile network packets. This line of work establishes network traffic as a primary vantage point for privacy auditing, but remains largely tied to the label schemes, feature representations, and training data available at development time. LLMs have also been explored for structured and unstructured information extraction, including NER and privacy-related labelling. GPT-NER [ 26 ] reformulates NER as a generation task with explicit boundary markers and self-verification, achieving strong performance particularly in low-resource regimes. In privacy settings, Cory et al. (2026) [ 4 ] present a modular LLM-based pipeline for fine-grained, word-level annotation of GDPR transparency requirements in privacy policies, combining passage-level classification, retrieval-augmented generation, and self-correction. Zeng et al. [ 29 ] similarly propose a staged LLM-driven pipeline

Read the complete text