A Comparative Study of light-weight Language Models for PII Masking and their deployment for real conversational texts

Prabigya Acharya, Liza Shrestha · 2025

Read A Comparative Study of light-weight Language Models for PII Masking and their deployment for real conversational texts by Prabigya Acharya, Liza Shrestha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18608.txt.

Introduction

Prabigya Acharya and Liza Shrestha ask whether small language models can mask personal information in conversation as well as giant frontier models do. Masking such details is essential for private chat systems. Frontier models do it well but raise data handling worries and cost far more. The pair fine tune two lightweight models, T5-small and Mistral-Instruct-v0.3, on AI4Privacy benchmark data.

Their datasets cover 24 standardized categories in several label variants, and both small models match frontier level performance. Normalizing labels improves results for every architecture. Mistral scores higher on F1 and recall and is more robust across information types, but it generates answers slowly. T5 is less robust on messy chat text, yet its outputs are structured and cheap to run. That tradeoff led the authors to deploy T5 in a live Discord bot, where tests on real messages show performance drops on informal input.

Catalog record

CREATORPrabigya Acharya, Liza Shrestha
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Comparative Study of light-weight Language Models for PII Masking and their deployment for real conversational texts Prabigya Acharya, Liza Shrestha Abstract Automated masking of Personally Identifiable Information (PII) is critical for privacy-preserving conversational systems. While current frontier large language models demonstrate strong PII masking capabilities, concerns about data handling and computational costs motivate exploration of whether lightweight models can achieve comparable performance. We compare encoder–decoder and decoder-only architectures by fine-tuning T5-small and Mistral-Instruct-v0.3 on English datasets constructed from the AI4Privacy benchmark. We create different dataset variants to study label standardization and PII representation, covering 24 standardized PII categories and higher-granularity settings. Evaluation using entity-level and character-level metrics, type accuracy, and exact match shows that both lightweight models achieve performance comparable to frontier LLMs for PII masking tasks. Label normalization consistently improves performance across architectures. Mistral achieves higher F1 and recall with greater robustness across PII types but incurs significantly higher generation latency. T5, while less robust in conversational text, offers more controllable structured outputs and lower inference cost, motivating its use in a real-time Discord bot for real-world PII redaction. Evaluation on live messages reveals performance degradation under informal inputs. These results clarify trade-offs between accuracy, robustness, and computational efficiency, demonstrating that lightweight models can provide effective PII masking while addressing data handling concerns associated with frontier LLMs. Keywords PII Masking, Large Language Models, Fine-tuning, Data Privacy, ROUGE Evaluation 1 Introduction Large language models (LLMs) have revolutionized natural language processing tasks like text generation, summarization, and question answering across healthcare, finance, and legal domains. However, user prompts often embed personally identifiable information (PII) [ 21 , 12 ] . PII refers to any data that may be used to directly or indirectly identify an individual, such as names, addresses, social security numbers, phone numbers, and financial and health records. As data handled increases, accurate and scalable redaction of sensitive information is non-negotiable specially as mishandling personal data can have serious legal, ethical, and financial consequences [ 8 ] . For example, HIPAA prohibits sharing electronic health records (EHRs) for research unless they are properly de-identified [ 25 ] . Hence, PII masking is required to protect user privacy, ensure regulatory compliance, and prevent unintended data leakage during model training and inference. Prior studies have shown that language models can memorize and reproduce sensitive information present in their training data, posing serious privacy risks when deployed in real world applications [ 4 , 3 ] . In interactive settings, LLMs may inadvertently expose personal details through logs, prompts, or generated outputs, particularly when handling conversational, medical, or customer service data [ 8 , 12 ] . As LLMs are increasingly integrated into high-stakes decision making pipelines, robust and automated PII masking becomes a critical safeguard for ethical, legal, and trustworthy AI deployment. Traditional PII redaction methods, including regex patterns and statistical matching, offer speed but falter on contextual nuances essential for accurate masking across varied texts. These rigid tools struggle with format variations like differing phone number conventions between regions limiting scalability for global, multilingual applications. Commercial APIs from AWS Comprehend [ 2 ] , Microsoft Presidio [ 16 ] , and Google Cloud [ 9 ] have boosted accuracy with deep learning, yet remain black-box solutions requiring data transmission to external providers. This poses compliance risks under GDPR or HIPAA for sensitive sectors. Open-source LLMs like Mistral and T5 address these issues through inherent contextual awareness, enabling fine-tuned redaction that generalizes broadly while supporting self-hosted deployment for full control and transparency. This study addresses the following research questions: 1. How do encoder-decoder models (T5) compare with decoder-only LLMs (Mistral) in PII masking accuracy? 2. How robust are these models across different PII types in conversational text? 3. What trade-offs exist between model performance, output controllability, and computational cost? To this end, we make the following contributions: • We present a systematic evaluation of encoder-decoder and decoder-only architectures for PII masking under controlled dataset variants. We fine-tune a T5 and Mistral models on 3 datasets, with varying level of control in input data and target labels. • We analyze the impact of dataset curation strategies using strict entity-level and character-level metrics. Our evaluation reveals how canonical normalization, regex-based replacement of unmasked PII, and removal of irrelevant PII candidates differentially affect robustness, boundary detection, and type accuracy. • We evaluate model performance on real world data using a discord bot to read and censor PII-containing messages. 1.1 Related Works 1.1.1 PII masking Personally Identifiable Information (PII) is any information about an individual maintained by an agency, including any information that can be used to distinguish or trace an individual’s identity, such as name, social security number, date and place of birth, mother’s maiden name, or biometric records; and any other information that is linked or linkable to an individual, such as medical, educational, financial, and employment information [ 15 ] . PII Masking is the systematic process of identifying these sensitive elements and applying transformations to hide the original values while attempting to preserve the data’s utility for downstream applications [ 5 ] . To achieve PII masking, several methods have evolved, transitioning from static pattern matching to context-aware generative models. I. Regex Based Methods Regular expressions(Regex) provide deterministic pattern matching for structured PII. Regex has been employed for general Named Entity Recognition tasks like automated resume parsing and high-risk PII detection. However, this approach suffers from high false positives and performs poorly on unstructured formats [ 20 ] . II. Named Entity Recognition (NER) Named entities are real-world objects represented by proper names or identities of interest. Named Entity Recognition (NER) and classification is essentially a sequence labeling task, where, given a sequence of tokens, a system aims to assign appropriate labels (NE classes) to each token [ 7 ] . However, traditional NER approaches face several limitations in accurately identifying personally identifiable information (PII) due to the sensitivity and complexity of the content. Challenges include ambiguity, polysemy, context-dependence or domain-dependence, and variations in phrasing such as nicknames, aliases, informal expressions, alternative representations, emerging terms, evolving naming conventions, syntax variations, typos, and misspellings [ 22 ] . Natural language processing platforms like OpenNLP and spaCy provide NER toolkits for identifying PIIs. Among the two, spaCy, a Python-based industrial-strength NLP library, outperforms OpenNLP on F-measure and recall for IME report de-identification [ 18 ] . III. Transformer-based Approaches The transformer architecture, replaces recurrent layers with stacked self-attention and point-wise feed-forward networks, enabling parallelization and capturing long-range dependencies without sequential processing [ 26 ] . • Encoder-Only Models Encoder-only architectures, exemplified by BERT [ 6 ] and its successors like RoBERTa [ 14 ] and DeBERTa [ 10 ] , utilize bidirectional self-attention to generate context-aware representations for every token along with a few modifications. In the context of PII, these models treat masking as a sequence labeling or token classification task. Within this paradigm, DeBERTa v3 has emerged as a high-performance baseline for identifying sensitive entities due to its disentangled attention mechanism. This architectural innovation represents word content and relative positions using separate vectors, allowing the model to more effectively capture the structural nuances of language such as the relationship between a name and its surrounding context that are critical for

Read the complete text