When FinTech Meets Privacy: Securing Financial LLMs with Differential Private Fine-Tuning

Sichen Zhu 1 , Hoyeung Leung 1 , Xiaoyi Wang 2 , Jia Wei 3 , Honghui Xu 3 4 · 2025

Read When FinTech Meets Privacy: Securing Financial LLMs with Differential Private Fine-Tuning by Sichen Zhu 1 , Hoyeung Leung 1 , Xiaoyi Wang 2 , Jia Wei 3 , Honghui Xu 3 4 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-08995.txt.

Introduction

Sichen Zhu and 4 co-authors secure financial language models with differential private fine-tuning in a 2025 paper. Finance firms use large models on sensitive data, increasingly on edge devices. Privacy breaches there carry high cost, yet performance must stay strong.

Their DPFinLLM combines a strong differential privacy mechanism, which limits what training data can leak, with a streamlined model design. It is built for on-device financial use. Tests on several sets of financial sentiment data show scores close to those of fully fine-tuned models, even when privacy limits are strict. The authors present it as a practical path for safe, efficient finance models on local hardware.

Catalog record

CREATORSichen Zhu 1 , Hoyeung Leung 1 , Xiaoyi Wang 2 , Jia Wei 3 , Honghui Xu 3 4
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-08995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When FinTech Meets Privacy: Securing Financial LLMs with Differential Private Fine-Tuning Sichen Zhu 1 , Hoyeung Leung 1 , Xiaoyi Wang 2 , Jia Wei 3 , Honghui Xu 3 4 Abstract The integration of Large Language Models (LLMs) into financial technology (FinTech) has revolutionized the analysis and processing of complex financial data, driving advancements in real-time decision-making and analytics. With the growing trend of deploying AI models on edge devices for financial applications, ensuring the privacy of sensitive financial data has become a significant challenge. To address this, we propose DPFinLLM, a privacy-enhanced, lightweight LLM specifically designed for on-device financial applications. DPFinLLM combines a robust differential privacy mechanism with a streamlined architecture inspired by state-of-the-art models, enabling secure and efficient processing of financial data. This proposed DPFinLLM can not only safeguard user data from privacy breaches but also ensure high performance across diverse financial tasks. Extensive experiments on multiple financial sentiment datasets validate the effectiveness of DPFinLLM, demonstrating its ability to achieve performance comparable to fully fine-tuned models, even under strict privacy constraints. I Introduction The proliferation of LLMs has revolutionized natural language understanding and generation, driving significant advancements in the FinTech sector. These models excel in processing complex financial data, enabling applications such as sentiment analysis [ 1 ] , risk management [ 2 ] , fraud detection [ 3 ] , and credit scoring [ 4 ] . By providing actionable insights and facilitating real-time decision-making, LLMs have become indispensable tools in modern financial services, empowering FinTech solutions to deliver smarter, faster, and more secure financial operations [ 5 ] . Recently, there has been a growing trend toward deploying AI models on edge devices for financial applications [ 6 ] . In light of this, on-device financial LLMs will offer several advantages, including real-time data processing, reduced dependency on cloud services, and enhanced user data privacy—critical components in the evolving FinTech landscape. However, this trend also introduces significant challenges. On-device models [ 7 , 8 , 9 , 10 , 11 ] must achieve a delicate balance between computational efficiency, privacy protection, and task performance, making their design and training particularly complex. Addressing these challenges is essential to advancing the integration of LLMs into secure, efficient, and innovative FinTech ecosystems. The sensitive nature of financial data amplifies the challenges associated with deploying on-device financial LLMs. Membership inference attacks [ 12 ] and model inversion attacks [ 13 ] on these AI models pose significant risks, enabling unauthorized access to private financial information. While existing differential privacy techniques have been explored across three key phases, input dataset preparation [ 14 ] , model training [ 15 ] , and model output generation [ 12 ] , their application to on-device financial LLMs remains under-researched. Bridging this gap is crucial to ensuring the secure and effective deployment of financial LLMs on edge devices, safeguarding sensitive data while maintaining financial LLMs’ performance. To address these challenges, we introduce DPFinLLM, a novel on-device financial large language model that integrates a lightweight architectural design with a robust differential privacy mechanism. DPFinLLM employs a privacy-enhanced training pipeline to safeguard sensitive financial data while maintaining high performance across various financial tasks. Drawing inspiration from state-of-the-art models like Llama2 and ChatGLM2, DPFinLLM features a streamlined architecture optimized for edge devices. By incorporating Low-Rank Adaptation (LoRA) for fine-tuning, the model achieves computational efficiency and supports task-specific optimization with minimal resource requirements. The key contributions of this paper are summarized as follows: • We propose DPFinLLM, a privacy-enhanced on-device financial LLM that integrates differential privacy techniques with a lightweight architectural design tailored for edge devices. • A robust differential privacy mechanism is incorporated into the fine-tuning process to protect sensitive financial data from privacy breaches. • Comprehensive experiments on multiple financial sentiment datasets validate the effectiveness of DPFinLLM, demonstrating performance comparable to baseline models even under strict privacy constraints. The remainder of this paper is structured as follows: Section II reviews existing research on financial LLMs and privacy-preserving techniques. Section III details the architectural design and privacy-preserving training framework of DPFinLLM. Section IV presents the experimental setup and results, and Section V concludes with insights and directions for future research. II Related Works This section will conclude the related work of financial language models and review the current mainstream privacy-preserving learning approaches. II-A Financial Large Language Models Recently, financial language models have demonstrated remarkable capabilities in handling complex tasks in the financial sector, including sentiment analysis [ 16 ] , risk management [ 2 ] , financial fraud detection [ 3 ] , and credit scoring [ 4 ] . By leveraging their predictive power, users in the economic domain can make more informed decisions, enabling better planning and strategic execution. Modern fine-tuned LLMs, such as FinGPT [ 17 ] and FinBERT [ 18 ] , have been specifically developed to handle natural language processing tasks in financial datasets, showcasing exceptional domain-specific adaptability. These models excel at processing both structured and unstructured data [ 19 ] from various sources, including APIs, web scraping tools, and direct database access. Their ability to integrate real-time data streams into model training provides outputs that reflect the most current market conditions or issue statuses, offering users actionable insights in a dynamic financial landscape [ 20 ] . As financial language models continue to evolve, a notable trend is emerging toward the development of on-device financial LLMs. However, designing on-device financial LLMs presents considerable challenges. Unlike cloud-based models, which can leverage extensive computational resources, on-device models must operate within the constraints of limited processing power, memory, and energy. Overcoming these hurdles will be crucial to unlocking the full potential of on-device financial LLMs and transforming the way financial insights are generated and utilized. II-B Privacy-Preserving Learning Mechanisms Privacy-preserving approaches address critical AI cybersecurity challenges by implementing mechanisms across three key phases: input dataset preparation, model training, and model output generation. (1) In the input dataset preparation phase, privacy is safeguarded by injecting noise (e.g., Gaussian or Laplace) into dataset features [ 21 , 14 ] , replacing sensitive text with anonymous tokens or artificial labels, and encrypting data to minimize leakage risks [ 22 ] . (2) During the model training process, some techniques ensure privacy by clipping gradients to limit individual data point influence and adding Gaussian noise to gradients for differential privacy [ 15 , 23 ] . (3) In the model output generation phase, methods introduce noise to prompts or outputs to protect sensitive information, followed by coherence refinement to maintain utility. Additionally, output filtering can replace sensitive terms with anonymized equivalents to prevent disclosure [ 12 ] . While these strategies effectively address privacy concerns in traditional large language models, there is a notable lack of research investigating data privacy mechanisms specifically for on-device financial LLMs. Developing privacy-preserving solutions tailored for on-device models is critical, as such systems face unique challenges, including constrained computational resources and heightened sensitivity to data privacy breaches. In this paper, we propose an on-device differential privacy-enhanced financial LLM (called DPFinLLM), tackling two critical challenges: designing a lightweight architecture suitable for resource-constrained edge devices and incorporating a differential privacy mechanism during model training to protect users’ sensitive data effectively. III DPFinLLM The proposed

Read the complete text