Reconciling Privacy and Explainability in High-Stakes: A Systematic Inquiry

Supriya Manna [email protected], Niladri Sett [email protected] · 2024

Read Reconciling Privacy and Explainability in High-Stakes: A Systematic Inquiry by Supriya Manna [email protected], Niladri Sett [email protected] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20798.txt.

Description

A 2024 arXiv research paper by Supriya Manna [email protected], Niladri Sett [email protected]. Deep learning’s preponderance across scientific domains has reshaped high-stakes decision-making, making it essential to follow rigorous operational frameworks that include both Right-to-Privacy (RTP) and Right-to-Explanation (RTE). This paper examines the complexities of combining these two requirements. For RTP, we focus on ‘Differential privacy’ (DP), which is considered the current gold standard for privacy- preserving machine learning due to its strong quantitative guarantee of privacy.

Catalog record

CREATORSupriya Manna [email protected], Niladri Sett [email protected]
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20798

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reconciling Privacy and Explainability in High-Stakes: A Systematic Inquiry Supriya Manna [email protected], Niladri Sett [email protected] Abstract Deep learning’s preponderance across scientific domains has reshaped high-stakes decision-making, making it essential to follow rigorous operational frameworks that include both Right-to-Privacy (RTP) and Right-to-Explanation (RTE). This paper examines the complexities of combining these two requirements. For RTP, we focus on ‘Differential privacy’ (DP), which is considered the current gold standard for privacy- preserving machine learning due to its strong quantitative guarantee of privacy. For RTE, we focus on post-hoc explainers: they are the go-to option for model auditing as they operate independently of model training. We formally investigate DP models and various commonly-used post-hoc explainers: how to evaluate these explainers subject to RTP, and analyze the intrinsic interactions between DP models and these explainers. Furthermore, our work throws light on how RTP and RTE can be effectively combined in high-stakes applications. Our study concludes by outlining an industrial software pipeline, with the example of a wildly used use-case, that respects both RTP and RTE requirements. 1 Introduction Deep learning has achieved massive success over the past decade in several domains and high stakes ( Dong et al., 2021 ) . It has been a cornerstone ever since in almost all aspects of scientific discoveries. However, it is to be noted that deep learning, although predominant in today’s scientific understandings, comes with its inherent shortcomings ( Talaei Khoei et al., 2023 ; Saeed and Omlin, 2023 ; Boulemtafes et al., 2020 ) . In this paper, we shall investigate the intricacies of two such pivotal shortcomings: Privacy and Explainability . We shall thereafter substantiate our findings with a famous use case from previous studies. Firstly, the deep models, although achieve superior performance across domains, are inherently prone to sensitive data leakage ( Boulemtafes et al., 2020 ) . It has been extensively shown that the models can leak information about the data with which it has been trained. Even, especially in the biomedical domain, it has been shown that simple ‘linkage’ attacks can exploit anonymized electronic health records ( Sweeney, 2015 ) . Due to the inherently vulnerable nature of deep models, it is not only prone to severe privacy attacks including membership inference (MIA) ( Shokri et al., 2017 ; Hu et al., 2022 ) , model stealing ( Oliynyk et al., 2023 ) , model inversion ( Fredrikson et al., 2015 ; Wang et al., 2021 ; Veale et al., 2018 ) etc but also impose a threat on the applicability of deep learning in real-world applications. Secondly, the deep models are hard to explain ( Saeed and Omlin, 2023 ) . Neural nets are nonlinear systems, often try to learn the (approximate) distribution of the training data. However, once a model is deployed in an open-world setting ( Zhu et al., 2024 ) , there is no gold label to cross-check the prediction obtained as a result, the subtle notion of trust on the prediction is pivotal to move forward with the same. But as these models are excessively critical, how did the model come to the prediction is a non-trivial phenomenon ( Jentzen et al., 2023 ) . To tackle the first problem, researchers have developed several strategies for privacy-preserving machine learning ( Boulemtafes et al., 2020 ) . This includes homomorphic encryption ( Pulido-Gaytan et al., 2020 ) , PATE ( Papernot et al., 2018 ) , differential privacy (DP) ( Abadi et al., 2016 ) etc; depending on the setting and motivation we incorporate these methods to make the model robust against diverse privacy attacks. However, differential privacy (DP), over the years, has been established as a gold-standard for privacy-preserving machine learning ( Blanco-Justicia et al., 2022 ; Suriyakumar et al., 2021 ) mainly due to its worst case guarantee against any inference attacks ( Abadi et al., 2016 ; Dwork et al., 2014 ) . Furthermore, other types of privacy attacks are not as mature and/or successful as MIA is ( Rigaki and Garcia, 2023 ) . In this paper, we have specifically worked on DP models 1 1 1 Throughout the paper we use DP model(s)/explanation(s) and private model(s)/explanation(s) interchangeably. with diverse privacy guarantee s aka \epsilon ( Dwork et al., 2014 ) . On the other hand, to tackle the explainability problem, researchers have developed an array of methods to audit the model decisions. This includes (but not limited to) local and global explainability methods, inherently interpretable machines, etc ( Saeed and Omlin, 2023 ) . Among these, local post-hoc explainers have emerged as popular and widely adopted tools for model auditing in recent times due to their ‘plug-and-play’ nature ( Bhatt et al., 2020 ) . In this work, we employ five popular post-hoc explainers relevant to our use case to obtain explanations. Although privacy and explainability aspects have been vastly explored independently, there is little to no work incorporating them together. Despite the rapid advancements in AI, integrating privacy and explainability in high-stakes domains remains an unsolved and pressing challenge. Existing research has treated privacy and explainability as separate challenges, leaving a significant gap in understanding their interplay. This gap becomes critical in high-stakes applications where both are non-negotiable. Researchers in privacy-preserving machine learning have investigated several aspects of differential privacy such as the scalability of DP models ( Beltran et al., 2024 ) , their fairness ( Fioretto et al., 2022 ) , robustness ( Tursynbek et al., 2020 ) , privacy-utility trade-off ( Zheng et al., 2024 ) etc. On the other side, researchers have worked on different aspects of explainable AI (XAI) such as faithfulness ( Lyu et al., 2024 ) , robustness ( Alvarez-Melis and Jaakkola, 2018 ) , and the quality of explanations ( Zhou et al., 2021 ) to name a few. In this paper, we take the first comprehensive step toward bridging the divide between privacy-preserving models and explainable AI in high-stakes domains. We explore the unique challenges that emerge when attempting to integrate privacy and explainability aspects in high-stakes applications. We investigate the underlying causes of these challenges, examine the trade-offs involved, and discuss key considerations necessary for developing frameworks to successfully incorporate these two critical aspects effectively. Integrating privacy and explainability is non-trivial, but addressing these issues is crucial for ensuring trustworthy and effective AI systems in high-stakes. In this context, we argue that it is equally important to identify use cases where achieving ‘the best of both worlds’ is not just desirable but a requirement . For example, if the training data is open source or an interpretable model (e.g. decision trees) is considered for the study, our research may not be well-motivated or the findings are not ‘worthy’. This is why, for this paper, we have considered the well-established use-case of disease detection from chest x-ray ( Al-qaness et al., 2024 ) . The reason to choose the same is that medical records are (almost) always subject to privacy preservation and it is of paramount importance that, with explanations, the stakeholders including both physician and patients can trust the model predictions. Post-hoc explainers, which we use in this paper, have been extensively utilized for our chosen and similar use cases in past

Read the complete text