Voice Privacy Preservation with Multiple Random Orthogonal Secret Keys: Attack Resistance Analysis
\authorblockN Kohei Tanaka \authorrefmark 1, Hitoshi Kiya \authorrefmark 1 and Sayaka Shiota \authorrefmark 1 \authorblockA \authorrefmark 1 Tokyo Metropolitan University, Japan · 2025
Read Voice Privacy Preservation with Multiple Random Orthogonal Secret Keys: Attack Resistance Analysis by \authorblockN Kohei Tanaka \authorrefmark 1, Hitoshi Kiya \authorrefmark 1 and Sayaka Shiota \authorrefmark 1 \authorblockA \authorrefmark 1 Tokyo Metropolitan University, Japan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19906.txt.
Introduction
Sending speech to cloud models exposes both what was said and who said it. Kohei Tanaka, Hitoshi Kiya, and Sayaka Shiota strengthen an encryption scheme that hides both. The base method scrambles speech with a secret random orthogonal key, letting cloud models work on the scrambled signal. Its weakness was limited resistance to attack and a narrow range of usable models.
Using several random orthogonal keys at once hardens the scheme. The authors also relax the constraints on which models can accept the encrypted input. Attack resistance is analyzed directly. Privacy-preserving inference would let hospitals and firms use cloud speech tools without leaking voices. Each relaxation of model limits widens where the method can actually be deployed.
Catalog record
| CREATOR | \authorblockN Kohei Tanaka \authorrefmark 1, Hitoshi Kiya \authorrefmark 1 and Sayaka Shiota \authorrefmark 1 \authorblockA \authorrefmark 1 Tokyo Metropolitan University, Japan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-19906
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Voice Privacy Preservation with Multiple Random Orthogonal Secret Keys: Attack Resistance Analysis \authorblockN Kohei Tanaka \authorrefmark 1, Hitoshi Kiya \authorrefmark 1 and Sayaka Shiota \authorrefmark 1 \authorblockA \authorrefmark 1 Tokyo Metropolitan University, Japan Abstract Recently, opportunities to transmit speech data to deep learning models executed in the cloud have increased. This has led to growing concerns about speech privacy, including both speaker-specific information and the linguistic content of utterances. As an approach to preserving speech privacy, a speech privacy-preserving method based on encryption using a secret key with a random orthogonal matrix has been proposed. This method enables cloud-based model inference while concealing both the speech content and the speaker identity. However, the method has limited attack resistance and is constrained in terms of the deep learning models to which the encryption can be applied. In this work, we propose a method that enhances the attack resistance of the conventional speech privacy-preserving technique by employing multiple random orthogonal matrices as secret keys. We also introduce approaches to relax the model constraints, enabling the application of our method to a broader range of deep learning models. Furthermore, we investigate the robustness of the proposed method against attacks using extended attack scenarios based on the scenarios employed in the Voice Privacy Challenge. Our experimental results confirmed that the proposed method maintains privacy protection performance for speaker concealment, even under more powerful attack scenarios not considered in prior work. 1 Introduction Recently, deep learning-based speech-processing systems have commonly been used with smartphones, home assistants, and other edge devices. Such devices have strict computational resource constraints that make it challenging to run inference locally. To address this limitation, the speech-processing systems on these edge devices are typically implemented by offloading the computations of deep learning models to cloud servers. However, the cloud-based systems require sharing speech data with cloud servers. Speech data contains not only the utterance content but also personal identifying information, including language, age, gender, and speaker identity [ 1 , 2 ] . The personal information embedded in speech data raises significant global privacy concerns. Therefore, under the General Data Protection Regulation (GDPR), an international data protection law, audio data is subject to privacy regulations [ 3 ] . Against this background, speech privacy issues have received significant focus [ 4 , 5 , 6 , 7 , 8 ] . For example, the Voice Privacy Challenge (VPC), an international competition aimed at advancing speech anonymization technologies, serves as a key initiative within these ongoing efforts [ 9 , 10 , 11 ] . The major speaker anonymization approaches involve voice conversion-based techniques, which are employed as the core technology in many methods submitted to VPC [ 12 , 13 , 14 ] . Such speech anonymization methods aim to conceal the speaker identity while preserving the linguistic content in an utterance. An encryption-based speech privacy-preserving method with a random orthogonal matrix as a secret key, distinct from the voice conversion-based methods, has been proposed to protect both the speaker identity information and the linguistic content of an utterance [ 15 ] . This method enables deep learning model inference while concealing both speaker identity and speech content. However, this method has limited resistance to inference attacks on speech content and speaker identity. It also cannot be applied to mainstream speech-processing self-supervised learning (SSL) models due to restrictions on compatible model architectures. Furthermore, existing evaluations have only considered attackers without access to encryption systems, indicating that more sophisticated adversaries are needed for comprehensive assessments of speech privacy. In this work, we propose a method that enhances the attack resistance of the conventional encryption-based speech privacy-preserving technique by employing multiple random orthogonal matrices as secret keys. We also introduce an approach to relax the model constraints of the conventional method. This approach enables the application of our method to a broader range of deep learning models, including mainstream SSL models. Furthermore, we investigate the robustness of the proposed method against attacks by proposing extended attack scenarios. These scenarios include assessments against sophisticated adversaries with access to the encryption system, inspired by the VPC attack scenario. The experimental results confirmed that the proposed method can be applied to automatic speech recognition (ASR) and automatic speaker verification (ASV) models that use a widely used SSL model as their frontend. Furthermore, we found that the proposed method maintains attack resistance, particularly for speaker identity concealment, even under these more sophisticated attack scenarios. The contributions of our work are as follows: 1. Enhanced attack resistance : We propose a method that employs multiple random orthogonal matrices as secret keys to enhance the attack resistance of the conventional encryption-based speech privacy-preserving method. 2. Broader model applicability : The proposed method also includes a function that relaxes the model constraints of the conventional method. The function enables the proposed method to adapt to a broader range of deep learning models, including mainstream SSL models, without requiring retraining. 3. Extended evaluation framework : We advocate comprehensive evaluations using two attack scenarios extended from the scenario employed in VPC, including assessments against sophisticated adversaries with access to the encryption system. 2 Privacy-Preserving Scenario Figure 1: Privacy-Preserving Scenario of the encryption-based methods. The model owner trains a model on unencrypted speech data, encrypts the model using a secret key, and deploys the encrypted model to the cloud server. Users encrypt their speech queries using the shared secret key before transmission. The cloud performs inference on encrypted queries using the encrypted model, ensuring speech privacy without decryption. This section explains the privacy protection scenarios of the encryption-based methods. Figure 1 shows an overview of the scenario, which consists of three domains: model owner, user, and cloud server. The cloud-based model performs inference on the encrypted speech to obtain results on speech processing tasks while protecting user speech privacy. The workflow operates as follows: The model owner trains the model using unencrypted speech datasets, encrypts the model using a secret key, and deploys it to the cloud. The secret key is then shared with an authorized user. The user encrypts query speech using the shared secret key and transmits it to the cloud. The cloud performs inference using the encrypted model without decrypting the query, thereby preventing third parties from accessing the original speech without the secret key. This approach ensures user privacy by making it difficult to infer information from the encrypted queries. 3 Proposed method This section describes the proposed method for enhancing the attack resistance of the conventional encryption-based method and enabling its application to a broader range of deep learning models without retraining. 3.1 Multiple random orthogonal matrices as secret keys for speech privacy preservation To enhance the attack resistance of the conventional method, we propose an encryption-based technique that utilizes multiple random orthogonal matrices as secret keys. This approach encompasses the conventional method in which only one matrix is used as a secret key. The following sections explain the speech encryption and model encryption processes of the proposed method in the scenario described in Section 2 . 3.1.1 Speech encryption Speech encryption is performed by dividing a one-dimensional speech waveform \bm{X} into blocks \bm{X}_{i} of size M and multiplying each block \bm{X}_{i} by an M -dimensional random orthogonal