Privacy Amplification Persists under Unlimited Synthetic Data Release Clément Pierquin, Aurélien Bellet, Marc Tommasi, Matthieu Boussard Abstract We study privacy amplification by synthetic data release , a phenomenon in which differential privacy guarantees are improved by releasing only synthetic data rather than the private generative model itself. Recent work by 40 established the first formal amplification guarantees for a linear generator, but they apply only in asymptotic regimes where the model dimension far exceeds the number of released synthetic records, limiting their practical relevance. In this work, we show a surprising result: under a bounded-parameter assumption, privacy amplification persists even when releasing an unbounded number of synthetic records , thereby improving upon the bounds of 40 . Our analysis provides structural insights that may guide the development of tighter privacy guarantees for more complex release mechanisms. 1 Introduction Differential privacy (DP) ( 18 ) provides a rigorous framework for protecting sensitive data in machine learning. Privacy-preserving models can be obtained through methods like output perturbation ( 14 ; 53 ; 34 ) , which adds noise to a trained model, or noisy gradient descent algorithms, which inject noise during training ( 46 ; 7 ; 1 ; 19 ) . A key property of DP is post-processing : once a model is differentially private, any further processing or analysis preserves its guarantees. This makes differentially private generative models ( 54 ; 51 ; 37 ; 27 ; 36 ; 31 ; 15 ; 9 ) particularly appealing, as the synthetic data they generate automatically inherits the model’s privacy guarantees, allowing synthetic datasets to be shared safely for downstream use, in contrast to heuristic approaches that may not provide meaningful privacy protection ( 55 ; 42 ) . Empirical evidence however suggests that synthetic data can sometimes provide stronger privacy than the formal guarantees of the underlying generative model ( 4 ; 25 ) , hinting at a possible privacy amplification effect. One intuitive explanation is that privacy leakage may be lower when the number of synthetic samples is small relative to the complexity of the generative model. This idea was recently formalized by 40 , who established the first formal privacy amplification guarantees via synthetic data release in the setting of a linear generator privatized via output perturbation. Their analysis attributes the amplification effect to the latent randomness in the data generation process. However, these guarantees only hold in a highly restrictive regime where the number n_{\mathrm{syn}} of synthetic records is small compared to the model dimension d . Furthermore, the privacy bound of 40 is expressed as a piecewise minimum between a standard post-processing guarantee and an amplification term. This piecewise formulation does not provide a unified view of amplification and post-processing effects. Overall, it remains unclear whether a single, non-asymptotic guarantee can be established that holds even for large n_{\mathrm{syn}} . In this paper, we substantially strengthen and extend these prior results by developing a unified framework for privacy amplification in synthetic data released by a linear generator. Our analysis identifies and precisely characterizes the mechanisms driving privacy amplification in this tractable model, offering insights that may guide the study of more general synthetic data release scenarios. Specifically, our main contributions are as follows: • We show that releasing infinitely many synthetic records (n_{\mathrm{syn}}\to+\infty) can still yield significant privacy amplification —a surprising result that holds under a bounded-parameter assumption. We propose a unified characterization of privacy in the high-privacy regime, and also derive a general privacy amplification upper bound. • To establish these results, we develop new technical tools based on Fisher information. Specifically, we introduce a criterion that allows controlling Rényi divergences via Fisher information, and we derive upper bounds on the Fisher information for the non-centrality parameter of non-central chi-squared and non-central Wishart distributions. These results are more general and unified than existing analyses, and may be of independent interest to the statistics community. • To complement our theoretical results, we numerically estimate the Rényi DP guarantees and compare them with our theoretical bounds across different parameters. Strikingly, we find that the privacy guarantees converge rapidly to their n_{\mathrm{syn}}\to+\infty limits, showing that focusing on this asymptotic regime captures the key amplification effect with little loss for practical amounts of synthetic data. All proofs and additional discussions are provided in the Appendix. Before presenting our main results in more details, we highlight two unexpected and conceptually important insights that emerged from our analysis. First, we invalidate the natural intuition , put forward by 40 , that releasing an unbounded number of synthetic records would eliminate any privacy benefit . According to this intuition, an adversary could, in principle, exploit central limit effects to detect even the smallest distributional shifts and reconstruct the private generative model with arbitrary precision. Our analysis shows this is not the case: the privacy loss can be effectively upper-bounded without restricting the number of released synthetic records. This yields a privacy accounting rule that, even for arbitrarily large numbers of synthetic records, is strictly tighter than the standard post-processing privacy guarantee obtained when releasing the model parameters. The second insight concerns a necessary condition for the above phenomenon: boundedness of the generative model parameters . Indeed, without restricting the parameter space, we show that releasing infinitely many records yields no amplification. This boundedness condition is mild in our setting, since the linear generator can, for example, be trained with \ell_{2} -regularization, which naturally enforces a bounded parameter norm (for example, it holds for \ell_{2} -regularized linear regression). More generally, a bounded parameter domain is a standard assumption in theoretical analyses of private optimization ( 7 ; 19 ; 3 ) , and is often required to ensure a Lipschitz objective in machine learning. Interestingly, in our setting, bounding the parameters serves a deeper purpose: it prevents an adversary from exploiting the norm of the released data to mount an attack whose leakage would match the post-processing guarantees. 2 Setting and Overview of our Results Rényi Differential Privacy. In this work, we express privacy guarantees using Rényi Differential Privacy (RDP), a variant of differential privacy that quantifies privacy in terms of the Rényi divergence ( 38 ) . Two datasets \mathcal{D} and \mathcal{D}^{\prime} of fixed size n are called adjacent if they differ in a single data point ( 38 ) . Then, a mechanism \mathcal{M} satisfies (\alpha,\varepsilon) -RDP if, for any adjacent datasets \mathcal{D} and \mathcal{D}^{\prime} , D_{\alpha}(\mathcal{M}(\mathcal{D})\|\mathcal{M}(\mathcal{D}^{\prime}))\leq\varepsilon, with D_{\alpha}(P,Q)=\frac{1}{\alpha-1}\log\mathbb{E}_{x\sim Q}[(P(x)/Q(x))^{\alpha}] the Rényi divergence of order \alpha>1 between distributions P and Q . By a slight abuse of notation, if V\sim P and W\sim Q , we write D_{\alpha}(V,W)=D_{\alpha}(P,Q) . The setting: synthetic data and linear generators. Many approaches for private synthetic data release are modeled as a two-stage randomized mechanism ( 51 ; 27 ; 31 ; 15 ) . In the first stage, a differentially private procedure \mathcal{M} outputs the parameters V=\mathcal{M}(\mathcal{D}) of a generative model trained on a sensitive dataset \mathcal{D} . In the second stage, the learned generator f_{V} draws a latent noise Z=(Z_{1},\dots Z_{n_{\text{syn}}}) from some (typically Gaussian) distribution \mu^{\otimes{n_{\text{syn}}}} and outputs n_{\text{syn}} synthetic records f_{V}(Z)=(f_{V}(Z_{1}),\dots,f_{V}(Z_{n_{\text{syn}}})) . Therefore, the privacy loss associated to synthetic data release is the Rényi divergence D_{\alpha}(f_{V}(Z),f_{W}(Z)) , which is upper-bounded by D_{\alpha}(V,W) by the post-processing property of DP. Privacy amplification by synthetic data release then corresponds to establishing a stronger bound of D_{\alpha}(f_{V}(Z),f_{W}(Z))\leq\eta D_{\alpha}(V,W) with \eta<1 , which can be attributed to the additional randomness introduced by Z . For realistic generators f_{V} (e.g. deep generative models), this divergence is intractable. Following ( 40 ) , we therefore study a tractable proxy: 1. We model f_{V} as a linear function f_{V}(Z)=ZV , where Z\in\mathbb{R}^{n_{\text{syn}}\times d} and V\in\mathbb{R}^{d\times k} with: • n_{\text{syn}} : the number of synthetic records, • k : the dimension of synthetic records, • d : the width of the model. Similar idealized model have been used to study model collapse in generative AI ( 16 ; 21 ) and learning dynamics of generative models ( 35 ; 20 ) . 2. We consider deterministic parameters v,w\in\mathbb{R}^{d\times k} obtained from adjacent datasets \mathcal{D} , \mathcal{D^{\prime}} via a bounded-sensitivity procedure, which are then privatized with output perturbation, i.e., by adding Gaussian noise. The bounded-sensitivity assumption is natural, and essentially without loss in the linear generator setting. For instance, under standard boundedness conditions, \ell_{2} -regularized multi-output linear regression has bounded sensitivity ( 14 ; 40 ) , meaning that \|v-w\|_{F}\leq\Delta for some sensitivity \Delta>0 that depends on the problem parameters. Formally, we consider the following synthetic data release mechanism. Definition 2.1 (Linear generation from Gaussian inputs; 40 ) . Let v\in\mathbb{R}^{d\times k} be a deterministic parameter. Let \mathcal{M}(v):=v+\sigma N , where N\in\mathbb{R}^{d\times k} has i.i.d. standard normal entries. Independently, draw a matrix Z\in\mathbb{R}^{n_{\mathrm{syn}}\times d} with i.i.d. entries Z_{ij}\sim\mathcal{N}(0,\sigma_{z}^{2}) . The released synthetic dataset is Z\mathcal{M}(v) . By the scale-invariance of Gaussian matrices, D_{\alpha}(Z(v+\sigma N),Z(w+\sigma N))=D_{\alpha}(Z(\frac{1}{\sigma}v+N),Z(\frac{1}{\sigma}w+N)) . Therefore, without loss of generality, we assume \sigma_{z}=\sigma=1 throughout the paper. We let the random variables V=\mathcal{M}(v) and W=\mathcal{M}(w) , for v,w\in\mathbb{R}^{d\times k} obtained from adjacent datasets. Assumption 2.2 (Non-degeneracy) . We assume that d\geq k . This ensures that the product ZV admits a density. Such an overparameterized regime is common in modern deep learning, where the number of model parameters typically exceeds the dimensionality of the output. Summary of our main result. We derive bounds on D_{\alpha}(ZV,ZW) , the privacy loss of releasing synthetic data, and compare them to D_{\alpha}(V,W) , the privacy loss of releasing the generative model parameters. Figure 1 summarizes our results and compares them to the upper bound of prior work by 40 . 1 1 1 The results of 40 were originally reported in the f -DP framework ( 17 ) . For comparison with our results, we converted them into upper bounds on Rényi divergences. Details are provided in Appendix B . We obtain a unified upper bound capturing both amplification and non-amplification regimes in a natural way. Strikingly, our upper bounds do not depend on the number of synthetic data points n_{\text{syn}} , relying only on the parameters v and w having Frobenius norm bounded by C . Prior work ( 40 ) : D_{\alpha}(ZV,ZW)\lesssim\min\left\{1,k\sqrt{\frac{n_{\text{syn}}}{d-k}}\right\}D_{\alpha}(V,W). Our work: D_{\alpha}(ZV,ZW)\lesssim\frac{C^{2}}{d/k+C^{2}}D_{\alpha}(V,W). Figure 1: Summary of our results highlighting the improvements over prior work by 40 . 3 Relationship between Fisher information and Rényi divergences This section presents some key technical results (some of which novel) that we will use to address our main goal: upper bounding the privacy loss induced in the infinite synthetic data release regime ( n_{\text{syn}}\rightarrow+\infty ). Let v,w\in\mathbb{R}^{d\times k} be the parameter produced by two adjacent datasets, with \|v-w\|_{F}\leq\Delta , and P_{v} and P_{w} the corresponding infinite release distributions of ZV and ZW . Unfortunately, as we will see in Sections 5.1 and 5.2 , the Rényi divergence D_{\alpha}(P_{v},P_{w}) is generally intractable and does not admit simple bounds. Instead of attempting a direct computation, we take an indirect approach. We embed the parameters v and w into a smooth parametric family \{P_{\theta},\theta\in\Theta\} that interpolate between P_{v} and P_{w} , and we study the Fisher information of this family with respect to \theta . While still not directly computable, this quantity is typically more tractable to bound than the Rényi divergence itself. Then, we leverage a relationship between Rényi divergences and the Fisher information. Rényi divergences admit a local second-order expansion governed by Fisher information. Proposition 3.1 (Local relationship between Rényi divergences and Fisher information; 24 ; 48 ) . Let \alpha>1 . Let P=\{P_{\theta},\;\theta\in\Theta\subset\mathbb{R}\} be a family of probability distributions, with associated densities p_{\theta} . For \theta\in\Theta , let the Fisher information I(\theta)=\mathbb{E}_{X\sim P_{\theta}}[(\partial_{\theta}\log p_{\theta}(X))^{2}] and \Delta>0 . Then, under some regularity conditions (see Definition C.1 ), D_{\alpha}(P_{\theta+\Delta},P_{\theta})=\frac{\alpha}{2}I(\theta)\Delta^{2}+o(\Delta^{2}). The above proposition establishes an approximation of the Rényi divergence D_{\alpha}(P_{\theta+\Delta},P_{\theta}) in terms of the Fisher information I(\theta) . This result is particularly useful for families of distributions in which the Fisher information is substantially easier to compute than the Rényi divergence. However, this approximation is local, in the sense that it holds when the sensitivity \Delta is small, and, to our knowledge, does not extend to global relationships between Rényi divergence and Fisher information. An existing attempt by 2 contains a flaw in its proof; see Appendix C.3 , where we also clarify in Appendix C.3 why global comparison statements based on pointwise Rényi divergences and Fisher information cannot hold in general, and we give a basic counterexample. For privacy certification, one requires non-asymptotic upper bounds. In this work, we proceed by introducing an upper envelope U on the Rényi divergence D_{2\alpha-1}(P_{z},P_{\theta}) along a path: D_{2\alpha-1}(P_{z},P_{\theta})\leq U(z,\theta) . Then, we show that for any such choice of U , a uniform bound on the Fisher information along the path yields a upper bound on Rényi divergence D_{\alpha}(\theta^{\prime},\theta) . In particular, whenever U is tractable, this provides a closed-form upper bound for D_{\alpha}(\theta^{\prime},\theta) . Proposition 3.2 (Upper bounding Rényi divergences through Fisher information) . Let \theta<\theta^{\prime}\in\Theta and \alpha>1 . We assume that there exists a function U:\mathbb{R}^{2}\to\mathbb{R^{+}} such that for all z\in(\theta,\theta^{\prime}) , D_{2\alpha-1}(P_{z},P_{\theta})\leq U(z,\theta) . We denote: \displaystyle D_{\theta}^{\theta^{\prime}}=\int_{\theta}^{\theta^{\prime}}e^{(\alpha-1)U(z,\theta)}dz. Then, under some regularity conditions (see Definition C.1 ): \displaystyle D_{\alpha}(P_{\theta^{\prime}},P_{\theta})\leq\frac{1}{\alpha-1}\log\bigg(1+\alpha\sup_{z\in(\theta,\theta^{\prime})}I(z)^{1/2}D_{\theta}^{\theta^{\prime}}\bigg). A discussion about the tightness of this upper bound can be found in Appendix C.4 . We note that related considerations and proof techniques appear in 28 to bound the rates of several statistical divergences under temporal evolution of the distributions. In contrast, Proposition 3.2 yields a global bound after integrating along a parameter path, provided a computable upper envelope U of D_{2\alpha-1} is available. In our setting, this condition follows from the post-processing inequality. In fact, as synthetic data is a randomized function of the parameters, we have D_{2\alpha-1}(P_{v},P_{w})\leq D_{2\alpha-1}(V,W) , yielding a closed-form upper envelope. This allows us to control the Rényi divergence in terms of the Fisher information. Crucially, if the Fisher information is small along a path, the upper bound forces D_{\alpha}(P_{\theta^{\prime}},P_{\theta}) to be small, which we will leverage to prove privacy amplification. In summary, we have obtained two complementary tools to characterize privacy amplification: • The local expansion in Proposition 3.1 will yield sharp privacy amplification heuristics in regimes where the sensitivity \Delta is small. • The global, non-asymptotic bound of Proposition 3.2 ensures rigorous guarantees. We will use it to show that privacy amplification holds for every value of \Delta . 4 From Outputs to Sufficient Gram Statistics As explained in Section 2 , our goal is to quantify the privacy loss D_{\alpha}(ZV,ZW) associated to synthetic data release, especially when n_{\mathrm{syn}} is large. In this section, we show that the test between ZV and ZW can be reduced to a test between V^{\top}V and W^{\top}W when n_{\mathrm{syn}}\to+\infty . We first show the following result. Proposition 4.1 . Assume n_{\mathrm{syn}}\geq d\geq k . Then, D_{\alpha}(ZV,ZW)=D_{\alpha}(V^{\top}Z^{\top}ZV,W^{\top}Z^{\top}ZW). The proof, provided in Appendix D.1 , relies on showing that the map T:X\mapsto X^{\top}X is a sufficient statistic for distinguishing between ZV and ZW . The matrix Z^{\top}Z follows a Wishart distribution \mathcal{W}(d,n_{\mathrm{syn}}) with shape d and degree of freedom n_{\mathrm{syn}} . Then, a careful analysis allows us to connect the divergence between ZV and ZW to the divergence between V^{\top}V and W^{\top}W in the asymptotic regime. Proposition 4.2 . Assume d\geq k . Then, D_{\alpha}(V^{\top}Z^{\top}ZV,W^{\top}Z^{\top}ZW)\overset{n_{\mathrm{syn}}\to\infty}{\longrightarrow}D_{\alpha}(V^{\top}V,W^{\top}W). Furthermore, for all n_{\mathrm{syn}}>1 , D_{\alpha}(ZV,ZW)\leq D_{\alpha}(V^{\top}V,W^{\top}W). This key proposition allows us to obtain privacy guarantees for any n_{\mathrm{syn}}\in\mathbb{N}^{*} by analyzing D_{\alpha}(V^{\top}V,W^{\top}W) . The limit depends on V,W only through the Gram matrices V^{\top}V and W^{\top}W . This reduction is the core technical simplification that unlocks closed-form upper bounds for the divergence. In particular D_{\alpha}(V^{\top}V,W^{\top}W) is the privacy loss associated to releasing an infinite amount of synthetic data. In other words, from a privacy perspective, the release of arbitrarily many synthetic data is equivalent to releasing the Gram matrix V^{\top}V . In the rest of the paper, we focus on deriving an upper bound for D_{\alpha}(V^{\top}V,W^{\top}W) . We first state a negative result: without additional assumptions, releasing infinitely many synthetic records provides no stronger privacy guarantees than releasing the generative model itself. Proposition 4.3 . (No worst-case amplification in the n_{\mathrm{syn}}\to+\infty regime). \sup_{v,w\in\mathbb{R}^{d\times k},\|v-w\|_{F}\leq\Delta}D_{\alpha}(V^{\top}V,W^{\top}W)=D_{\alpha}(V,W). The idea of the proof is to take v_{\theta}=te_{1},w_{t}=(t+\Delta)e_{1} , where e_{1}=(\delta_{11})\in\mathbb{R}^{d\times k} . As t tends to +\infty , the matrices V^{\top}V and W^{\top}W diverge in a way that allows an adversary to recover V and W . To rule out this pathological behavior, we henceforth assume that the parameters are bounded. Assumption 4.1 (Parameter space boundedness) . We assume that there exists C>0 such that for any parameters v,w obtained from adjacent datasets: \max\{\|v\|_{F},\|w\|_{F}\}\leq C. We note that parameter boundedness is a standard assumption in the theory of private learning algorithms such as Projected Noisy Gradient Descent ( 19 ; 3 ) . 5 Privacy Amplification in Linear Synthetic Data Generation In this section, we build on the results of Sections 3 and 4 to derive our main privacy amplification guarantees for synthetic data generation with a linear generator. 5.1 Releasing One-Dimensional Synthetic Data We begin by analyzing the simpler case k=1 , which corresponds to releasing one-dimensional synthetic data points. When k=1 , V^{\top}V=\|N+v\|^{2} and W^{\top}W=\|N+w\|^{2} are scalar-valued non-central chi-squared random variables with the same degree of freedom d and different non-centrality parameters \|v\|^{2},\|w\|^{2} . We denote this distribution by \chi_{d}^{2}(\theta^{2}) , where \theta^{2} is the non-centrality parameter. In particular, we have V^{\top}V\sim\chi_{d}^{2}(\|v\|^{2}) and W^{\top}W\sim\chi_{d}^{2}(\|w\|^{2}) . The probability density function of \chi_{d}^{2}(\|v\|^{2}) is given by: p_{v}(x)=p_{0}(x)e^{-\|v\|^{2}/2}\|v\|^{1-d/2}I_{d/2-1}\big(\|v\|\sqrt{x}\big), where I_{d/2-1} is the modified Bessel function of the first kind with index d/2-1 and p_{0} is the density of a central chi-squared distribution with degree of freedom d . We denote the family of non-central chi-squared distributions with respect to the amplitude parameter as P:=\{p_{\theta}=\chi_{d}^{2}(\theta^{2});\;\theta\in\mathbb{R^{+}}\}, and the Fisher information associated with this family as I(d,\theta):=\mathbb{E}_{Y\sim p_{\theta}}[(\partial_{\theta}\log p_{\theta})^{2}]. The Fisher information of non-central chi-squared distribution for the amplitude parameter is not available in closed form ( 26 ) . As a result, much of the literature has focused either on deriving lower bounds ( 12 ; 26 ; 47 ) or on numerical estimation methods ( 12 ) . The literature on upper bounds remains limited, aside from the work of 26 which proposes an upper bound for the Fisher information of generalized Rician distributions (more details in Appendix E.1 ). In this section, we tighten the upper bound of 26 . Proposition 5.1 . (Fisher information of non-central \chi^{2} ). For all \theta>0,d>2 and p_{\theta}\in P , and : \frac{2\theta^{2}}{2\theta^{2}+d}\leq I(d,\theta)\leq\frac{2\theta^{2}}{2\theta^{2}+d-3}. This upper bound nearly matches the lower bound of 26 reproduced in the proposition. Taken together, the two bounds tightly sandwich the Fisher information within the same functional form 2\theta^{2}/(2\theta^{2}+d) , up to a constant shift in the denominator. In particular, as noted for example by 26 , they capture the asymptotic behavior: I(d,\theta)\overset{d\to+\infty}{\longrightarrow}0,\;\;I(d,\theta)\overset{\theta\to+\infty}{\longrightarrow}1. These bounds characterize the dependence of the Fisher information on both the degrees of freedom d and the non-centrality parameter \theta^{2} . They will play a crucial role in deriving upper bounds on the Rényi divergence between non-central chi-squared distributions. Using the Fisher information bounds derived above and the local comparison principle in Proposition 3.1 , we can derive our first key result: explicit lower and upper bounds on the Rényi divergence D_{\alpha}(V^{\top}V,W^{\top}W) . Theorem 5.1 (Privacy amplification in the n_{\mathrm{syn}}\to+\infty and high privacy regime, k=1 ) . Assume without loss of generality that \|w\|_{F}^{2}\leq\|v\|_{F}^{2}\leq C . We note the amplification factor: \eta_{d}=\frac{D_{\alpha}(V^{\top}V,W^{\top}W)}{D_{\alpha}(V,W)}. For k=1 , we have: \displaystyle\frac{2\|w\|^{2}}{2\|w\|^{2}+d}+o(1)\leq\eta_{d}\leq\frac{2C^{2}}{2C^{2}+d-3}+o(1), where the o(1) term is relative to the sensitivity \Delta . This result provides a sharp characterization of the divergence. The rightmost inequality shows that the divergence is upper bounded by the factor 2C^{2}/(2C^{2}+d-3) that quantifies the degree of privacy amplification. We observe two distinct behaviors. As \|w\|,C\to+\infty , the amplification factor lower bound 2C^{2}/(2C^{2}+d)\to 1 , yielding no amplification, consistent with Proposition 4.3 . However, for a fixed C>0 , the amplification factor tends to 0 at rate O(d^{-1}) . Importantly, the lower bound that we obtain shows that our analysis is sharp: this behavior is not an artifact of our proof technique. As a result, our upper bound provides a unified and tight expression capturing both amplification and post-processing. While this bound is only relevant when the sensitivity is small, this is typically the case if parameters are obtained through regularized linear regression, where \Delta is of order 1/n ( 14 ) . Nevertheless, we also derive a global upper bound based on our criterion in Proposition 3.2 . Theorem 5.2 . (Privacy amplification in the n_{\mathrm{syn}}\to+\infty regime, k=1 ). Let us denote: f(\alpha,C,d,\Delta)=1+\alpha\sqrt{\frac{2C^{2}}{2C^{2}+d-3}}\frac{e^{(\alpha-1)(2\alpha-1)\frac{\Delta^{2}}{2}}-1}{(\alpha-1)(2\alpha-1)\Delta}. Then, for k=1 , we have: \displaystyle D_{\alpha}(V^{\top}V,W^{\top}W)\leq\frac{1}{\alpha-1}\log f(\alpha,C,d,\Delta). Due to our proof technique, we obtain a o(d^{-1/2}) rate, relying on the asymptotic equivalence \log(1+x)\sim x when d is small relative to the other considered quantities. However, Theorem 5.1 suggests that a o(d^{-1}) exact rate is achievable. 5.2 Releasing Multi-Dimensional Synthetic Data We now turn to the evaluation D_{\alpha}(V^{\top}V,W^{\top}W) in the general case where k>1 . In this multi-dimensional case, the matrix V^{\top}V now follows a non-central Wishart distribution \mathcal{W}(k,d,v^{\top}v) with size k , degree of freedom d and non-centrality parameter \Omega_{v}=v^{\top}v . When d\geq k , V^{\top}V admits a density ( 39 ) : P_{v}(y)=P_{0}(y)e^{-tr(\Omega_{v})/2}{}_{0}F_{1}\Big(\frac{d}{2},\frac{1}{4}y^{1/2}\Omega_{v}y^{1/2}\Big), where P_{0} denotes the density of the (central) Wishart distribution \mathcal{W}(k,d) of size k and with degree of freedom d , and {}_{0}F_{1} is the generalized hypergeometric function of a matrix argument. The function {}_{0}F_{1} can be characterized as a series involving zonal polynomials of a matrix argument. However, these latter polynomials are notoriously hard to handle ( 30 ) , making direct computation of D_{\alpha}(V^{\top}V,W^{\top}W) intractable. Furthermore, the Fisher information upper bounds of Proposition 5.1 do not readily extend to the k>1 setting. In fact, although the diagonal elements of V^{\top}V are independent non-central chi-squared random variables, the non-diagonal elements are not independent, and follow a substantially more complex joint distribution. To our knowledge, there is no existing analysis in the literature characterizing the Fisher information of non-central Wishart distributions with respect to the non-centrality parameter. Let v=U_{v}^{\top}\Sigma_{v}S_{v},w=U_{w}^{\top}\Sigma_{w}S_{w} denote the singular value decomposition of v and w , and define the Gram operator G(M)=M^{\top}M . Leveraging the invariance of Gaussian matrices by orthogonal transformations, we obtain the following result. Lemma 5.1 (Rényi divergence equivalence along the SVD) . Let G_{v}=G((N+\Sigma_{v})S_{v}) , G_{w}=G((N+\Sigma_{v})S_{w}) . Then, D_{\alpha}(V^{\top}V,W^{\top}W)=D_{\alpha}(G_{v},G_{w}). The distribution of V^{\top}V is invariant under left multiplication of v by orthogonal matrices. Then, distinguishing between V^{\top}V and W^{\top}W is equivalent to distinguishing between between the corresponding non-centrality matrices G(\Sigma_{v}) and G(\Sigma_{w}S_{w}S_{v}^{\top}) . In order to do this, we introduce a smooth parametrization between these two non-centralities and derive a upper bound on the associated Fisher information. This construction leverages the geometry and the invariance of the non-centrality parameter: v\overset{(1-\theta)v+\theta Uw}{\longrightarrow}w , for any orthogonal matrix U\in O(d) . Our analysis relies on the decomposition of V^{\top}V=\sum_{i=1}^{d}V_{i}^{\top}V_{i} into a sum of independent non-central Wishart matrices. Then, we aggregate these terms into \lfloor d/k\rfloor non-central Wishart matrices with rank 1 non-centrality parameters. Coupled with a post-processing argument, this allows to relate the considered Fisher information to the Fisher information of non-central chi-squared distributions, which has been analyzed in Section 5.1 . Below, we get a upper bound of Fisher information for the straight line parametrization. Theorem 5.3 . (Fisher information of non-central Wishart distributions). Assume d\geq k . For v,w\in\mathbb{R}^{d\times k} , there exists a smooth path \theta\in[0,1]\mapsto v_{\theta}\in\mathbb{R}^{d\times k} for the family of distributions: P:=\{p_{\theta}=\mathcal{W}(k,d,v_{\theta}^{\top}v_{\theta});\;\theta\in[0,1]\}, such that p_{0}=p_{v} , p_{1}=p_{w} , and for all \theta\in[0,1] , the Fisher information I(\theta) of the family of distribution P with respect to \theta verifies: I(\theta)\leq\inf_{U\in O(d)}\|v-Uw\|_{F}^{2}\frac{2\|v_{\theta}\|_{F}^{2}}{2\|v_{\theta}\|_{F}^{2}+\lfloor d/k\rfloor-3}, with \|v_{\theta}\|_{F}=\|(1-\theta)v+\theta Uw\|_{F} . Using the Fisher information upper bound derived above and integrating along the path, we can upper bound the Rényi divergence using the general Fisher information criterion of Proposition 3.2 . Leveraging these results, we can upper bound the privacy loss associated to releasing an infinite number of synthetic data for k>1 . Theorem 5.4 . (Privacy in the n_{\mathrm{syn}}\to+\infty regime, k>1 ). Assume without loss of generality that \|w\|_{F},\|v\|_{F}\leq C . We note the amplification factor: \textstyle\eta_{d,k}=\frac{D_{\alpha}(V^{\top}V,W^{\top}W)}{D_{\alpha}(V,W)}. For k>1 , we have: \displaystyle\eta_{d,k}\leq\frac{2C^{2}}{2C^{2}+\lfloor d/k\rfloor-3}+o(1), where the o(1) term is relative to the sensitivity \Delta . Moreover, we have the general upper bound: D_{\alpha}(V^{\top}V,W^{\top}W)\leq\frac{1}{\alpha-1}\log f(\alpha,C,\lfloor d/k\rfloor,\Delta). This upper bound establishes a privacy amplification phenomenon for releasing k -dimensional synthetic data. The regime d\geq k is practically relevant, as in many modern regimes, the model is overparametrized, with d\gg k . Remark 5.5 . We conjecture that our Fisher information upper bound is improvable. Some looseness might arise from our post-processing technique, which decomposes the non-centrality matrix into rank 1 matrices. In particular, sharper Fisher information upper bounds might be obtained by leveraging the structure of the hypergeometric function of a matrix argument {}_{0}F_{1} . In Appendix F.3 , we outline such an approach in the case k=1 , providing an alternative proof of the Fisher information bound from Proposition 5.1 , which we believe could be extended to non-central Wishart distributions. We then discuss the additional ingredients required to generalize the argument to the case k>1 . 6 Experiments In this section, we perform experiments to illustrate our theoretical results. While sampling from ZV or V^{\top}V is straightforward, their densities are analytically intractable. Therefore, we rely on variational inference methods to estimate the Rényi divergences. Specifically, we numerically estimate the privacy loss using the Convex-Conjugate Rényi Variational Formula , introduced by 10 . In order to estimate the Rényi divergence between two distributions P and Q , we proceed as follows. We initialize a two-layer neural network f(w_{0},\cdot) with negative poly-softplus activation ( 10 ) and train it to maximize: \sup_{w}\Big\{\int f(w)dQ+\frac{1}{\alpha-1}\log\int|f(w)|^{(\alpha-1)/\alpha}dP\Big\}. Optimization is performed via minibatch stochastic gradient ascent. At each iteration, we sample X=(X_{1},\dots,X_{l})\sim P^{\otimes l} , Y=(Y_{1},\dots,Y_{l})\sim Q^{\otimes l} for some batch size l>0 and update the parameters using the empirical objective: \displaystyle F(w,X,Y) \displaystyle=\textstyle\frac{1}{l}\sum_{i=1}^{l}f(w,Y_{i}) \displaystyle+\textstyle\frac{1}{\alpha-1}\log\Big(-\frac{1}{l}\sum_{i=1}^{l}f(w,X_{i})^{(\alpha-1)/\alpha}\Big). A more detailed discussion can be found in Appendix G . 6.1 Estimating D_{\alpha}(ZV,ZW) as a function of n_{\text{syn}} We estimate the divergence D_{\alpha}(ZV,ZW) as a function of the number of released synthetic samples n_{\text{syn}} , in order to qualitatively assess its convergence behavior. Remarkably, Figure 2 shows that the Rényi divergence rapidly converges to a plateau, whose value is strictly lower than that of the corresponding post-processing privacy guarantee. Empirically, the number of synthetic samples required to reach the plateau appears to increase with d . The different values of the plateaus (dotted lines) highlight the amplification, which is significant even for the small values of d\in(1,10) considered here. This empirical behavior illustrates our theoretical finding that privacy amplification persists under unlimited data release, and validates our focus on the infinite synthetic data release regime. Figure 2: Empirical estimation of D_{\alpha}(ZV,ZW) as a function of the number of released synthetic data n_{\text{syn}} for multiple values of d , k=\Delta=1 , C=\alpha=2 . 6.2 Estimating D_{\alpha}(V^{\top}V,W^{\top}W) as a function of d,\Delta In this section, we aim to estimate the empirical tightness of the local Rényi divergence bounds derived from Fisher information (Theorem 5.1 ) in the case where k=1 . These bounds are asymptotic, and are thus expected to be informative only in the high privacy regime \Delta\ll 1 . Specifically, we assess, for \Delta\in(0,1) , the extent to which the relation predicted by Theorem 5.1 is satisfied: 2\|w\|_{F}^{2}/(2\|w\|_{F}^{2}+d)\lesssim\eta_{d}\lesssim 2C^{2}/(2C^{2}+d-3) . We estimate the Rényi divergence D_{\alpha}(V^{\top}V,W^{\top}W) and compare the results to the theoretical envelope induced by the above bounds. In our experiments, we set 1\leq\|w\|_{F}\leq\|v\|_{F}\leq C=2 , and evaluate D_{\alpha}(V^{\top}V,W^{\top}W) as a function of \Delta . The results, represented in Figure 3 , show that for small perturbations ( \Delta<1 ), the theoretical local bounds provide an accurate estimation of the amplification. In particular, for d=50 , the amplification effect is clearly observable. Figure 3: Empirical estimation of D_{\alpha}(V^{\top}V,W^{\top}W) as a function of \Delta for multiple values of d , k=1 , \|w\|_{F}\geq 1 , C=2 . 7 Discussion In this section, we discuss the tightness of the infinite data release privacy loss D_{\alpha}(V^{\top}V,W^{\top}W) with respect to D_{\alpha}(ZV,ZW) . We also discuss potential directions for extending privacy amplification results to linear generators trained with widely used private learning algorithms. 7.1 On the tightness of the plateau D_{\alpha}(V^{\top}V,W^{\top}W) As discussed in Section 6 and 2 , the privacy loss D_{\alpha}(ZV,ZW) converges to D_{\alpha}(V^{\top}V,W^{\top}W) when n_{\text{syn}}\to+\infty . A natural question is whether the plateau D_{\alpha}(V^{\top}V,W^{\top}W) is a meaningful approximation of D_{\alpha}(ZV,ZW) for finite n_{\text{syn}} , especially when d is large. We show that, in the tractable case k=1 , for sufficiently large d , the rate of convergence of the difference D_{\alpha}(V^{\top}V,W^{\top}W)-D_{\alpha}(ZV,ZW) to 0 is at least as fast as O(1/n_{\text{syn}}) , with constants that can be chosen uniformly over all large d . Proposition 7.1 (Privacy gap is O(1/n_{\text{syn}}) for k=1 ) . Let \alpha\geq 2 and v,w\in\mathbb{R}^{d} . Let \tau=(\|v\|,\|w\|,\Delta,\alpha) and assume that k=1 . Then, there exists K=K(\tau),K^{\prime}=K^{\prime}(\tau) and a threshold d_{0}=d_{0}(\tau) such that for all d\geq d_{0} and n_{\text{syn}}\geq 1 : \textstyle D_{\alpha}(V^{\top}V,W^{\top}W)-D_{\alpha}(ZV,ZW)\leq\frac{K}{n_{\text{syn}}}+K^{\prime}e^{-cn_{\text{syn}}}. We are interested in regimes where D_{\alpha}(V^{\top}V,W^{\top}W) itself decreases with d (e.g., scaling like d^{-\gamma} for some \gamma>0 , \gamma\geq 1/2 follows from Theorem 5.4 ). In such regimes, the additive gap bound above can be used to assess tightness: achieving a constant-factor approximation of the plateau requires the gap to be smaller than the plateau scale. Corollary 7.1 (Constant factor tightness for large n_{\text{syn}} ) . Assume that D_{\alpha}(V^{\top}V,W^{\top}W)=\Omega(d^{-\gamma}) , k=1 . Then, choosing n_{\text{syn}}\gtrsim d^{\gamma} suffices to make D_{\alpha}(ZV,ZW) a constant-factor approximation of the plateau (up to exponentially small terms). A tighter characterization of the regime n_{\text{syn}}\leq d than the work of 40 is an interesting direction for future work, as it seems that some additional privacy gains may be achievable in this regime. Remark 7.1 . The constants in Proposition 7.1 can be drastically improved by leveraging the fact that D_{\alpha}(V^{\top}V,W^{\top}W)\lesssim d^{-\gamma}D_{\alpha}(V,W) , as proved in Theorem 5.4 . 7.2 On the generalization to other training procedures The approach developed in this paper may extend to alternative training procedures for linear generators. Importantly, the sufficient-statistics limit D_{\alpha}(ZV,ZW)\overset{n_{\text{syn}}\to\infty}{\longrightarrow}D_{\alpha}(V^{\top}V,W^{\top}W) of Proposition 4.1 does not rely on V or W being isotropic Gaussian matrices (related to output perturbation). As noted by 13 , Noisy Gradient Descent applied to linear regression converges to a matrix Gaussian distribution, suggesting that our framework could naturally accommodate this more general setting. Extending the analysis in this direction is a promising avenue for future work. 8 Conclusion In this work, we studied privacy amplification for synthetic data released by a linear generator and showed that, under parameter boundedness, amplification persists even when releasing an unbounded number of synthetic samples. Our analysis provides a unified characterization of the privacy loss that smoothly interpolates between classical post-processing guarantees and genuine amplification. Notably, it does not depend on the number of released synthetic records, improving on the work of 40 . A key technical insight is that, in the infinite-release regime, the privacy loss is fully characterized by Gram statistics V^{\top}V , which enables tractable bounds via Fisher information. This yields local (high privacy) Rényi DP upper bounds with tight amplification factors. Beyond the local regime, we derived a general upper bound on Rényi divergence based on a criterion relating Fisher information to Rényi divergences provided a computable upper envelope on the Rényi divergence. While this bound is fully non-asymptotic and broadly applicable in our setting, we do not expect it to be tight in general. An important open question is whether this criterion can be improved to recover the local rates globally. More broadly, our results provide concrete analytical insights, such as the role of parameter boundedness, the Fisher information analysis, the explicit dependence of amplification on the model dimension, and the sufficient statistics reduction. We expect that these insights will provide both intuition and methodological guidance for understanding privacy amplification in more complex models and data release mechanisms. An important direction for future work in this regard is to extend these results to deep generative models trained with practical differentially private algorithms like DP-SGD, in order to determine whether similar privacy amplification phenomena manifest in realistic settings. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. This appendix provides detailed discussions, proofs and supplementary content. Appendix A Background on differential privacy This section briefly reviews the differential privacy notions and tools used in the paper. Throughout, \mathcal{M} denotes a randomized mechanism. We call two datasets \mathcal{D} and \mathcal{D}^{\prime} of the same size n adjacent if they differ in exactly one data record. Rényi Differential Privacy quantifies privacy via the Rényi divergence between the output distributions induced by \mathcal{M} on adjacent datasets ( 38 ) . Definition A.1 (Rényi Differential Privacy (RDP) ( 38 ) ) . For \alpha>1,\varepsilon>0 , the Rényi divergence of order \alpha between distributions P and Q is D_{\alpha}(P,Q):=\frac{1}{\alpha-1}\log\mathbb{E}_{x\sim P}\Big[\big(\frac{P(x)}{Q(x)}\big)^{\alpha}\Big] . For X\sim P,W\sim Q , we note D_{\alpha}(V,W)=D_{\alpha}(P,Q) . Then, \mathcal{M} satisfies (\alpha,\varepsilon) -RDP if for every pair of adjacent datasets \mathcal{D},\mathcal{D}^{\prime} , D_{\alpha}(\mathcal{M}(\mathcal{D})\|\mathcal{M}(\mathcal{D}^{\prime}))\leq\varepsilon. The prior results of 40 are formulated for trade-off functions in the f -Differential privacy ( f -DP) framework. Trade-off functions ( 17 ) describe the optimal relationship between type I and type II errors for tests that attempt to distinguish the outputs produced from two adjacent datasets. Formally, let P and Q be two distributions and consider testing: H_{1}:\text{the distribution is }P\text{~~ vs ~~}H_{2}:\text{the distribution is }Q. For a rejection rule \phi\in[0,1] , the type I error is \mathbb{E}_{P}[\phi] and the type II error is 1-\mathbb{E}_{Q}[\phi] . Definition A.2 (Trade-off function) . Let t\in(0,1) . Then, the trade-off function of the statistical test between H_{1} and H_{2} is defined as: T(P,Q)(t)=\inf_{\phi}\{1-\mathbb{E}_{Q}[\phi]:\mathbb{E}_{P}[\phi]\leq t\}. For X\sim P,W\sim Q , we note T(V,W)=T(P,Q) . f -Differential privacy is defined as a upper bound condition on the function f for any trade-off function between the statistical test between adjacent datasets: H_{1}:\text{the distribution is }\mathcal{M}(\mathcal{D})\text{~~ vs ~~}H_{2}:\text{the distribution is }\mathcal{M}(\mathcal{D}^{\prime}). Definition A.3 . Let f:[0,1]\to[0,1] be a decreasing convex function. A mechanism \mathcal{M} satisfies f -differential privacy ( f -DP) if for all adjacent datasets \mathcal{D},\mathcal{D}^{\prime} , T(\mathcal{M}(\mathcal{D}),\mathcal{M}(\mathcal{D}^{\prime}))\geq f. Appendix B Comparison of our upper bounds to the prior results of 40 In Section 2 , we mentioned that the privacy loss upper bounds obtained by 40 follow the relationship: D_{\alpha}(ZV,ZW)\lesssim\min\left\{1,k\sqrt{\frac{n_{\text{syn}}}{d-k}}\right\}D_{\alpha}(V,W). We clarify that this upper bound is not a guarantee offered by 40 (after conversion from trade-off functions to RDP). It represents the best best scaling one can hope to extract from their proof technique. This means that any conversion of the results of 40 to RDP guarantees is at least as loose as this bound. In the rest of the section, we make precise this claim and highlight several limitations of these guarantees. The privacy loss upper bounds of 40 are formulated in terms of trade-off functions. We restate below the result we use, with the notational change that the trade-off function argument is denoted by t\in(0,1) (reserving \alpha for Rényi orders). Theorem B.1 (Privacy amplification for synthetic data, adapted from ( 40 ) ) . Let d>0 . We denote C_{n_{\text{syn}},k,d}=C^{\prime}k\sqrt{\frac{n_{\text{syn}}}{d-k}} , where C^{\prime}>0 is an absolute constant. Let N\in\mathbb{R}^{n_{\text{syn}}\times d} be a Gaussian matrix independent of Z , and define the trade-off function: G_{(d-k,v,w)}=T\left(\sqrt{d-k}N+Zv,\sqrt{d-k}N+Zw\right), with G_{(d-k,v,w)}(t)=0 for t>1 . Then, for all t\in(0,1) : f:=\max\left\{T(V,W),G_{(d-k,v,w)}\left(t+C_{n_{\text{syn}},k,d}\right)-C_{n_{\text{syn}},k,d}\right\}\leq T(ZV,ZW)(t). (1) Specifically, we show that: • Proposition B.2 : Any (\varepsilon,\delta) -DP conversion of the amplification term G_{(d-k,v,w)}\left(t+C_{n_{\text{syn}},k,d}\right)-C_{n_{\text{syn}},k,d} of Equation 1 yields \delta=\Omega\left(k(n_{\text{syn}}/d)^{1/2}\right) . In particular, to get \delta=o(n^{-1}) (with n the sample size), one would need n_{\text{syn}}=o\left(\frac{d}{k^{2}n^{2}}\right) . • Proposition B.3 : Equation 1 does not yield any amplification in the high privacy regime \Delta<2C^{\prime}k\sqrt{\frac{2\pi n_{\text{syn}}}{d-k}} . In this regime, the privacy loss is no better than D_{\alpha}(V,W) . • Proposition B.4 : In the privacy amplification regime, the Rényi divergence conversion l_{\alpha}(f) of the trade-off function f derived from Equation 1 scales as 2C^{\prime}k\sqrt{\frac{n_{\text{syn}}}{d-k}} : l_{\alpha}(f)\geq 2C^{\prime}k\sqrt{\frac{n_{\text{syn}}}{d-k}}D_{\alpha}(V,W). First, we explain our proof strategy to derive RDP bounds from Equation 1 . Figure 4 represents the associated trade-off function. This function is piecewise, separated into two regions: a Gaussian trade-off region and an amplification region. By symmetry, the trade-off function is Gaussian at the edges, in the set (0,x_{-})\cup(x_{+},1) for some 0\leq x_{-}\leq x_{+}\leq 1 , and G_{(d-k,v,w)}\left(t+C_{n_{\text{syn}},k,d}\right)-C_{n_{\text{syn}},k,d} at the center, in the interval (x_{-},x_{+}) . Then, Propositions B.2 and B.3 become clear: any (\varepsilon,\delta) conversion of the amplification regime yields \delta\geq C_{n_{\text{syn}},k,d} . Moreover, there is no amplification if the curve G_{(d-k,v,w)}\left(t+C_{n_{\text{syn}},k,d}\right)-C_{n_{\text{syn}},k,d} does not intersect the curve T(V,W) . We identify the regimes where it happens. Figure 4: Representation of the trade-off function f for \Delta=1,C^{\prime}=1,d=60,n_{\text{syn}}=1,k=1 . Relaxing the trade-off function f : We rely on a simplification of the G_{(d-k,v,w)}\left(t+C_{n_{\text{syn}},k,d}\right)-C_{n_{\text{syn}},k,d} part. Let t\in(0,1) . From Equation 1 , T(ZV,ZW)(t)\geq f(t) . f is the certifiable privacy loss upper bound obtained by 40 . Then, for any trade-off function h>f , the analysis of 40 is not sufficient to certify privacy. In particular, any trade-off function T satisfies T(t)\leq 1-t . As G_{(d-k,v,w)} is a trade-off function, we have G_{(d-k,v,w)}\leq 1-I_{d} . Then, f\leq\max\left\{T(V,W),1-I_{d}-2C_{n_{\text{syn}},k,d}\right\}=\max\left\{\Phi(\Phi^{-1}(1-\alpha)-\Delta),1-I_{d}-2C_{n_{\text{syn}},k,d}\right\}:=g, where \Phi is the Gaussian cumulative distribution function. Replacing G_{(d-k,v,w)}\left(t+C_{n_{\text{syn}},k,d}\right)-C_{n_{\text{syn}},k,d} by 1-I_{d}-2C_{n_{\text{syn}},k,d} unlocks simpler Rényi divergence conversion. This yields lower bounds and minimal conditions for amplification. A useful reparametrization : Now, assume that the curves \Phi(\Phi^{-1}(1-\cdot) and 1-I_{d}-2C_{n_{\text{syn}},k,d} intersect (which means that there is amplification). Note z=\Phi^{-1}(1-t) . Then, f(z)=\max\{\Phi(z-\Delta),\Phi(z)-2C_{n_{\text{syn}},k,d}\} . The intersection is located at the solutions of \Phi(z)-\Phi(z-\Delta)=2C_{n_{\text{syn}},k,d} . We note z_{-}\leq 0\leq z_{+} the lower and upper solutions. Converting trade-off function into RDP guarantees : From 17 , trade-off functions convert into Rényi divergences through the following functional: Proposition B.1 (Conversion from f -DP to RDP ( 17 ) ) . If a mechanism is f -DP, then it is (\alpha,l_{\alpha}(f)) -RDP for all \alpha>1 with: l_{\alpha}(f)=\begin{cases}\frac{1}{\alpha-1}\log\int|f^{\prime}(t)|^{1-\alpha}dt\text{ if }z_{f}=1,\\ +\infty\text{ else,}\end{cases} with z_{f}=\inf\{t\in(0,1);f(t)=0\}=1 . We can now prove Propositions B.2 , B.3 and B.4 : Proposition B.2 (DP conversion forces \delta of order C_{n_{\text{syn}},k,d} ) . Any (\varepsilon,\delta) -DP guarantee derived from the amplification part of Equation 1 necessarily satisfies \delta=\Omega(C_{n_{\text{syn}},k,d}) (uniformly over \varepsilon>0 ). Therefore, n_{\text{syn}}=o\left(\frac{d}{k^{2}n^{2}}\right) . Proof. Immediate from the form of the trade-off function and the (\varepsilon,\delta) conversion result of 17 . Therefore, for a reasonable privacy protection \delta=o(n^{-1}) , we have n^{-1}>C_{n_{\text{syn}},k,d} , giving the desired result. Moreover, converting the trade-off function of Equation 1 either yields no amplification (in the Gaussian region) or large \delta (in the amplification region). ∎ Proposition B.3 (No amplification in the high-privacy regimes) . If \Delta<2C^{\prime}k\sqrt{\frac{2\pi n_{\text{syn}}}{d-k}} , the RDP conversion l_{\alpha}(f) of Equation 1 yields l_{\alpha}(f)=D_{\alpha}(V,W) . In particular, G_{(d-k,v,w)}\left(t+C_{n_{\text{syn}},k,d}\right)-C_{n_{\text{syn}},k,d} does not intersect T(V,W) . Proof. The function H=\Phi(z)-\Phi(z-\Delta) is symmetric around \Delta/2 . In fact, \displaystyle H\left(\frac{\Delta}{2}+z\right) \displaystyle=P\left(N\leq\frac{\Delta}{2}+z\right)-P\left(N\leq z-\frac{\Delta}{2}\right) \displaystyle=P\left(N\geq-\frac{\Delta}{2}-z\right)-P\left(N\geq-z+\frac{\Delta}{2}\right) \displaystyle=P\left(N\leq\frac{\Delta}{2}-z\right)-P\left(N\leq-\frac{\Delta}{2}-z\right)=H\left(\frac{\Delta}{2}-z\right). It is also increasing then decreasing. In fact, H^{\prime}(z)=\frac{1}{\sqrt{2\pi}}(e^{-z^{2}/2}-e^{-(z-\Delta)^{2}/2})\geq 0\iff z\leq\frac{\Delta}{2} . Its maximum value is attained for z=\frac{\Delta}{2} . Therefore, the intersection between \Phi(\Phi^{-1}(1-t)-\Delta) and (1-t-2C_{n_{\text{syn}},k,d}) exists if and only if H(\Delta/2)\geq 2C_{n_{\text{syn}},k,d} . Also, H(\Delta/2)=\Phi(\Delta/2)-\Phi(-\Delta/2)=2\Phi(\Delta/2)-1 . Then, there is no amplification for: 2C_{n_{\text{syn}},k,d}>\frac{\Delta}{\sqrt{2\pi}}\geq\sqrt{\frac{2}{\pi}}\int_{0}^{\Delta/2}e^{-t^{2}/2}dt=2\Phi(\Delta/2)-1. ∎ Now, we show that the conversion of the trade-off function f into Rényi divergence l_{\alpha}(f) satisfies l_{\alpha}(f)\geq C_{n_{\text{syn}},k,d}D_{\alpha}(V,W) when C_{n_{\text{syn}},k,d} is sufficiently small. Proposition B.4 . Assume that d>n_{\text{syn}},k . Then, in the privacy amplification regime, or equivalently, when G_{(d-k,v,w)}\left(t+C_{n_{\text{syn}},k,d}\right)-C_{n_{\text{syn}},k,d} intersects with T(V,W) , the following relationship holds: l_{\alpha}(f)\geq 2C^{\prime}k\sqrt{\frac{n_{\text{syn}}}{d-k}}D_{\alpha}(V,W). Proof. We give a lower bound of the Rényi divergence associated to the trade-off function f . We decompose the integral into 3 pieces representing the Gaussian regime and the amplification regime, and perform the change of variable z=\Phi^{-1}(1-t) . \displaystyle\exp((\alpha-1)l_{\alpha}(f)) \displaystyle=\int_{0}^{1}|f^{\prime}(t)|^{1-\alpha}dt \displaystyle=\int_{0}^{1}|\partial_{\theta}\max\left\{\Phi(\Phi^{-1}(1-t)-\Delta),1-t-C_{n_{\text{syn}},k,d}\right\}|^{1-\alpha}dt \displaystyle=\int_{0}^{1-\Phi(z_{+})}|\partial_{\theta}\Phi(\Phi^{-1}(1-t)-\Delta)|^{1-\alpha}dt \displaystyle+\int_{1-\Phi(z_{-})}^{1}|\partial_{\theta}\Phi(\Phi^{-1}(1-t)-\Delta)|^{1-\alpha}dt+|\Phi(z_{-})-\Phi(z_{+})| \displaystyle=\int_{0}^{1-\Phi(z_{+})}-(\Phi^{-1})^{\prime}(1-t)^{1-\alpha}\phi(\Phi^{-1}(1-t)-\Delta)^{1-\alpha}dt \displaystyle+\int_{1-\Phi(z_{-})}^{1}-(\Phi^{-1})^{\prime}(1-t)^{1-\alpha}\phi(\Phi^{-1}(1-t)-\Delta)^{1-\alpha}dt+|\Phi(z_{-})-\Phi(z_{+})| \displaystyle=\int_{z_{+}}^{+\infty}\frac{\phi(z)^{\alpha}}{\phi(z-\Delta)^{\alpha-1}}dz+\int_{-\infty}^{z_{-}}\frac{\phi(z)^{\alpha}}{\phi(z-\Delta)^{\alpha-1}}dz+|\Phi(z_{-})-\Phi(z_{+})|\;,\;(z=\Phi^{-1}(1-t)), where \phi is the Gaussian density function. By completing the square, for x1 , we have: \displaystyle A=\Phi(z_{-}+(\alpha-1)\Delta)+\Phi(-(z_{+}+(\alpha-1)\Delta)\geq\Phi(z_{-}+(\alpha-1)\Delta)\geq\Phi(z_{-})\geq 2C_{n_{\text{syn}},k,d}. Also, we obtain a lower bound of A+L by showing that (\Phi(z_{-}+(\alpha-1)\Delta)-\Phi(z_{-}))-(\Phi(z_{+}+(\alpha-1)\Delta)-\Phi(z_{+}))\geq 0 . \displaystyle A+L \displaystyle=1+(\Phi(z_{-}+(\alpha-1)\Delta)-\Phi(z_{-}))-(\Phi(z_{+}+(\alpha-1)\Delta)-\Phi(z_{+})) \displaystyle=1+\int_{0}^{(\alpha-1)\Delta}(\phi(t+z_{-})-\phi(t+z_{+}))dt \displaystyle=1+\frac{1}{\sqrt{2\pi}}\int_{0}^{(\alpha-1)\Delta}e^{((t+z_{+})^{2}-(t+z_{-})^{2})/2}dt \displaystyle=1+\frac{1}{\sqrt{2\pi}}\int_{0}^{(\alpha-1)\Delta}e^{(z_{+}-z_{-})(z_{+}+z_{-}+2t)/2}dt\geq 1. By the same reasoning, 0\leq A=1+\Phi(z_{-}+(\alpha-1)\Delta)-\Phi(z_{+}+(\alpha-1)\Delta)=1+\int_{z_{+}}^{z_{-}}\phi(t+(\alpha-1)\Delta)dt\leq 1. Then, by convexity of the x\mapsto e^{x} function for the input A\alpha(\alpha-1)\Delta^{2}/2+(1-A)\cdot 0 : e^{\alpha(\alpha-1)\Delta^{2}/2}A+L\geq e^{\alpha(\alpha-1)\Delta^{2}/2}A+1-A\geq e^{A\alpha(\alpha-1)\Delta^{2}/2}\geq e^{C_{n_{\text{syn}},k,d}\alpha(\alpha-1)\Delta^{2}}. Recall that D_{\alpha}(V,W)=\alpha\Delta^{2}/2 . Then, \exp((\alpha-1)l_{\alpha}(f))\geq\exp(2(\alpha-1)C_{n_{\text{syn}},k,d}D_{\alpha}(V,W)). Then, the desired result holds: l_{\alpha}(f)\geq 2C^{\prime}k\sqrt{\frac{n_{\text{syn}}}{d-k}}D_{\alpha}(V,W). ∎ Appendix C Relationship between Rényi divergences and Fisher information In this section, we introduce Fisher information and its properties needed for the understanding of the paper. Then, we examine the relationship between Rényi divergences and Fisher information. C.1 Fisher information and its properties This appendix briefly recalls the notion of Fisher information and the few properties we rely on throughout the paper. In this paper, we focus on Fisher information of families of probabilities indexed by scalar parameters. In this paper, all considered distributions are "sufficiently regular". We define sufficient regularity conditions for the usual Fisher information properties to hold. Definition C.1 (Regular family of probability measures) . Let P=\{P_{\theta}:\theta\in\Theta\subset\mathbb{R}\} be a family of probability measures on (\mathbb{R},\mathcal{B}(\mathbb{R})) . Then, P is regular if: • \Theta is a right-open set of \mathbb{R} : for every point \theta\in\Theta , there exists a neighborhood of the form [\theta,\theta+\delta)\subset\Theta . • All P_{\theta}\in P have a common support. • For all \theta\in\Theta , P_{\theta} is absolutely continuous with respect to the Lebesgue measure. We note the corresponding densities p_{\theta} . • For all \theta\in\Theta , \theta\mapsto\log p_{\theta}(x) is twice almost everywhere differentiable. • For all \theta\in\Theta , there exist a neighborhood U and integrable functions \tilde{h}_{1},\tilde{h}_{2} such that for all \theta^{\prime}\in U , \left|\partial_{\theta^{\prime}}p_{\theta^{\prime}}(x)\right|\leq\tilde{h}_{1}(x),\;\;\left|\partial_{\theta^{\prime}}^{2}p_{\theta^{\prime}}(x)\right|\leq\tilde{h}_{2}(x). This condition ensures the sufficient regularity of P for generic properties of Fisher information to hold. • For all \theta\in\Theta , there exist a neighborhood U and integrable functions h_{1},h_{2} such that for all \theta^{\prime}\in U , \left|\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x)\right|\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}}\leq h_{1}(x),\;\;\left(\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x)^{2}+\left|\partial_{\theta^{\prime}}^{2}\log p_{\theta^{\prime}}(x)\right|\right)\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}}\leq h_{2}(x). This condition is specific to the differentiability of Rényi divergences. Note that these are not minimal conditions for the existence of Fisher information. Now, we define Fisher information and score functions: Definition C.2 (Fisher information and score function) . Let P=\{P_{\theta}:\theta\in\Theta\subset\mathbb{R}\} be a regular family of probability measures. Then, the score function is defined by: s_{\theta}(x)=\partial_{\theta}\log p_{\theta}(x). The Fisher information of the family P with respect to the parameter \theta\in\Theta is: I(\theta)=\mathbb{E}_{X\sim P_{\theta}}[s_{\theta}(X)^{2}]\in\mathbb{R}^{+}. Here are some properties of the Fisher information that are used throughout the paper: Proposition C.1 (Fisher information and score properties) . Let P=\{P_{\theta}:\theta\in\Theta\subset\mathbb{R}\} be regular of probability measures such that the Fisher information I(\theta) exists for all \theta\in\Theta . Then, the Fisher information and the score function satisfy the following properties: • Mean-zero score and information identity, Lemma 5.3 of 32 : \mathbb{E}_{X\sim P_{\theta}}[s_{\theta}(X)]=0 , hence I(\theta)=\Var_{X\sim P_{\theta}}(s_{\theta}(X)) . Moreover, when \partial_{\theta}^{2}\log p_{\theta}(X) exists almost averywhere, the information identity holds: I(\theta)=-\mathbb{E}_{X\sim P_{\theta}}[\partial_{\theta}^{2}\log p_{\theta}(X)]. • Post-processing inequality ( 44 ) : Let X\sim P_{\theta} and let Y\sim K(X) be generated from a Markov kernel K applied on X . Then, I_{Y}(\theta)\leq I_{X}(\theta). • Additivity under independent products, adapted from Theorem 5.8 of 32 : If X=(X_{1},\dots,X_{n}) are jointly independent given \theta , with joint density p_{\theta}(x_{1},\dots,x_{m})=\prod_{i=1}^{n}p^{(i)}_{\theta}(x_{i}) , then the Fisher information of X with respect to the parameter \theta decomposes as: I_{(X_{1},\dots,X_{n})}(\theta)=\sum_{i=1}^{n}I_{X_{i}}(\theta). C.2 A local relationship between Rényi divergences and Fisher information In parametric distributions, the second order derivative of most f -divergences coincide the Fisher information ( 41 ) . Rényi divergences are obtained as a smooth monotone transformation of a f -divergence. This yields a local expansion of the second order of Rényi divergences as a function of Fisher information under small perturbations of the parameter for sufficiently regular families of distributions, as noted by 24 and 48 . However, in these works, the exact technical conditions on the parametrization that are needed are not explicited, and the authors do not know references for these conditions ( 48 ) . For completeness, we prove that under the regularity conditions of Definition C.1 , the local expansion of Rényi divergences as a function of Fisher information hold. In this paper, all considered families of distributions satisfy these conditions. Note that we do not claim that these conditions are minimal. Proposition C.2 (Local relationship between Rényi divergences and Fisher information) . Let \{P_{\theta},\theta\in\Theta\subset\mathbb{R}\} be a family of probability distributions. Let \theta\in\Theta . Let \Delta>0 such that \theta+\Delta\in\Theta . Then, if P is regular (Definition C.1 ), D_{\alpha}(P_{\theta+\Delta},P_{\theta})=\frac{\alpha}{2}I(\theta)\Delta^{2}+o(\Delta^{2}). Proof. Let \alpha>1 and P be a regular family of distribution, as defined in Definition C.1 . For \theta,\theta^{\prime}\in\Theta , we note H(\theta^{\prime},\theta)=\int\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}}dx . Fix \theta\in\Theta and note for every \theta^{\prime}\in\Theta , G(\theta^{\prime})=H(p_{\theta^{\prime}},p_{\theta}) . Let \Delta>0 such that \theta+\Delta\in\Theta . Since P is regular, the conditions of Definition C.1 are sufficient to ensure the twice differentiability of G . Then G(\theta)=1 and: \displaystyle\partial_{\theta^{\prime}}G(\theta^{\prime}) \displaystyle=\alpha\int\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x)\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}}dx, \displaystyle\partial_{\theta^{\prime}}^{2}G(\theta^{\prime}) \displaystyle=\alpha\int(\partial_{\theta^{\prime}}^{2}\log p_{\theta^{\prime}}(x)+\alpha(\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x))^{2})\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}}dx. Since score has mean 0 (Proposition C.1 ), \partial_{\theta}G(\theta)=0 and \partial_{\theta}^{2}G(\theta)=\alpha\mathbb{E}_{X\sim P_{\theta}}[\partial_{\theta}^{2}\log p_{\theta}(X)]+\alpha^{2}I(\theta) . By the information equality (Proposition C.1 ), I(\theta)=-\mathbb{E}_{X\sim P_{\theta}}[\partial_{\theta}^{2}\log p_{\theta}(X)] . Therefore, \partial_{\theta}^{2}G(\theta)=\alpha(\alpha-1)I(\theta) . Since G is positive and twice differentiable on \Theta , D:\theta^{\prime}\mapsto D_{\alpha}(P_{\theta^{\prime}},P_{\theta})=\frac{1}{\alpha-1}\log G(\theta^{\prime}) is twice differentiable on \Theta . Then, \displaystyle\partial_{\theta^{\prime}}D(\theta^{\prime}) \displaystyle=\frac{1}{\alpha-1}\frac{\partial_{\theta^{\prime}}G(\theta^{\prime})}{G(\theta^{\prime})}, \displaystyle\partial_{\theta^{\prime}}^{2}D(\theta^{\prime}) \displaystyle=\frac{1}{\alpha-1}\left(\frac{\partial_{\theta^{\prime}}^{2}G(\theta^{\prime})}{G(\theta^{\prime})}-\frac{\partial_{\theta^{\prime}}G(\theta^{\prime})^{2}}{G(\theta^{\prime})^{2}}\right). Therefore, \partial_{\theta}D(\theta)=0 and \partial_{\theta}^{2}D(\theta)=\alpha I(\theta) . Thus, the following asymptotic expansion holds: D_{\alpha}(P_{\theta+\Delta},P_{\theta})=\frac{\alpha}{2}I(\theta)\Delta^{2}+o(\Delta^{2}). ∎ C.3 Beyond the local expansion: can Fisher information control Rényi divergence globally? A natural question is whether this local relationship between Fisher information and Rényi divergences extends beyond infinitesimal perturbations. More precisely, one may ask if there is a distribution-free functional F such that, given an arbitrary regular family of distribution P and any \theta<\theta^{\prime}\in\Theta : D_{\alpha}(P_{\theta^{\prime}},P_{\theta})=F(\alpha,\{I(u);\;u\in[\theta,\theta^{\prime}]\}). In general, no such characterization exists: Fisher information is a local quantity, and it does not determine Rényi divergence for arbitrary parameter values. Our counterexample of Theorem 4.1 of 2 , studied below, is a simple illustration of two family distributions that can have the same Fisher information but different Rényi divergences. There are, however, specific families of distributions in which global formulas are available. First, for distributions belonging to the same exponential family, Fisher information ( 32 , Theorem 5.4) and Rényi divergences ( 33 , Chapter 2) admit a closed form. Specific choices of parameters yield an integral representation of Rényi divergences in terms of the Fisher information ( 8 ) , but these parametrizations do not simplify the theoretical calculation of the associated Rényi divergences, as the ratio \frac{p_{\theta^{\prime}}(x)^{t}}{p_{\theta}(x)^{1-t}} (t\in(0,\alpha)) appear in the parametrized densities. A different line of work attempts to obtain global comparison criterions. For instance, one may hope that pointwise ordering of Fisher information implies ordering of Rényi divergences: \text{ For all }\;\theta\in\Theta,\;I_{P}(\theta)\leq I_{Q}(\theta)\implies\text{For all }\theta<\theta^{\prime}\in\Theta,\;D_{\alpha}(P_{\theta^{\prime}},P_{\theta})\leq D_{\alpha}(Q_{\theta^{\prime}},Q_{\theta}), where P and Q are two families of distributions indexed by the same parameter set \Theta , and the corresponding Fisher information are I_{P} and I_{Q} . Two existing attempts in the literature for Rényi divergences ( 2 , Theorem 4.1) and Kullback-Leibler (KL) divergences ( 23 , Theorem 3.1) obtain such a statement. However, but they are false in general, even for smooth regular families. We provide a counterexample and discuss the flaw in the proofs below. Theorem C.3 (Adapted from Theorem 4.1 of 2 ) . Let P=\{P_{\theta},\ \theta\in\Theta\} and Q=\{Q_{\theta},\ \theta\in\Theta\} be two families of distributions. Let 0<\theta_{0}<\theta_{1}\in\Theta . Let \alpha>1 . Assume that both families have finite Rényi divergence and Fisher information, continuous in \theta . If the following conditions hold: 1. For all \theta\in I=[\theta_{0},\theta_{1}],\;I_{P}(\theta)-I_{Q}(\theta)\geq d_{0}>0 , 2. For all \theta\in(\theta_{0},\theta_{1}) , \delta\mapsto D_{\alpha}(p_{\theta+\delta},p_{\theta}) and \delta\mapsto D_{\alpha}(q_{\theta+\delta},q_{\theta}) are three times differentiable and the third derivatives are uniformly bounded in a neighborhood I=[0,c] of zero, then: D_{\alpha}(p_{\theta_{1}},p_{\theta_{0}})>D_{\alpha}(q_{\theta_{1}},q_{\theta_{0}}). We start by pointing a counterexample to this theorem. Counterexample of Theorem 4.1 from 2 : Fix \alpha=2 . Let a,\sigma>0 . Let P_{a}=\{\Cauchy(\theta,a),\;\theta\in\mathbb{R}\} , where \theta is the location parameter of the Cauchy distribution and a is the scale parameter, and Q_{\sigma}=\{\mathcal{N}(\theta,\sigma^{2}),\;\theta\in\mathbb{R}\} . Let \theta_{0}<\theta<\theta_{1}\in\mathbb{R} . Then, for the location parameter the Fisher information is constant for both families: I_{P_{a}}(\theta)=\frac{1}{2a^{2}},\;\;\;I_{Q_{\sigma}}(\theta)=\frac{1}{\sigma^{2}}. Also, the Rényi divergence between Cauchy distributions ( 49 ) and between Gaussian distributions is known: D_{2}(p_{\theta_{1}},p_{\theta_{0}})=\log\left(1+\frac{(\theta_{1}-\theta_{0})^{2}}{2a^{2}}\right),\;\;\;D_{2}(q_{\theta_{1}},q_{\theta_{0}})=\frac{(\theta_{1}-\theta_{0})^{2}}{\sigma^{2}}. Both families of distributions satisfy the condition 2 of Theorem C.3 . Moreover, choosing a<\frac{1}{\sqrt{2}}\sigma allows to satisfy the condition 1 : for all \theta\in[\theta_{0},\theta_{1}] , I_{Q_{\sigma}}(\theta)\leq I_{P_{a}}(\theta) . However, for sufficiently large values of \theta_{1}-\theta_{0} , it is clear that D_{2}(p_{\theta_{1}},p_{\theta_{0}})\leq D_{2}(q_{\theta_{1}},q_{\theta_{0}}) . Flaw in the proofs of Theorem 4.1 from 2 and Theorem 3.1 from 23 : The core idea of 2 and 23 relies on showing that, under condition 1 , for all \theta\in\Theta , there exists \delta,d_{1}>0 such that \frac{1}{\delta^{2}}(D_{\alpha}(p_{\theta+\delta},p_{\theta})-D_{\alpha}(q_{\theta+\delta},q_{\theta}))\geq d_{1} . Then, they incorrectly drop the 1/\delta^{2} , by stating that when c<1 (so that \delta<1 ), D_{\alpha}(p_{\theta+\delta},p_{\theta})-D_{\alpha}(q_{\theta+\delta},q_{\theta})\geq d_{1} , which is generally false. The proof is not fixable independently of the regularity conditions. In fact, they try to generalize this statement for larger values of \delta by induction to derive a lower bound on D_{\alpha}(p_{\theta+k\delta},p_{\theta})-D_{\alpha}(q_{\theta+k\delta},q_{\theta}) for k>1 , by a telescoping argument, but now that the lower bound of D_{\alpha}(p_{\theta+\delta},p_{\theta})-D_{\alpha}(q_{\theta+\delta},q_{\theta}) depends on \delta , which depends on \theta , the recurrence fails. The theorem of 23 for KL divergence follows the same proof technique and is also false. C.4 A sufficient condition to control Rényi divergence with Fisher information (Proposition 3.2 ) In this section, we introduce a criterion that allows to obtain a upper bound of Rényi divergences as a function of Fisher information along a parameter segment [\theta,\theta^{\prime}] . The derivation assumes an explicit upper envelope on the Rényi divergence that is available in our setting. It has the desirable property that it vanishes when the Fisher information becomes small on [\theta,\theta^{\prime}] (provided D_{2\alpha-1} remains controlled). This upper bound is generally not tight, as it scales as \sqrt{I(\theta)} when the Fisher information is small, while the the local expansion of Theorem 5.1 suggests a I(\theta) rate. Here is the statement of the criterion: Proposition C.3 (Upper bounding Rényi divergences through Fisher information) . Let \{P_{\theta},\theta\in\Theta\subset\mathbb{R}\} be a family of distributions. Let \theta<\theta^{\prime}\in\Theta . We assume that there exists a function U:\mathbb{R}^{2}\to\mathbb{R^{+}} such that for all z\in(\theta,\theta^{\prime}) , D_{2\alpha-1}(P_{z},P_{\theta})\leq U(z,\theta) . We also assume that z\mapsto D_{\alpha}(P_{z},P_{\theta}) is differentiable on (\theta,\theta^{\prime}) . Then, D_{\alpha}(P_{\theta^{\prime}},P_{\theta})\leq\frac{1}{\alpha-1}\log\left(1+\alpha\sup_{z\in(\theta,\theta^{\prime})}\sqrt{I(z)}\int_{\theta}^{\theta^{\prime}}e^{(\alpha-1)U(z,\theta)}dz\right). Proof. For \theta,\theta^{\prime}\in\Theta , we note H(\theta^{\prime},\theta)=\int\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}}dx . Fix \theta\in\Theta and note for every \theta^{\prime}\in\Theta , G(\theta^{\prime})=H(p_{\theta^{\prime}},p_{\theta}) . Then, by Cauchy-Scharz inequality: \displaystyle\partial_{\theta^{\prime}}G(\theta^{\prime}) \displaystyle=\alpha\int\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x)\frac{P_{\theta^{\prime}}(x)^{\alpha}}{P_{\theta}(x)^{1-\alpha}}dx \displaystyle=\alpha\;\mathbb{E}_{X\sim P_{\theta^{\prime}}}\left[\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(X)\left(\frac{P_{\theta^{\prime}}}{P_{\theta}}\right)^{\alpha-1}\right] \displaystyle\leq\alpha\;(\mathbb{E}_{X\sim P_{\theta^{\prime}}}\left[(\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(X))^{2}\right])^{1/2}\left(\mathbb{E}_{X\sim P_{\theta^{\prime}}}\left[\left(\frac{P_{\theta^{\prime}}}{P_{\theta}}\right)^{2(\alpha-1)}\right]\right)^{1/2} \displaystyle\leq\alpha\sqrt{I(\theta^{\prime})}e^{(\alpha-1)D_{2\alpha-1}(P_{\theta^{\prime}},P_{\theta})} \displaystyle\leq\alpha\sup_{z\in(\theta,\theta^{\prime})}\sqrt{I(z)}e^{(\alpha-1)U(\theta^{\prime},\theta)}. Integrating over (\theta,\theta^{\prime}) , we get: G(\theta^{\prime})=G(\theta)+\int_{\theta}^{\theta^{\prime}}\partial_{z}G(z)dz\leq 1+\alpha\sup_{z\in(\theta,\theta^{\prime})}\sqrt{I(z)}\int_{\theta}^{\theta^{\prime}}e^{(\alpha-1)U(z,\theta)}dz, giving the desired result. ∎ This upper bound can be directly leveraged in our setting. In fact, the Rényi divergence D_{2\alpha-1}(ZV,ZW) is upper bounded by post-processing: D_{2\alpha-1}(V^{\top}V,W^{\top}W)\leq D_{2\alpha-1}(V,W) . Therefore, for any choice of parametrization of V^{\top}V , we can obtain a upper bound of D_{2\alpha-1}(V^{\top}V,W^{\top}W) as a function of the post-processing upper bound and the Fisher information related to this parametrization. As a validation step, we can compare this upper bound to the exact Rényi divergence in the case of the Rényi divergence between Gaussian distributions: D_{\alpha}(N+z,N+w)=\frac{\alpha}{2\sigma^{2}}(z-w)^{2} , and the Fisher information is I(z)=\frac{1}{\sigma^{2}} . Then, \displaystyle D_{\alpha}(N+v,N+w) \displaystyle\leq\frac{1}{\alpha-1}\log\left(1+\frac{\alpha}{\sigma}\int e^{(\alpha-1)(2\alpha-1)\frac{(z-w)^{2}}{2\sigma^{2}}}dz\right) \displaystyle\leq\frac{1}{\alpha-1}\log\left(1+\frac{2\alpha\sigma}{(\alpha-1)(2\alpha-1)(v-w)}\left(e^{(\alpha-1)(2\alpha-1)\frac{(v-w)^{2}}{2\sigma^{2}}}-1\right)\right) Figure 5 represent the upper bound and the original divergence in the case \sigma=1,\alpha=2 , as a function of \Delta . Figure 5: Comparison between the Rényi divergence D_{\alpha}(N+v,N+w) and the criterion obtained from Proposition 3.2 as a function of \Delta , in the case \sigma=1,\alpha=2 . Appendix D From Outputs to Sufficient Gram Statistics D.1 Proof of Proposition 4.1 See 4.1 Proof. For n_{\text{syn}}\geq d\geq k , the distributions of ZV admits a density, which is a mixture of matrix normal distributions: ZV|V=v\sim\mathcal{MN}_{n_{\text{syn}}\times k}(0,0,v^{T}v) . The density writes: \displaystyle p_{ZV}(y)=\int p(v)p_{Zv}(y)dv \displaystyle=\int p_{V}(v)(2\pi)^{-n_{\text{syn}}k/2}\det\left(v^{\top}v\right)^{-n_{\text{syn}}/2}\exp\left(-\frac{1}{2}\tr\left((v^{\top}v)^{-1}y^{\top}y\right)\right)dv Now, define g_{v}(z)=\int p_{V}(v)(2\pi)^{-n_{\text{syn}}k/2}\det\left(v^{\top}v\right)^{-n_{\text{syn}}/2}\exp\left(-\frac{1}{2}\tr\left((v^{\top}v)^{-1}z\right)\right)dv . Then, p_{ZV}(y)=g_{v}(y^{\top}y),\;p_{ZW}(y)=g_{w}(y^{\top}y). By the Fisher–Neyman factorization theorem, D_{\alpha}(ZV,ZW)=D_{\alpha}(V^{\top}Z^{\top}ZV,W^{\top}Z^{\top}ZW). ∎ D.2 Proof of Proposition 4.2 See 4.2 Proof. First, observe that \frac{1}{n_{\text{syn}}}Z^{\top}Z=\frac{1}{n_{\text{syn}}}\sum_{i=1}^{n_{\text{syn}}}Z_{i}^{\top}Z_{i}\overset{n_{\text{syn}}\to\infty}{\to}I_{d}, by the law of large numbers. This means that \frac{1}{n_{\text{syn}}}V^{\top}Z^{\top}ZV\to V^{\top}V,\frac{1}{n_{\text{syn}}}W^{\top}Z^{\top}ZW\to W^{\top}W . Also, Rényi divergences are lower semi-continuous for the weak topology ( 48 ) . This means that: \underset{n_{\text{syn}}\to\infty}{\lim}\inf D_{\alpha}(V^{\top}Z^{\top}ZV,W^{\top}Z^{\top}ZW)=\underset{n_{\text{syn}}\to\infty}{\lim}\inf D_{\alpha}\left(\frac{1}{n_{\text{syn}}}V^{\top}Z^{\top}ZV,\frac{1}{n_{\text{syn}}}W^{\top}Z^{\top}ZW\right)\geq D_{\alpha}(V^{\top}V,W^{\top}W). Also, we define the kernel K:\Sigma_{v}\mapsto\frac{1}{n_{\text{syn}}}\mathcal{W}(k,n_{\text{syn}},\Sigma_{v}) , for \Sigma_{v}\in\mathbb{S}_{k}^{+}(\mathbb{R}) , or equivalently: K(\Sigma_{v})=\Sigma_{v}^{1/2}G^{\top}G\Sigma_{v}^{1/2} . Now, we note \Sigma_{v}=V^{\top}V . Then, V admits a polar decomposition V=U_{v}\Sigma_{v}^{1/2} , where U\in\mathbb{R}^{d\times k} verifies U_{v}^{\top}U_{v}=I_{k} . Let G\in\mathbb{R}^{n_{\text{syn}}\times k} be a standard Gaussian matrix. By invariance of Gaussian matrices by isometries, ZV=ZU_{v}\Sigma_{v}^{1/2}\overset{d}{=}G\Sigma_{v}^{1/2} , and: \displaystyle D_{\alpha}(V^{\top}V,W^{\top}W)\geq D_{\alpha}(K(V^{\top}V),K(W^{\top}W)) \displaystyle=D_{\alpha}\left(\Sigma_{v}^{1/2}G^{\top}G\Sigma_{v}^{1/2},\Sigma_{w}^{1/2}G^{\top}G\Sigma_{w}^{1/2}\right) \displaystyle=D_{\alpha}\left(V^{\top}U_{v}^{\top}Z^{\top}ZU_{v}V,W^{\top}U_{w}^{\top}Z^{\top}ZU_{w}W\right) \displaystyle=D_{\alpha}\left(V^{\top}Z^{\top}ZV,W^{\top}Z^{\top}ZW\right). Finally, D_{\alpha}(V^{\top}Z^{\top}ZV,W^{\top}Z^{\top}ZW)\leq D_{\alpha}(V^{\top}V,W^{\top}W)\leq\underset{n_{\text{syn}}\to\infty}{\lim}\inf D_{\alpha}(V^{\top}Z^{\top}ZV,W^{\top}Z^{\top}ZW), giving the desired result. ∎ D.3 Proof of Proposition 4.3 See 4.3 Proof. The idea of the proof is to take v_{\theta}=te_{1},w_{t}=(t+\Delta)e_{1} , where e_{1}=(\delta_{11})\in\mathbb{R}^{d\times k} . We note V_{t}=N+v_{t} and W_{t}=N+w_{t} . First, by post-processing: D_{\alpha}(V_{t}^{\top}V_{t},W_{t}^{\top}W_{t})\geq D_{\alpha}((V_{t}^{\top}V_{t})_{11},(W_{t}^{\top}W_{t})_{11}))=D_{\alpha}(\chi_{d}^{2}(t^{2}),\chi_{d}^{2}((t+\Delta)^{2})). Now, take f:x\mapsto\sqrt{|x|} . As t\to\infty , f becomes injective on a set of probability p_{t}\to 1 . The distribution of f(\chi_{d}^{2}(t^{2})) converges weakly to \mathcal{N}(t,1) . Then, for all t>0 : \underset{t\to\infty}{\lim}\inf D_{\alpha}(f(\chi_{d}^{2}(t)),f(\chi_{d}^{2}((t+\Delta)^{2}))\geq D_{\alpha}(\mathcal{N}(t,1),\mathcal{N}(t+\Delta,1))=D_{\alpha}(V,W), which is finite, giving the desired result. ∎ Appendix E Privacy amplification in linear regression E.1 Computing the Fisher information of \chi_{d}^{2}(\theta^{2}) for the amplitude parameter \theta In this section, we restate the introduction of Section 5.1 and add relevant results and details. The Fisher information of non-central chi-squared distribution for the amplitude parameter does not seem tractable ( 26 ) . As a result, most of the literature have focused either on deriving lower bounds ( 12 ; 26 ; 47 ) or on numerical estimation ( 12 ) . In contrast, the literature on upper bounds remains limited, aside from the trivial post-processing inequality I(d,\theta)\leq 1 . In the work ( 26 ) , authors focus on the Fisher information of Rician and generalized Rician distributions with respect to the amplitude parameter \theta . Generalized Rician distributions R(\alpha,\theta) reduce to non-central chi distribution when the scale parameter \alpha=1 or equivalently, to square root of non-central chi-squared random variables. Denoting I_{X}(d,\theta) the Fisher information of generalized Rician distributions for the parameter \theta , and noting that x\mapsto\sqrt{x} is bijective, we have I(d,\theta)=I_{X}(d,\theta) . Then, computing a upper bound for I_{X}(d,\theta) also give a upper bound to I(\theta) . 26 find the following upper bound: I_{X}(d,\theta)\leq\min\left\{1,\frac{2\theta^{2}}{d}\right\}. While useful in the d\gg\theta^{2} regime, this bound is piecewise and does not naturally capture the asymptotic behaviour I_{X}(d,\theta)\overset{\theta\to+\infty}{\to}1 . In this section, we propose a tighter upper bound to I_{X}(\theta) , and therefore to I(d,\theta) by leveraging the properties of the modified Bessel function of first kind I_{d} . Proposition E.1 (Fisher information of non-central \chi_{d}^{2} , adapted from 26 ) . Let d>1 . Then, the Fisher information admits the representation: I_{X}(\theta)=\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}[Y(R_{d/2-1}(Y)-R_{d/2}(Y))], where R_{d}=\frac{I_{d+1}}{I_{d}} is the modified Bessel quotient of the first kind and \mathcal{R}_{d/2,\theta} is a generalized Rician distribution with non-centrality parameter \theta^{2} , scale \theta and degree d/2 ( 26 ) : \mathcal{R}_{d/2,\theta}(y)=\frac{y^{d/2+1}}{\theta^{d+2}}e^{-\frac{y^{2}}{2\theta^{2}}-\theta^{2}/2}I_{d/2}(y). Motivated by this characterization, we seek to improve upon the upper bounds of 26 by deriving a sharp upper bound on Y(R_{d/2-1}(Y)-R_{d/2}(Y)) : Lemma E.1 . Let d>2,x>0 . Then, x(R_{d-1}(x)-R_{d}(x))\leq\frac{xR_{d-1}(x)}{xR_{d-1}(x)+d-\frac{1}{2}}. Proof. We note R_{d}(x)=\frac{I_{d+1}(x)}{I_{d}(x)} the modified Bessel quotient of the first kind. The modified Bessel functions of first kind admit the following recurrence relations ( 5 , 11.115 and 11.116) : I_{d-1}(x)-I_{d+1}(x)=\frac{2d}{x}I_{d}(x). I_{d-1}(x)+I_{d+1}(x)=2I_{d}^{\prime}(x). Then, we relate x(R_{d-1}(x)-R_{d}(x)) to I_{d}(x)^{2}-I_{d+1}(x)I_{d-1}(x) : \displaystyle x(R_{d-1}(x)-R_{d}(x)) \displaystyle=x\left(\frac{I_{d}(x)}{I_{d-1}(x)}-\frac{I_{d+1}(x)}{I_{d}(x)}\right) \displaystyle=x\frac{I_{d}(x)^{2}-I_{d+1}(x)I_{d-1}(x)}{I_{d-1}(x)I_{d}(x)} \displaystyle=xR_{d-1}(x)\frac{I_{d}(x)^{2}-I_{d+1}(x)I_{d-1}(x)}{I_{d}(x)^{2}}. Since xR_{d}(x)\geq 0 , to prove the lemma, we prove that: \frac{I_{d}(x)^{2}-I_{d+1}(x)I_{d-1}(x)}{I_{d}(x)^{2}}\leq\frac{1}{xR_{d-1}(x)+d-\frac{1}{2}}. Leveraging Theorem 1 from 6 , we have: I_{d}(x)^{2}-I_{d-1}(x)I_{d+1}(x)\leq\frac{1}{\sqrt{x^{2}+d^{2}-\frac{1}{4}}}I_{d}(x)^{2}. Then, to get the desired result, we must show that: \sqrt{x^{2}+d^{2}-\frac{1}{4}}\geq xR_{d-1}(x)+d-\frac{1}{2}. 45 proves the following inequality [p. 526]: x\frac{I_{d-1}^{\prime}(x)}{I_{d-1}(x)}\leq\sqrt{x^{2}+\left(d-\frac{1}{2}\right)^{2}}-\frac{1}{2}. Then leveraging the Bessel function recurrence relationships and d>1 : \displaystyle xR_{d-1}(x)+d-\frac{1}{2}=x\frac{I_{d-1}^{\prime}(x)}{I_{d-1}(x)}+\frac{1}{2} \displaystyle\leq\sqrt{x^{2}+\left(d-\frac{1}{2}\right)^{2}} \displaystyle\leq\sqrt{x^{2}+d^{2}-d+\frac{1}{4}} \displaystyle\leq\sqrt{x^{2}+d^{2}-\frac{1}{4}}. Then, we obtain the result: x(R_{d-1}(x)-R_{d}(x))\leq\frac{xR_{d-1}(x)}{xR_{d-1}(x)+d-\frac{1}{2}}. ∎ Now, we can upper bound the Fisher information: See 5.1 Proof. Let \theta>0,d>2 . We have, by concavity of x\mapsto\frac{x}{x+c} : \displaystyle I_{X}(\theta) \displaystyle=\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}[Y(R_{d/2-1}(Y)-R_{d/2}(Y))] \displaystyle\leq\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[\frac{YR_{d/2-1}(Y)}{YR_{d/2-1}(Y)+\frac{d-1}{2}}\right]\;\;\;\;\;\text{ (Lemma~\ref{lem:bessel-upper-bound})} \displaystyle\leq\frac{\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[YR_{d/2-1}(Y)\right]}{\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[YR_{d/2-1}(Y)\right]+\frac{d-1}{2}}.\;\;\;\;\,\text{ (Jensen inequality)} ∎ From there, we could simply use YR_{d/2-1}(Y)\leq Y and the fact that x\mapsto\frac{x}{x+c} is increasing for x\in\mathbb{R}^{+} to get the simple upper bound: I(\theta)\leq\frac{\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[Y\right]}{\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[Y\right]+\frac{d-1}{2}}\leq\frac{\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[Y^{2}\right]^{1/2}}{\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[Y^{2}\right]^{1/2}+\frac{d-1}{2}}\leq\frac{\theta(\theta+\sqrt{d+2})}{\theta(\theta+\sqrt{d+2})+\frac{d-1}{2}}, but we lose a d^{1/2} factor. Instead, we use the expectation characterization of I_{X}(\theta) to obtain a tighter upper bound. We compute \mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[YR_{d/2-1}(Y)\right] . The random variable Y=\theta\sqrt{X} follows a generalized Rician distribution \mathcal{R}_{d/2,\theta} , where X\sim\chi_{d+2}(\theta^{2}) ( 26 ) . We first write the integral expansion of I_{X}(\theta) : \displaystyle I_{X}(\theta) \displaystyle=\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}[Y(R_{d/2-1}(Y)-R_{d/2}(Y))] \displaystyle=\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}[YR_{d/2-1}(Y)]-\int yR_{d/2}(y)\frac{y^{d/2+1}}{\theta^{d+2}}e^{-\frac{y^{2}}{2\theta^{2}}-\theta^{2}/2}I_{d/2}(y)dy \displaystyle=\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}[YR_{d/2-1}(Y)]-\int\frac{y^{d/2+2}}{\theta^{d+2}}e^{-\frac{y^{2}}{2\theta^{2}}-\theta^{2}/2}I_{d/2+1}(y)dy \displaystyle=\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}[YR_{d/2-1}(Y)]-\theta^{2}\int P_{\theta\chi_{d+4}(\theta^{2})}(y)dy \displaystyle=\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}[YR_{d/2-1}(Y)]-\theta^{2}. Therefore, \mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}[YR_{d/2-1}(Y)]=I_{X}(\theta)+\theta^{2}. Putting all together, we have: I_{X}(\theta)\leq\frac{\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[YR_{d/2-1}(Y)\right]}{\mathbb{E}_{Y\sim\mathcal{R}_{d/2,\theta}}\left[YR_{d/2-1}(Y)\right]+\frac{d-1}{2}}\leq\frac{I_{X}(\theta)+\theta^{2}}{I_{X}(\theta)+\theta^{2}+\frac{d-1}{2}}. Solving this quadratic inequality, we obtain: I_{X}(\theta)\leq\frac{1}{2}\left(\frac{3}{2}-d-\theta^{2}+\sqrt{\left(\theta^{2}+\frac{d-3}{2}\right)^{2}+4\theta^{2}}\right), or, equivalently: I_{X}(\theta)\leq\frac{2\theta^{2}}{\theta^{2}+d-\frac{3}{2}+\sqrt{\left(\theta^{2}+\frac{d-3}{2}\right)^{2}+4\theta^{2}}}, This upper bound is better than the I_{X}(\theta)\leq\frac{2\theta^{2}}{d} bound of 26 for every \theta>0,d>6 . Using the identity x\leq\sqrt{x^{2}+y^{2}} , we obtain: I_{X}(\theta)\leq\frac{2\theta^{2}}{2\theta^{2}+d-3}, giving the simplified desired result. This upper bound is tighter than the I_{X}(\theta)\leq\frac{2\theta^{2}}{d} bound of 26 for every \theta>\sqrt{\frac{3}{2}} . E.2 Proof of Theorem 5.2 Before proving the theorem, here is a sharp elementary upper bound to \int_{0}^{x}e^{\sigma t^{2}}dt : Lemma E.2 . Let \sigma,x>0 . Then: \int_{0}^{x}e^{\sigma t^{2}}dt\leq\frac{e^{\sigma x^{2}}-1}{\sigma x}. See 5.2 Proof. For generality, we do not set in this proof \sigma=1 . We leverage the inequality of Lemma E.2 : D_{\alpha}(V^{\top}V,W^{\top}W)\leq\frac{1}{\alpha-1}\log\left(1+\alpha\sup_{z\in(v,w)}\sqrt{I(z)}\int_{v}^{w}e^{(\alpha-1)D_{2\alpha-1}(\|N+z\|_{F}^{2},W^{\top}W)}dz\right). Using the post-processing property: \sup_{z\in(v,w)}D_{2\alpha-1}(\|N+z\|_{F}^{2},W^{\top}W)\leq(2\alpha-1)\frac{(\|w\|_{F}-z)^{2}}{2\sigma^{2}}. We get: \displaystyle\int_{v}^{w}e^{(\alpha-1)\sup_{z\in(v,w)}D_{2\alpha-1}(\|N+z\|_{F}^{2},WW^{T})}dz \displaystyle=\int_{v}^{w}e^{(\alpha-1)(2\alpha-1)\frac{(z-w)^{2}}{\sigma^{2}}}dz \displaystyle=\int_{0}^{v-w}e^{(\alpha-1)(2\alpha-1)\frac{z^{2}}{\sigma^{2}}}dz \displaystyle\leq\sigma^{2}\frac{e^{(\alpha-1)(2\alpha-1)\frac{(\|v\|_{F}-\|w\|_{F})^{2}}{\sigma^{2}}}-1}{2(\alpha-1)(2\alpha-1)(\|v\|_{F}-\|w\|_{F})}. Then, leveraging Proposition 5.1 : \sup_{z\in(v,w)}\sqrt{I(z)}\leq\left(\frac{2C^{2}}{2C^{2}+d-3}\right)^{1/2}. Finally: D_{\alpha}(V^{\top}V,W^{\top}W)\leq\frac{1}{\alpha-1}\log\left(1+\alpha\sigma^{2}\left(\frac{2C^{2}}{2C^{2}+d-3}\right)^{1/2}\frac{e^{(\alpha-1)(2\alpha-1)\frac{(\|v\|_{F}-\|w\|_{F})^{2}}{2\sigma^{2}}}-1}{(\alpha-1)(2\alpha-1)(\|v\|_{F}-\|w\|_{F})}\right), ∎ E.3 Interpretation of the Fisher comparison principle and Theorem 5.2 Then, we can also represent the privacy guarantees obtained from Theorem 5.2 as a function of d , as shown in Figure 6 , illustrating the privacy gains. Figure 6: Privacy bounds of Theorem 5.2 as a function of d for multiple values of \Delta , C=\sigma=1 . Appendix F Privacy amplification in multivariate linear regression In this section, we note P_{v} the distribution of (N+v)^{\top}(N+v) , which is a non-central Wishart distribution \mathcal{W}(n,d,\Omega_{v}) with non-centrality parameter \Omega_{v}=v^{\top}v : P_{v}(y)=P_{0}(y)e^{-tr(\Omega_{v})/2}{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}y^{1/2}\Omega_{v}y^{1/2}\right), where P_{0} is the central Wishart density \mathcal{W}(k,d) , and {}_{0}F_{1} is the generalized hypergeometric function. We state some properties of {}_{0}F_{1} : Proposition F.1 (Properties of {}_{0}F_{1} ( 39 ) ) . {}_{0}F_{1} has the following properties: • Conjugation invariance: Let U\sim O(k) , then: {}_{0}F_{1}(a,Z)={}_{0}F_{1}(a,UZU^{T}). • Dependence on eigenvalues: if Z,Z^{\prime} have the same eigenvalues with the same algebraic multiplicities, then: {}_{0}F_{1}(a,Z)={}_{0}F_{1}(a,Z^{\prime}). F.1 Proof of Theorem 5.3 See 5.3 Proof. Let \Sigma_{v},\Sigma_{w}\in\mathbb{R}^{d\times k} be two rectangular diagonal matrices. Let S_{v},S_{w}\in O(k) and U_{v},U_{w}\in O(d) be orthogonal matrices. We note v=U_{v}^{\top}\Sigma_{v}S_{v},w=U_{w}^{\top}\Sigma_{w}S_{w} . Let \theta\in(0,1) . Let U\in O(d) be an orthogonal matrix. Define f:\omega_{1},\dots,\omega_{k+1}\mapsto\sum_{i=1}^{k+1}\omega_{i} . Now, let \Sigma_{\theta}\in\mathbb{R}^{n\times d} such that \Sigma_{0}=\Sigma_{v} and \Sigma_{1}=\Sigma_{w} . Let S_{\theta}\in O(k) such that S_{0}=I_{k} and S_{1}=S_{w} , and U_{\theta}\in O(d) such that U_{0}=U_{v} and U_{1}=U_{w}U . For i\in\{1,\dots,d\} , we note \mu(\theta)=\Sigma_{\theta}S_{\theta} and v_{\theta}=U_{\theta}\mu(\theta) Since \Sigma_{\theta} is rectangular diagonal, \mu(\theta) has the form: \mu(\theta)=\begin{pmatrix}M(t)\\ 0_{(d-k)\times k}\end{pmatrix} . \mu_{i}(\theta)\in\mathbb{R}^{k} is a column vector. Also, we note V_{\theta}=N+v_{\theta} . Then, by invariance of Gaussian matrices by orthogonal transformations: V_{\theta}^{\top}V_{\theta}=(N+U_{\theta}\mu(\theta))^{\top}(N+U_{\theta}\mu(\theta))\overset{d}{=}(N+\mu(\theta))^{\top}(N+\mu(\theta)) We define \omega_{i}=(N_{i}+\mu_{i}(\theta)^{\top})^{\top}(N_{i}+\mu_{i}(\theta))+\sum_{j=1}^{\lfloor d/k\rfloor-1}N^{\top}_{jk+i}N_{jk+i} . Then, we have: \displaystyle V_{\theta}^{\top}V_{\theta} \displaystyle=\left(\sum_{i=1}^{k}\left((V_{\theta}^{\top})_{i}(V_{\theta})_{i}+\sum_{j=1}^{\lfloor d/k\rfloor-1}(V_{\theta}^{\top})_{jk+i}(V_{\theta})_{jk+i}\right)\right)+\sum_{i=k\lfloor d/k\rfloor}^{d}(V_{\theta})^{\top}_{i}(V_{\theta})_{i} \displaystyle=\left(\sum_{i=1}^{k}\left((N_{i}+\mu_{i}(\theta)^{\top})^{\top}(N_{i}+\mu_{i}(\theta)^{\top})+\sum_{j=1}^{\lfloor d/k\rfloor-1}N^{\top}_{jk+i}N_{jk+i}\right)\right)+\sum_{i=k\lfloor d/k\rfloor}^{d}N^{\top}_{i}N_{i} \displaystyle=f\left(\omega_{1},\dots,\omega_{k},\sum_{i=k\lfloor d/k\rfloor}^{d}N^{\top}_{i}N_{i}\right) Then, by post-processing and independence of the packs: \displaystyle I(\theta) \displaystyle=I(\theta,f(\mathcal{W}(k,\lfloor d/k\rfloor,\mu_{1}(\theta)\mu_{1}(\theta)^{\top}),\dots\mathcal{W}(k,\lfloor d/k\rfloor,\mu_{k}(\theta)\mu_{k}(\theta)^{\top}),\mathcal{W}(k,d-k\lfloor d/k\rfloor,0))) \displaystyle\leq\sum_{i=1}^{k}I(\theta,\mathcal{W}(k,\lfloor d/k\rfloor,\mu_{i}(\theta)\mu_{i}(\theta)^{\top})) We note m=\lfloor d/k\rfloor . In the pack \mathcal{W}(k,m,\mu_{i}(\theta)\mu_{i}(\theta)^{\top}) , we note G\in\mathbb{R}^{m\times k} a Gaussian matrix and X_{i}(\theta)=G+e_{1}\mu_{i}(\theta)^{\top} . Then, X_{i}(\theta)^{\top}X_{i}(\theta)\sim\mathcal{W}(k,m,\mu_{i}(\theta)\mu_{i}(\theta)^{\top}) . Now, we compute a upper bound to the Fisher information I(\theta,\mathcal{W}(k,m,\mu_{i}(\theta)\mu_{i}(\theta)^{\top}) of \mathcal{W}(k,m,\mu_{i}(\theta)\mu_{i}(\theta)^{\top}) for the parameter \theta . The density writes: p_{t}(y)=p_{0}(y)e^{-\tr(\mu_{i}(\theta)^{\top}\mu_{i}(\theta))/2}{}_{0}F_{1}\left(\frac{m}{2},\frac{1}{4}y^{1/2}\mu_{i}(\theta)\mu_{i}(\theta)^{\top}y^{1/2}\right). Since \mu_{i} is a vector, the {}_{0}F_{1} term reduces to the scalar {}_{0}F_{1} function of its single non-zero eigenvalue: {}_{0}F_{1}\left(\frac{m}{2},\frac{1}{4}y^{1/2}\mu_{i}(\theta)\mu_{i}(\theta)^{\top}y^{1/2}\right)={}_{0}F_{1}\left(\frac{m}{2},\frac{1}{4}\mu_{i}(\theta)^{\top}y\mu_{i}(\theta)\right). In fact, y^{1/2}\mu_{i}(\theta)\mu_{i}(\theta)^{\top}y^{1/2} has \mu_{i}(\theta)^{\top}y\mu_{i}(\theta) as only non-zero eigenvalue. We note, for x>0 , F(d,x)=\log{}_{0}F_{1}(d,x) . Also, we note \mu_{i}(\theta)=\rho_{i}(\theta)u_{i}(\theta) , where \rho_{i}(\theta)>0 and \|u_{i}(\theta)\|=1 . This parametrization is interesting because \langle u_{i}(\theta),\dot{u}_{i}(\theta)\rangle=0 , so that \langle\mu_{i}(\theta),\dot{\mu}_{i}(\theta)\rangle=\rho_{i}(\theta)\dot{\rho}_{i}(\theta) . Then, \dot{\mu}_{i}(\theta)=\dot{\rho}_{i}(\theta)u_{i}(\theta)+\rho_{i}(\theta)\dot{u}_{i}(\theta) , \mu_{i}(\theta)^{\top}y\mu_{i}(\theta)=\rho_{i}(\theta)^{2}q_{\theta} and \mu_{i}(\theta)^{\top}y\mu_{i}(\theta)=\rho_{i}(\theta)(\dot{\rho}_{i}(\theta)u(t)^{\top}+\rho_{i}(\theta)\dot{u}(t))yU_{\theta}(t)=\rho_{i}(\theta)^{2}r_{\theta}+\rho_{i}(\theta)\dot{\rho}_{i}(\theta)q_{\theta} . We also write q_{\theta}=u_{i}(\theta)^{\top}yu_{i}(\theta)=\|X_{i}(\theta)u_{i}(\theta)\|^{2} and r_{\theta}=\dot{u}_{i}(\theta)^{\top}yu_{i}(\theta)=\langle X_{i}(\theta)u_{i}(\theta),X_{i}(\theta)\dot{u}_{i}(\theta)\rangle . Then, the score function writes: \displaystyle\partial_{\theta}\log p_{t}(y) \displaystyle=-\frac{1}{2}\partial_{\theta}\|\Omega_{\theta}\|_{F}^{2}+\partial_{\theta}\log{}_{0}F_{1}\left(\frac{m}{2},\frac{1}{4}\mu_{i}(\theta)^{\top}y\mu_{i}(\theta)\right) \displaystyle=-\langle\mu_{i}(\theta),\dot{\mu}_{i}(\theta)\rangle+\frac{1}{2}F^{\prime}\left(\frac{m}{2},\frac{1}{4}\mu_{i}(\theta)^{\top}y\mu_{i}(\theta)\right)\dot{\mu}_{i}(\theta)^{\top}y\mu_{i}(\theta) \displaystyle=-\langle\mu_{i}(\theta),\dot{\mu}_{i}(\theta)\rangle+\frac{1}{2}F^{\prime}\left(\frac{m}{2},\frac{\rho_{i}(\theta)^{2}}{4}q_{\theta}\right)(\rho_{i}(\theta)^{2}r_{\theta}+\rho_{i}(\theta)\dot{\rho}_{i}(\theta)q_{\theta}), Also, we relate the scalar {}_{0}F_{1} function to modified Bessel functions of first kind: {}_{0}F_{1}\left(\frac{m}{2},x\right)=\Gamma(m/2)x^{1/2-m/4}I_{m/2-1}(2\sqrt{x}). Differentiating \log{}_{0}F_{1}(m/2,x) and leveraging the relationship 2\sqrt{x}\frac{I_{m/2-1}^{\prime}(2\sqrt{x})}{I_{m/2-1}(2\sqrt{x})}=\frac{m}{2}-1+2\sqrt{x}R_{m/2-1}(2\sqrt{x}) from the proof of Proposition 5.1 , we get: \displaystyle G(m/2,x) \displaystyle=\frac{1-\frac{m}{2}}{2x}+\frac{1}{\sqrt{x}}\frac{I_{m/2-1}^{\prime}(2\sqrt{x})}{I_{m/2-1}(2\sqrt{x)}}=\frac{1}{\sqrt{x}}R_{m/2-1}(2\sqrt{x}). Then, the score function writes: \partial_{\theta}\log p_{t}(y)=\dot{\rho}_{i}(\theta)\left(-\rho_{i}(\theta)+\sqrt{q_{\theta}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)+\frac{\rho_{i}(\theta)r_{\theta}}{\sqrt{q_{\theta}}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right). Now, we observe that X_{i}(\theta)u_{i}(\theta) is a Gaussian vector. Then, q_{\theta}=\|X_{i}(\theta)u_{i}(\theta)\|^{2}=\|Gu_{i}(\theta)+\rho_{i}(\theta)e_{1}\|^{2} follows a non-central chi-squared distribution \chi_{m}^{2}(\rho_{i}(\theta)^{2}) with non-centrality \rho_{i}(\theta)^{2} and degree of freedom m . Moreover, we observe that X_{t}u_{i}(\theta) and X_{t}\dot{u}_{i}(\theta) are independent. In fact, \langle u_{i}(\theta),\dot{u}_{i}(\theta)\rangle=0 . Also, X_{i}(\theta)\dot{u}_{i}\sim\mathcal{N}(\rho_{i}(\theta)\langle u_{i}(\theta),\dot{u}_{i}(\theta)\rangle e_{1},\|\dot{u}_{i}(\theta)\|^{2}I_{m})=\mathcal{N}(0,\|\dot{u}_{i}(\theta)\|^{2}I_{m}) . Then, we have: r_{\theta}|\{X_{i}(\theta)u_{i}(\theta)=z\}=\langle X_{i}(\theta)u_{i}(\theta),X_{i}(\theta)\dot{u}_{i}(\theta)\rangle|\{X_{i}(\theta)u_{i}(\theta)=z\}\sim\mathcal{N}(0,z^{2}\|\dot{u}_{i}(\theta)\|^{2}). In particular, we have: \mathbb{E}[r_{\theta}|q_{\theta}]=0,\;\;\mathbb{E}[r_{\theta}^{2}|q_{\theta}]=q_{\theta}\|\dot{u}_{i}(\theta)\|^{2}. Also, we recall that \|\dot{\mu}_{i}(\theta)\|^{2}=\|\dot{\rho}_{i}(\theta)u_{i}(\theta)+\rho_{i}(\theta)\dot{u}_{i}(\theta)\|^{2}=\dot{\rho}_{i}(\theta)^{2}+\rho_{i}(\theta)\|\dot{u}_{i}(\theta)\|^{2} . We can now compute the Fisher information: \displaystyle I(t,\mathcal{W}(k,\lfloor d/k\rfloor,\mu_{i}(\theta)\mu_{i}(\theta)^{\top}) \displaystyle=\mathbb{E}\left[\left(\dot{\rho}_{i}(\theta)\left(-\rho_{i}(\theta)+\sqrt{q_{\theta}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)+\frac{\rho_{i}(\theta)r_{\theta}}{\sqrt{q_{\theta}}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)^{2}\right] \displaystyle=\mathbb{E}\left[\mathbb{E}\left[\left(\dot{\rho}_{i}(\theta)\left(-\rho_{i}(\theta)+\sqrt{q_{\theta}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)+\frac{\rho_{i}(\theta)r_{\theta}}{\sqrt{q_{\theta}}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)^{2}\Bigg|q_{\theta}\right]\right] \displaystyle=\mathbb{E}\left[\dot{\rho}_{i}(\theta)^{2}\left(-\rho_{i}(\theta)+\sqrt{q_{\theta}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)^{2}\right] \displaystyle+2\mathbb{E}\left[\left(\dot{\rho}_{i}(\theta)\left(-\rho_{i}(\theta)+\sqrt{q_{\theta}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)\frac{\rho_{i}(\theta)\mathbb{E}[r_{\theta}|q_{\theta}]}{\sqrt{q_{\theta}}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)^{2}\right] \displaystyle+\mathbb{E}\left[\frac{\rho_{i}(\theta)^{2}\mathbb{E}[r_{\theta}^{2}|q_{\theta}]}{q_{\theta}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)^{2}\right] \displaystyle=\dot{\rho}_{i}(\theta)^{2}\mathbb{E}\left[\left(-\rho_{i}(\theta)+\sqrt{q_{\theta}}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)\right)^{2}\right]+\rho_{i}(\theta)^{2}\|\dot{u}_{i}(\theta)\|^{2}\mathbb{E}\left[R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)^{2}\right] Now, we relate this expectation to the Fisher information of a non-central chi-squared distribution \chi_{m}^{2}(\rho_{i}(\theta)^{2}) for the amplitude parameter \rho_{i}(\theta) . The density writes: p_{\rho_{i}(\theta)}(x)=\frac{1}{2}e^{-(x+\rho_{i}(\theta)^{2})/2}\left(\frac{x}{\rho_{i}(\theta)^{2}}\right)^{m/4-1/2}I_{m/2-1}\big(\rho_{i}(\theta)\sqrt{x}\big). Then, the score function is: \displaystyle\partial_{\rho_{i}(\theta)}\log p_{\rho_{i}(\theta)}(x)=-\rho_{i}(\theta)+\frac{1-m/2}{\rho_{i}(\theta)}+\sqrt{x}\frac{I_{m/2-1}^{\prime}(\rho_{i}(\theta)\sqrt{x})}{I_{m/2-1}(\rho_{i}(\theta)\sqrt{x})}=-\rho_{i}(\theta)+\sqrt{x}R_{m/2-1}(\rho_{i}(\theta)\sqrt{x}). Then, noting Y\sim\chi_{m}^{2}(\rho_{i}(\theta)^{2}) and leveraging \mathbb{E}_{Y}[\partial_{\rho_{i}(\theta)}\log p_{\rho_{i}(\theta)}(Y)]=0 , the associated Fisher information I(\rho_{i}(\theta)) is equal to: \displaystyle I(\rho_{i}(\theta)) \displaystyle=\mathbb{E}_{Y}\left[\left(-\rho_{i}(\theta)+\sqrt{Y}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{Y}\right)\right)^{2}\right] \displaystyle=\rho_{i}(\theta)^{2}+\mathbb{E}_{Y}\left[\left(\sqrt{Y}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{Y}\right)\right)^{2}\right]-2\rho_{i}(\theta)\mathbb{E}_{Y}\left[\sqrt{Y}R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{Y}\right)\right] \displaystyle=\mathbb{E}_{Y}\left[YR_{m/2-1}\left(\rho_{i}(\theta)\sqrt{Y}\right)^{2}\right]-\rho_{i}(\theta)^{2}. Then, the Fisher information writes: I(\theta,\mathcal{W}(k,\lfloor d/k\rfloor,\mu_{i}(\theta)\mu_{i}(\theta)^{\top})=\dot{\rho}_{i}(\theta)^{2}I(\rho_{i}(\theta))+\rho_{i}(\theta)^{2}\|\dot{u}_{i}(\theta)\|^{2}\mathbb{E}\left[R_{m/2-1}\left(\rho_{i}(\theta)\sqrt{q_{\theta}}\right)^{2}\right]. Now, we connect I(\rho_{i}(\theta)) to \mathbb{E}\left[R_{m/2-1}(2\sqrt{q_{\theta}})^{2}\right]=\mathbb{E}_{Y}\left[R_{m/2-1}\left(2\sqrt{Y}\right)^{2}\right] . Since R_{m/2-1} is increasing, we can apply the Chebyshev integral inequality: \displaystyle\mathbb{E}\left[R_{m/2-1}\left(2\sqrt{q_{\theta}}\right)^{2}\right]\leq\frac{\mathbb{E}_{Y}\left[YR_{m/2-1}\left(2\rho_{i}(\theta)\sqrt{Y}\right)^{2}\right]}{\mathbb{E}_{Y}\left[Y\right]}=\frac{I(\rho_{i}(\theta))+\rho_{i}(\theta)^{2}}{m+\rho_{i}(\theta)^{2}} \displaystyle\leq\frac{\rho_{i}(\theta)^{2}(2\rho_{i}(\theta)^{2}+m-1)}{(2\rho_{i}(\theta)^{2}+m-3)(\rho_{i}(\theta)^{2}+m)} \displaystyle\leq\frac{2\rho_{i}(\theta)^{2}}{2\rho_{i}(\theta)^{2}+m-3}, leveraging the Fisher information upper bound for non-central chi-squared distributions of 3.2 I(v_{i})\leq\frac{2v_{i}^{2}}{2v_{i}^{2}+m-3} . Therefore, \displaystyle I(\theta,\mathcal{W}(k,\lfloor d/k\rfloor,U_{\theta}^{\top}\Lambda^{\top}_{i}\Lambda_{i}U_{\theta})) \displaystyle\leq\dot{\rho}_{i}(\theta)^{2}I(\rho_{i}(\theta))+\rho_{i}(\theta)^{2}\|\dot{u}_{i}(\theta)\|^{2}\frac{2\rho_{i}(\theta)^{2}}{2\rho_{i}(\theta)^{2}+m-3} \displaystyle\leq(\dot{\rho}_{i}(\theta)^{2}+\rho_{i}(\theta)^{2}\|\dot{u}_{i}(\theta)\|^{2})\frac{2\rho_{i}(\theta)^{2}}{2\rho_{i}(\theta)^{2}+m-3} \displaystyle\leq\|\dot{\mu}_{i}(\theta)\|^{2}\frac{2\rho_{i}(\theta)^{2}}{2\rho_{i}(\theta)^{2}+m-3}. From there, we upper bound the total Fisher information by leveraging the concavity of x\mapsto\frac{x}{x+c} on \mathbb{R}^{+} for the inputs \|\dot{\mu}_{i}(\theta)\|^{2} : \displaystyle I(\theta) \displaystyle\leq\sum_{i=1}^{k}\|\dot{\mu}_{i}(\theta)\|^{2}\frac{2\rho_{i}(\theta)^{2}}{2\rho_{i}(\theta)^{2}+m-3} (2) \displaystyle\leq\|\dot{\mu}(\theta)\|_{F}^{2}\frac{2\sum_{i=1}^{k}\rho_{i}(\theta)^{2}\dot{\mu}_{i}(\theta)^{2}/\|\dot{\mu}(\theta)\|_{F}^{2}}{\sum_{i=1}^{k}\rho_{i}(\theta)^{2}\dot{\mu}_{i}(\theta)^{2}/\|\dot{\mu}(\theta)\|_{F}^{2}+m-3} (3) \displaystyle\leq\|\dot{\mu}(\theta)\|_{F}^{2}\frac{\rho(\theta)^{2}}{\rho(\theta)^{2}+m-3}. (4) Now, taking \mu(\theta)=(1-\theta)v+\theta Uw , we get \dot{\mu}(\theta)=Uw-v , and \rho(\theta)=\|(1-\theta)v+\theta Uw\|_{F} , giving the desired result. ∎ F.2 Upper bounding the divergence in the general case F.3 Proof of Proposition 5.1 In this section, we outline another upper bound for the Fisher information of non-central chi-squared distributions. This upper bound is worse than the upper bound of Proposition 5.1 , but we think that the proof technique is worth of noting. In fact, the proof technique is different and relies on a Poincaré inequality step. We highlight this proof technique because we think that a similar proof could be developed for non-central Wishart distributions, giving a potential better Fisher information than the one we obtain in this paper. To develop the same upper bounds for non-central Wishart distribution, we need a control on the spectrum of the Hessian of {}_{0}F_{1} , which appears when performing the Poincaré inequality step. We believe that this could be done through the integral representation of {}_{0}F_{1} (Proposition H.3 ). Proposition F.2 . (Fisher information of non-central \chi^{2} ). For all \theta>0 and p_{\theta}\in P , and : I(\theta)\leq\frac{2\theta^{2}(\theta^{2}+2d)}{2d^{2}+\theta^{2}(\theta^{2}+2d)}. Proof. We note \lambda=\theta^{2}/2 . Let \pi_{\lambda}(i)=e^{-\lambda}\lambda^{i}/i! be the Poisson probability mass function and f_{i}=\frac{x^{i/2-1}e^{-x/2}}{2^{i/2}\Gamma(i/2)} the chi-squared density with i degrees of freedom. We write non-central chi squared random variables as Poisson-weighted mixture of central chi squared random variables: p_{\theta}=\sum_{i=0}^{+\infty}\pi_{\theta^{2}/2}(i)f_{d+2i}. Then, using \partial_{\theta}\pi_{\theta^{2}/2}(i)=(-\theta+\frac{2i}{\theta})\pi_{\theta^{2}/2}(i) \partial_{\theta}p_{\theta}=\sum_{i=0}^{+\infty}\left(-\theta+\frac{2i}{\theta}\right)\pi_{\theta^{2}/2}(i)f_{d+2i}, We note K\sim Poisson(\theta^{2}/2) , and Y|K=i\sim Gamma(d/2+i,2) . Then, s_{\theta}^{2}=\left(\frac{\partial_{\theta}p_{\theta}}{p_{\theta}}\right)^{2}=\mathbb{E}\left[-\theta+\frac{2K}{\theta}\bigg|Y\right]^{2}=\frac{4}{\theta^{2}}\left(\mathbb{E}[K|Y]-\frac{\theta^{2}}{2}\right)^{2}, and finally I(\theta)=\mathbb{E}_{\theta}\left[s_{\theta}^{2}\right]=\frac{4}{\theta^{2}}\Var\left(\mathbb{E}[K|Y]\right). We want to compute a upper bound to \Var(\mathbb{E}[K|Y]) . The variable K|Y=y follows the Bessel distribution ( 52 ) with parameters Bessel(\sqrt{\lambda y},d/2-1) . Then, E[K|Y]=\frac{1}{2}\sqrt{\lambda Y}\frac{I_{d/2}(\sqrt{\lambda Y})}{I_{d/2-1}(\sqrt{\lambda Y})}, where I_{d/2} is the modified Bessel function of index d/2 . Let g:Z\mapsto\sum_{i=1}^{d}(z_{i}+v_{i})^{2}=\|Z+V\|_{F}^{2} , with Z\sim\mathcal{N}(0,I_{d}) , f_{\nu}:y\mapsto\frac{1}{2}yR_{\nu}(y) , and x=\sqrt{\lambda y} . Then, \frac{df_{\nu}}{dy}(x)=\frac{\lambda}{4x}\left(R_{\nu}(x)+xR_{\nu}^{\prime}(x)\right). Using the identity: R^{\prime}_{\nu}(x)=1-\frac{2(\nu-1)}{x}R_{\nu}(x)-R_{\nu}(x)^{2}, we get: \frac{df_{\nu}}{dy}(x)=\frac{\lambda}{4}\left(1-R_{\nu}(x)^{2}-\frac{2(\nu-1)}{x}R_{\nu}(x)\right). A known sharp inequality for Bessel quotients is, for \nu>1/2 ( 43 ) : \frac{x}{\nu+\sqrt{\nu^{2}+x^{2}}}\leq\frac{I_{\nu}(x)}{I_{\nu-1}(x)}\leq\frac{x}{\nu-1/2+\sqrt{(\nu-1/2)^{2}+x^{2}}}. Then, \displaystyle\frac{df_{\nu}}{dy}(x) \displaystyle\leq\frac{\lambda}{4}\left(1-\left(\frac{x}{\nu+\sqrt{\nu^{2}+x^{2}}}\right)^{2}-\frac{2(\nu-1)}{x}\frac{x}{\nu+\sqrt{\nu^{2}+x^{2}}}\right) \displaystyle\leq\frac{\lambda}{4}\left(\frac{2\nu}{\nu+\sqrt{\nu^{2}+x^{2}}}-\frac{2(\nu-1)}{\nu+\sqrt{\nu^{2}+x^{2}}}\right) \displaystyle\leq\frac{\lambda}{2}\frac{1}{\nu+\sqrt{\nu^{2}+x^{2}}}. The Poincaré inequality gives: \displaystyle\Var\left(\frac{1}{2}\sqrt{\lambda Y}\frac{I_{d/2}(\sqrt{\lambda Y})}{I_{d/2-1}(\sqrt{\lambda Y})}\right)= \displaystyle\;\Var\left(f_{(d-1)/2}\circ g(Z)\right) \displaystyle\leq \displaystyle\;\mathbb{E}\left[\|\nabla(f_{(d-1)/2}\circ g(Z))\|_{F}^{2}\right] \displaystyle= \displaystyle\;\mathbb{E}\left[4(f^{\prime}_{(d-1)/2}\circ g(Z))^{2}\|Z+v\|_{F}^{2}\right] \displaystyle= \displaystyle\;\mathbb{E}\left[4\left(\frac{\lambda}{2}\frac{1}{\nu+\sqrt{\nu^{2}+\lambda Y}}\right)^{2}Y\right] \displaystyle= \displaystyle\;\lambda^{2}\mathbb{E}\left[\frac{Y}{2\nu^{2}+\lambda Y}\right]. Then, taking \nu=d/2 and by concavity of y\mapsto\frac{y}{2\nu^{2}+\lambda y} , Jensen inequality gives: I(\lambda)\leq 2\lambda\frac{\mathbb{E}\left[Y\right]}{2(d/2)^{2}+\lambda\mathbb{E}\left[Y\right]}=\frac{4\lambda(\lambda+d)}{d^{2}+2\lambda(\lambda+d)}\leq\frac{2\theta^{2}(\theta^{2}+2d)}{2d^{2}+\theta^{2}(\theta^{2}+2d)}. ∎ Appendix G Numerical estimation of the divergence In this section, we restate the discussion of Section 6 and add details for completeness. We perform a series of experiments designed to illustrate our theoretical results. While sampling from ZV is straightforward, its density is analytically intractable. Therefore, we rely on variational inference methods to estimate the Rényi divergence D_{\alpha}(ZV,ZW) . Specifically, we numerically estimate the Rényi divergence induced by releasing ZV using the Convex-Conjugate Rényi Variational Formula , introduced ( 10 ) (Theorem 2.1). Theorem G.1 (Convex-Conjugate Rényi Variational Formula (Theorem 2.1 from ( 10 ) )) . Let (\Omega,\mathcal{M}) be a measurable space, P,Q\in\mathcal{P}(\Omega) , and \alpha\in(0,1)\cup(1,\infty) . Then, D_{\alpha}(P,Q)=\alpha\sup_{g\in\mathcal{M}_{b}(\Omega);g<0}\left\{\int gdQ+\frac{1}{\alpha-1}\log\int|g|^{(\alpha-1)/\alpha}dP\right\}+\log\alpha+1, where \mathcal{M}_{b}(\Omega) is the space of bounded functions on \Omega . Among existing variational methods to estimate the Rényi divergence, this formula has the specificity of being well-behaved when \alpha>1 ( 10 ) , which aligns with our setting. In order to estimate the Rényi divergence between two distributions P,Q , we proceed as follows. We initialize a two layers neural network f(w_{0},\cdot) with negative poly-softplus activation ( 10 ) and train it to maximize: \sup_{w}\left\{\int f(w)dQ+\frac{1}{\alpha-1}\log\int|f(w)|^{(\alpha-1)/\alpha}dP\right\} Optimization is performed via minibatch stochastic gradient ascent. At each iteration, we sample X=(X_{1},\dots,X_{l})\sim P^{\otimes l} , Y=(Y_{1},\dots,Y_{l})\sim Q^{\otimes l} for some batch size l>0 and update the network parameters using the empirical objective: F(w,X,Y)=\frac{1}{l}\sum_{i=1}^{l}f(w,Y_{i})+\frac{1}{\alpha-1}\log\left(-\frac{1}{l}\sum_{i=1}^{l}f(w,X_{i})^{(\alpha-1)/\alpha}\right). Each estimation is performed 10 times and the results are averaged. For reproducibility, here are some experimental details: • Number of training steps: 2\cdot 10^{4} . • Batch size: 512 . • Learning rate: 10^{-3} . • Hidden dimension: 64 . • Final activation: Poly-softplus ( 10 ) . • Optimizer: Adam ( 29 ) . • Validation score is computed every 100 steps with a resampled validation set of size 5\times 10^{4} . • Early stopping criterion: No validation score improvement for 10 validation steps. • Once trained, the divergence estimation is made on a sample of size 5\times 10^{4} . At each step, data is resampled from the distributions of ZV and ZW . G.1 Proofs of Proposition 7.1 and Corrolary 7.1 In this section, we derive a lower bound on D_{\alpha}(ZV,ZW)-D_{\alpha}(V^{\top}V,W^{\top}W) in the case k=1 . In this setting, we leverage the fact that D_{\alpha}(ZV,ZW)=D_{\alpha}(V^{\top}Z^{\top}ZV,W^{\top}Z^{\top}ZW)=D_{\alpha}(\|ZV\|^{2},\|ZW\|^{2}) . Here, ZV\in\mathbb{R}^{n_{\text{syn}}} . Also, V^{\top}V=\|V\|^{2} follows a non-central chi-squared distribution \chi_{d}^{2}(\|v\|^{2}) with degree of freedom d and non-centrality parameter \|v\|^{2} . To simplify notations, we note n=n_{\text{syn}} and: S_{v}=\frac{1}{n_{\text{syn}}}\|ZV\|^{2} and Y_{v}=\|V\|^{2} . We note the measures: \mu_{n}=P\left(\frac{1}{n_{\text{syn}}}\|ZV\|^{2}\right),\nu_{n}=P\left(\frac{1}{n_{\text{syn}}}\|ZW\|^{2}\right),\mu=\|V\|^{2} and \nu=\|W\|^{2} . Also, for s>0 , we note r(x)=\frac{d\mu}{d\nu}(s) . Lemma G.1 . Assume that k=1 . Let \alpha>1 . For two distributions P,Q , we note H_{\alpha}(P,Q)=\exp((\alpha-1)D_{\alpha}(P,Q)) . Let X_{n}\sim\frac{1}{n}\chi_{n}^{2} follow a (rescaled) chi-squared distribution with degree of freedom n . Then, H_{\alpha}(\mu,\nu)-H_{\alpha}(\mu_{n},\nu_{n})=\mathbb{E}_{S_{w}\sim\nu_{n}}\left[\mathbb{E}_{Y_{w}\sim\nu}[r(Y_{w})^{\alpha}|Y_{w}X_{n}=S_{w}]-\mathbb{E}_{Y_{w}\sim\nu}\left[r(Y_{w})\big|Y_{w}X_{n}=S_{w}\right]^{\alpha}\right]. Proof. First, we notice that ZV|V\sim\mathcal{N}(0,\|V\|^{2}I_{n}) . Let X_{n}\sim\frac{1}{n}\chi_{n}^{2} . Then, S_{v}|Y_{v}=Y_{v}X_{n} . \mu_{n} and \nu_{n} can be seen as the output of the same Markov kernel K:y\mapsto yX_{n} for the measures \mu and \nu . Thus, the density ratio \frac{d\mu_{n}}{d\nu_{n}}(s) writes: \frac{d\mu_{n}}{d\nu_{n}}(s)=\mathbb{E}_{Y_{w}\sim\nu}\left[\frac{d\mu}{d\nu}(Y_{w})\big|Y_{w}X_{n}=s\right]=\mathbb{E}_{Y_{w}\sim\nu}\left[r(Y_{w})\big|Y_{w}X_{n}=s\right] Now, by definition, H_{\alpha}(S_{v},S_{w})=\mathbb{E}_{S_{w}\sim\nu_{n}}\left[\mathbb{E}_{Y_{w}\sim\nu}\left[r(Y_{w})\big|Y_{w}X_{n}=S_{w}\right]^{\alpha}\right] Also by tower property, H_{\alpha}(Y_{v},Y_{w})=\mathbb{E}_{Y_{w}\sim\nu}\left[r(Y_{w})^{\alpha}\right]=\mathbb{E}_{S_{w}\sim\nu_{n}}\left[\mathbb{E}_{Y_{w}\sim\nu}\left[r(Y_{w})^{\alpha}\big|Y_{w}X_{n}=S_{w}\right]\right], giving the desired result. ∎ Now, we upper bound the quantity H_{\alpha}(\mu,\nu)-H_{\alpha}(\mu_{n},\nu_{n}) . Our strategy relies on showing that, for large values of n , the ratio r(Y_{w}) is concentrated around \mathbb{E}[r(Y_{w})|Y_{w}X_{n}=S_{w}] with high probability. Lemma G.2 . Let \alpha\geq 2 . We note M(S_{w})=\mathbb{E}[r(Y_{w})|Y_{w}X_{n}=S_{w}] . Then, H_{\alpha}(\mu,\nu)-H_{\alpha}(\mu_{n},\nu_{n})\leq\alpha(\alpha-1)2^{\alpha-3}e^{(\alpha-1)(\alpha-2)\Delta^{2}/2}\|r(Y_{w})-M(S_{w})\|_{\alpha}^{2}. Proof. We have, by definition: \mathbb{E}[r(Y_{w})-M(S_{w})|Y_{w}X_{n}=S_{w}]=\mathbb{E}[r(Y_{w})-M(S_{w})|S_{w}]=0. Then, \mathbb{E}[\alpha M^{\alpha-1}(S_{w})(r(Y_{w})-M(S_{w}))|S_{w}]=0 . Let \phi_{\alpha}:x\mapsto x^{\alpha} Then, leveraging \alpha\geq 2 and by the mean value theorem in the expectation in the interval (\min(M(S_{w}),r(Y_{w})),\max(M(S_{w}),r(Y_{w})) , and \max(M(S_{w}),r(Y_{w}))\leq M(S_{w})+r(Y_{w}) : \displaystyle H_{\alpha}(\mu,\nu)-H_{\alpha}(\mu_{n},\nu_{n}) \displaystyle=\mathbb{E}\left[\mathbb{E}[r(Y_{w})^{\alpha}|S_{w}]-M(S_{w})^{\alpha}\right] \displaystyle=\mathbb{E}\left[\mathbb{E}[r^{\alpha}(Y_{w})-M(S_{w})^{\alpha}-\alpha M^{\alpha-1}(S_{w})(r(Y_{w})-M(S_{w}))|S_{w}]\right] \displaystyle=\mathbb{E}\left[\mathbb{E}[\phi(r(Y_{w}))-\phi(M(S_{w}))-(r(Y_{w})-M(S_{w}))\phi^{\prime}(M(S_{w}))|S_{w}]\right] \displaystyle=\frac{1}{2}\mathbb{E}\left[\mathbb{E}[\phi^{\prime\prime}(Z)(M(S_{w})-r(Y_{w}))^{2}|S_{w}]\right],\;\;\;\;Z\in(\min(M(S_{w}),r(Y_{w})),\max(M(S_{w}),r(Y_{w})) \displaystyle\leq\frac{\alpha(\alpha-1)}{2}\mathbb{E}\left[\mathbb{E}[(M(S_{w})+r(Y_{w}))^{\alpha-2}(M(S_{w})-r(Y_{w}))^{2}|S_{w}]\right] \displaystyle\leq\frac{\alpha(\alpha-1)}{2}\|r(Y_{w})+M(S_{w})\|_{\alpha}^{\alpha-2}\|r(Y_{w})-M(S_{w})\|_{\alpha}^{2}, where the last step is the Hölder inequality with exponents \frac{\alpha}{\alpha-2} and \frac{\alpha}{2} . Now, by tower property and the Jensen inequality, \|M(S_{w})\|_{\alpha}^{\alpha}=\mathbb{E}[M(S_{w})^{\alpha}]=\mathbb{E}[\mathbb{E}[r(Y_{w})|S_{w}]^{\alpha}]\leq\mathbb{E}[\mathbb{E}[r(Y_{w})^{\alpha}|S_{w}]]=\|r(Y_{w})\|_{\alpha}^{\alpha}. Then, triangle inequality implies: \|r(Y_{w})+M(S_{w})\|_{\alpha}\leq\|r(Y_{w})\|_{\alpha}+\|M(S_{w})\|_{\alpha}\leq 2\|r(Y_{w})\|_{\alpha}. Therefore, H_{\alpha}(\mu,\nu)-H_{\alpha}(\mu_{n},\nu_{n})\leq\alpha(\alpha-1)2^{\alpha-3}\|r(Y_{w})\|_{\alpha}^{\alpha-2}\|r(Y_{w})-M(S_{w})\|_{\alpha}^{2}. Also, we can relate \|r(Y_{w})\|_{\alpha} to D_{\alpha}(V,W) by a post-processing argument: \|r(Y_{w})\|_{\alpha}^{\alpha}=\mathbb{E}\left[r(Y_{w})^{\alpha}\right]=e^{(\alpha-1)D_{\alpha}(\|V\|^{2},\|W\|^{2})}\leq e^{(\alpha-1)D_{\alpha}(V,W)}=e^{\alpha(\alpha-1)\Delta^{2}/2}. Then, we obtain: H_{\alpha}(\mu,\nu)-H_{\alpha}(\mu_{n},\nu_{n})\leq\frac{\alpha(\alpha-1)}{2}2^{\alpha-2}e^{(\alpha-1)(\alpha-2)\Delta^{2}/2}\|r(Y_{w})-M(S_{w})\|_{\alpha}^{2}. ∎ Now, we upper bound \|r(Y_{w})-M(S_{w})\|_{\alpha}^{2} by a concentration argument: Proposition G.1 . Let \alpha\geq 2 . Then, there exists C>0 such that d>C implies: H_{\alpha}(\mu,\nu)-H_{\alpha}(\mu_{n},\nu_{n})=O(1/n), where the rate does not depend on d . Proof. First, for any measurable function g , \|r(Y_{w})-M(S_{w})\|_{\alpha}\leq 2\|r(Y_{w})-g(S_{w})\|_{\alpha} . In fact, r(Y_{w})-M(S_{w})=r(Y_{w})-g(S_{w})-\mathbb{E}[r(Y_{w})-g(S_{w})|S_{w}] and the conditional expectation is a contraction of L^{\alpha}(\mathbb{R)} . Let A=\{X_{n}\in(1/2,2)\} . Then, choosing g(S_{w})=r(S_{w})\mathbbm{1}_{A} , we have: \displaystyle\|r(Y_{w})-g(S_{w})\|_{\alpha}^{\alpha}=\mathbb{E}[|r(Y_{w})-r(S_{w})\mathbbm{1}_{A}|^{\alpha}]=\mathbb{E}[|r(Y_{w})-r(S_{w})|^{\alpha}\mathbbm{1}_{A}]+\mathbb{E}[|r(Y_{w})|^{\alpha}\mathbbm{1}_{A^{C}}]. Therefore, we can decompose \mathbb{E}[(r(Y_{w})-M(S_{w}))^{\alpha}] into two terms: \mathbb{E}[(r(Y_{w})-M(S_{w}))^{\alpha}]\leq 2^{\alpha}(\mathbb{E}[|r(Y_{w})-r(S_{w})|^{\alpha}\mathbbm{1}_{A}]+\mathbb{E}[|r(Y_{w})|^{\alpha}\mathbbm{1}_{A^{C}}]). We upper bound both terms. First, we upper bound P(A^{C}) by concentration. X_{n}\sim\frac{1}{n}\chi_{n}^{2} . The moment-generating function of X_{n} is finite for t\in(-\infty,n/2) and writes M_{X_{n}}(t)=\left(1-\frac{2t}{n}\right)^{-n/2} . Let t\in(0,n/2) . Then, the Chernoff inequality gives: P(X_{n}\geq 2)\leq M_{X_{n}}(t)e^{-2t}=\exp\left(-2t-\frac{n}{2}\log(1-2t/n)\right). We find the lower bound of this inequality at t^{*}=n(1-\frac{1}{x})/2 . P(X_{n}\geq 2)\leq e^{-\frac{n}{2}(1-\log 2)} . Now, let t<0 . Since x\mapsto e^{tx} is decreasing: P(X_{n}\leq 1/2)=P(e^{Xt}\geq e^{t/2})\leq M_{X_{n}}(t)e^{-t/2}=\exp\left(-t/2-\frac{n}{2}\log(1-2t/n)\right). We find the lower bound of this inequality at t^{*}=-n/2 . Then, P(X_{n}\leq 1/2)\leq e^{-\frac{n}{2}(\log 2-1/2)} Also, \log 2-1/2\approx 0.19\leq 1-\log 2\approx 0.31 . Therefore, P(A^{C})\leq 2e^{-cn} , with c=\frac{1}{2}(\log 2-1/2) . Let p,q>0 such that 1/p+1/q=1 . \displaystyle\mathbb{E}[(r(Y_{w})-M(S_{w}))^{\alpha}\mathbbm{1}_{A^{C}}] \displaystyle\leq P(A^{C})^{1/p}\mathbb{E}[(r(Y_{w})-M(S_{w}))^{\alpha q}]^{1/q} \displaystyle\leq 2^{1/p}e^{-cn/p}\mathbb{E}[(r(Y_{w})-M(S_{w}))^{\alpha q}]^{1/q} Also, leveraging Lemma G.2 , \mathbb{E}[r(Y_{w})^{\alpha q}]\leq e^{(\alpha q-1)\Delta^{2}/2} . Therefore, taking p=q=2 , \mathbb{E}[(r(Y_{w})-r(S_{w}))^{\alpha}\mathbbm{1}_{A^{C}}]\leq 2^{1/2}\exp(\alpha(2\alpha-1)\Delta^{2}/2-cn/2). Now, we upper bound the quantity \mathbb{E}[|1-X_{n}|^{\alpha}] . nX_{n}-n is a sub-Gamma random variable, since nX_{n}-n\sim\Gamma(2n,2) . From Theorem 2.3. of 11 , for even \alpha>1 , \mathbb{E}[|nX_{n}-n|^{\alpha}]\leq(\alpha/2)!(16n)^{\alpha/2}+\alpha!8^{\alpha}\lesssim C_{\alpha}n^{\alpha/2} . Dividing by n^{\alpha} , there exists C_{\alpha}>0 such that \mathbb{E}[|1-X_{n}|^{\alpha}]\leq C_{\alpha}n^{-\alpha/2} . For other values of \alpha , we use the monotonicity of the L_{p} norm to obtain the upper bound. Also, noticing (r(Y_{w})-r(Y_{w}X_{n}))^{\alpha}\mathbbm{1}_{A}\leq|Y_{w}-Y_{w}X_{n}|^{\alpha}\sup_{t\in(Y_{w}/2,2Y_{w})}r^{\prime}(t)^{\alpha}=Y_{w}^{\alpha}|1-X_{n}|^{\alpha}\sup_{x\in(1/2,2)}r^{\prime}(Y_{w}x)^{\alpha} , we get: \displaystyle\mathbb{E}[(r(Y_{w})-r(S_{w}))^{\alpha}\mathbbm{1}_{A}] \displaystyle\leq\mathbb{E}[Y_{w}^{\alpha}|1-X_{n}|^{\alpha}\sup_{x\in(1/2,2)}r^{\prime}(Y_{w}x)^{\alpha}] \displaystyle\leq\mathbb{E}[|1-X_{n}|^{\alpha}]\mathbb{E}[Y_{w}^{\alpha}\sup_{x\in(1/\beta,\beta)}r^{\prime}(Y_{w}x)^{\alpha}] \displaystyle\leq C_{\alpha}n^{-\alpha/2}\mathbb{E}[Y_{w}^{\alpha}\sup_{x\in(1/2,2)}r^{\prime}(Y_{w}x)^{\alpha}] Now, we evaluate \mathbb{E}[Y_{w}^{\alpha}\sup_{x\in(1/\beta,\beta)}r^{\prime}(Y_{w}x)^{\alpha}] . We note \lambda_{v}=\|v\|,\lambda_{w}=\|w\| . Let y>0 . The density of Y_{w} writes: \displaystyle p_{\lambda_{v}}(y)=p_{0}(y)e^{-\lambda_{v}^{2}/2}\lambda_{v}^{1-d/2}I_{d/2-1}\big(\lambda_{v}\sqrt{y}\big). Then, we write the ratio r(y) : \displaystyle r(y)=e^{(\lambda_{w}^{2}-\lambda_{v}^{2})/2}\left(\frac{\lambda_{v}}{\lambda_{w}}\right)^{1-d/2}\frac{I_{d/2-1}\big(\lambda_{v}\sqrt{y}\big)}{I_{d/2-1}\big(\lambda_{w}\sqrt{y}\big)}. We recall the following relationship: \lambda_{v}\sqrt{y}R_{d/2-1}(\lambda_{v}\sqrt{y})+d/2-1=\lambda_{v}\sqrt{y}\frac{I_{d/2-1}^{\prime}(\lambda_{v}\sqrt{y})}{I_{d/2-1}(\lambda_{v}\sqrt{y})}. Then, we compute a upper bound to \frac{dr}{dy}(y) . \displaystyle\frac{dr}{dy}(y) \displaystyle=\partial_{y}\log I_{d/2-1}(\lambda_{v}\sqrt{y})-\partial_{y}\log I_{d/2-1}(\lambda_{w}\sqrt{y}) \displaystyle=\frac{1}{2\sqrt{y}}\left(\lambda_{v}\frac{I_{d/2-1}^{\prime}(\lambda_{v}\sqrt{y})}{I_{d/2-1}(\lambda_{v}\sqrt{y})}-\lambda_{w}\frac{I_{d/2-1}^{\prime}(\lambda_{w}\sqrt{y})}{I_{d/2-1}(\lambda_{w}\sqrt{y})}\right) \displaystyle=\frac{1}{2\sqrt{y}}\left(\lambda_{v}R_{d/2-1}(\lambda_{v}\sqrt{y})-\lambda_{w}R_{d/2-1}(\lambda_{w}\sqrt{y})\right) \displaystyle\leq\frac{1}{2\sqrt{y}}\left(\lambda_{v}R_{d/2-1}(\lambda_{v}\sqrt{y})+\lambda_{w}R_{d/2-1}(\lambda_{w}\sqrt{y})\right) \displaystyle\leq\frac{\lambda_{v}^{2}+\lambda_{w}^{2}}{2d}, where the Bessel quotient upper bound is obtained leveraging the following inequality: 0\leq R_{\nu}(y)=\frac{I_{\nu+1}(y)}{I_{\nu}(y)}\leq\frac{y}{2(\nu+1)}. Therefore, noting B=(\lambda_{v}^{2}+\lambda_{w}^{2})/2 , |r^{\prime}(y)|=r(y)|\partial_{y}\log r(y)|\leq\frac{B}{d}r(y) . Then, by Gronwall’s inequality, for any x\in(1/2,2) , r(yx)\leq r(y)e^{By(x-1)/d}. Therefore, \sup_{x\in(1/2,2)}|r^{\prime}(Y_{w}x)|\leq\frac{B}{d}\sup_{x\in(1/2,2)}|r(Y_{w}x)|\leq\frac{B}{d}r(Y_{w})e^{2yB/d} . Then, we apply the generalized Hölder inequality: \displaystyle\mathbb{E}[Y_{w}^{\alpha}\sup_{x\in(1/\beta,\beta)}r^{\prime}(Y_{w}x)^{\alpha}] \displaystyle\leq\frac{B^{\alpha}}{d^{\alpha}}\mathbb{E}[Y_{w}^{\alpha}r(Y_{w})^{\alpha}e^{2\alpha Y_{w}C/d}] \displaystyle\leq\frac{B^{\alpha}}{d^{\alpha}}\mathbb{E}[Y_{w}^{3\alpha}]^{1/3}\mathbb{E}[e^{6\alpha Y_{w}B/d}]^{1/3}\mathbb{E}[r(Y_{w})^{3\alpha}]^{1/3} \displaystyle\leq\frac{B^{\alpha}}{d^{\alpha}}\mathbb{E}[Y_{w}^{3\alpha}]^{1/3}\mathbb{E}[e^{6\alpha Y_{w}B/d}]^{1/3}e^{(3\alpha-1)D_{2\alpha}(V,W)/3} \displaystyle\leq\frac{B^{\alpha}}{d^{\alpha}}\mathbb{E}[Y_{w}^{3\alpha}]^{1/3}\mathbb{E}[e^{6\alpha Y_{w}B/d}]^{1/3}e^{\alpha(3\alpha-1)\Delta^{2}/3} Also, leveraging the moment-generating function of non-central chi-squared distributions, for d>12\alpha\lambda_{w}B : \mathbb{E}[e^{6\alpha Y_{w}B/d}]=(1-12\alpha B/d)^{-d/2}\exp(6\alpha B\lambda_{w}/(d-12\alpha B)). Also, leveraging Lemma G.3 , we can upper bound \mathbb{E}[Y_{w}^{3\alpha}] : \mathbb{E}[Y_{w}^{3\alpha}]\leq(d+\alpha C_{1}(2+\lambda_{w})/2)^{3\alpha}, with C_{1}>0 . Then, we obtain: \displaystyle\mathbb{E}[Y_{w}^{\alpha}\sup_{x\in(1/\beta,\beta)}r^{\prime}(Y_{w}x)^{\alpha}] \displaystyle\leq\frac{B^{\alpha}}{d^{\alpha}}\mathbb{E}[Y_{w}^{3\alpha}]^{1/3}\mathbb{E}[e^{6\alpha Y_{w}B/d}]^{1/3}e^{\alpha(3\alpha-1)\Delta^{2}/3} \displaystyle\leq B^{\alpha}\left(1+\frac{\alpha C_{1}(2+\lambda_{w})}{2d}\right)^{\alpha}\left(1-\frac{12\alpha B}{d}\right)^{-d/6}e^{\alpha(3\alpha-1)\Delta^{2}/3+6\alpha B\lambda_{w}/3(d-12\alpha B)}. \displaystyle\lesssim C^{\prime}_{v,w,\alpha}\text{, independent of $d$ for sufficiently large $d$}. In particular, this upper bound does not diverge when d\to\infty . This means that for d large enough, we get convergence rates that do not depend on the dimensionality of the model d . However, our proof technique generates a factor \exp(6\alpha B\lambda_{w}/(d-12\alpha B)) appear, which means that our analysis only holds for large values of d . Then, \|r(Y_{w})-M(S_{w})\|_{\alpha}^{2}\leq 2(2^{1/\alpha}e^{(2\alpha-1)\Delta^{2}-\alpha cn}+(C_{\alpha}C^{\prime}_{\alpha})^{2/\alpha}n^{-1})=O(n^{-1}). Therefore, H_{\alpha}(\mu,\nu)-H_{\alpha}(\mu_{n},\nu_{n})=O(1/n) . ∎ Lemma G.3 . Let d>1,\theta>0 and Y_{w}\sim\chi_{d}^{2}(\theta^{2}) follow a non-central chi-squared distribution. Let \alpha>1 . Then, there exists C_{0}>0 such that \|Y\|_{\alpha}\leq 2(d/2+\alpha C_{1}(2+\theta^{2})). Proof. Let K\sim\text{Poisson}(\theta^{2}/2) . Then, non-central chi-squared distributions admit the following Poisson mixture of chi-squared distributions representation: Y|K\sim\chi_{d+2K}^{2} . Also, central chi-squared distributions X_{d}\sim\chi_{d}^{2} admit the following moments: E[X_{d}^{\alpha}]=2^{\alpha}\frac{\Gamma(\alpha+d/2)}{\Gamma(d/2)} , where \Gamma denotes the \Gamma function. Therefore, leveraging the convexity inequality (x+y)^{\alpha}\leq 2^{\alpha-1}(x^{\alpha}+y^{\alpha}) : \displaystyle\mathbb{E}[Y^{\alpha}]=\mathbb{E}[\mathbb{E}[Y^{\alpha}|K]]=\mathbb{E}\left[\mathbb{E}[X_{d+2K}^{\alpha}|K]\right]=2^{\alpha}\mathbb{E}\left[\frac{\Gamma(\alpha+K+d/2)}{\Gamma(K+d/2)}\right] \displaystyle\leq 2^{\alpha}\mathbb{E}\left[(\alpha+K+d/2)^{\alpha}\right]. Then, by triangle inequality: \|Y\|_{\alpha}\leq 2\|\alpha+d/2+K\|_{\alpha}\leq 2(\alpha+d/2+\|K\|_{\alpha}) . The moment generating function of K writes \mathbb{E}[e^{tK}]=\exp(\theta^{2}(e^{t}-1)/2) . Since K is subexponential, we can simply upper bound \|K\|_{\alpha} with Orlicz norms ( 50 ) : \|K\|_{\alpha}\leq C_{0}\alpha\|K\|_{\psi_{1}} , with C_{0}>0 an absolute constant. Since E[K]=\theta^{2}/2 , we can upper bound the Orlicz norm by \|K\|_{\psi_{1}}\lesssim 1+\theta^{2}/2 . Therefore, there exists C_{1}>0 such that \|K\|_{\alpha}\leq C_{1}\alpha(1+\theta^{2}/2) Then, \|Y\|_{\alpha}\leq 2(d/2+\alpha(1+C_{1}(1+\theta^{2}/2)). ∎ Appendix H Regularity of the studied distributions and technical lemmas In this paper, we study several distributions and parametrizations. Our results hold under regularity conditions, imposed by Proposition 3.2 and Proposition C.3 . In this section, we prove that the studied distributions are sufficiently regular, as defined by Definition C.1 . H.1 Regularity of non-central chi-squared distributions Let d\in\mathbb{N}^{*} . We consider the family of distributions P=\{P_{\theta}:=\chi_{d}^{2}(\theta^{2}),\theta>0\} . Proposition H.1 . The family of distributions P is regular. Proof. We proceed from Definition C.1 by verifying each point: • \Theta=\mathbb{R}^{+} is a right-open set of \mathbb{R} . • For each \theta\in\Theta , \supp(P_{\theta})=\mathbb{R}^{+} . • For each \theta\in\Theta , P_{\theta} admits a density: \text{for all }x>0,\;p_{\theta}(x)=p_{0}(x)e^{-\theta^{2}/2}\theta^{1-d/2}I_{d/2-1}\big(\theta\sqrt{x}\big), where p_{0} is the density of a central chi-squared distribution with degree of freedom d . • For all \theta\in\Theta , \log p_{\theta}(x) is twice differentiable. In fact, modified Bessel functions of the first kind are differentiable ( 5 ) , and, leveraging Lemma H.1 : \displaystyle\partial_{\theta}\log p_{\theta}(x) \displaystyle=-\theta+\sqrt{x}R_{d/2-1}(\theta\sqrt{x}), \displaystyle\partial_{\theta}^{2}\log p_{\theta}(x) \displaystyle=-1+\frac{1}{2\sqrt{x}}R_{d/2-1}(\theta\sqrt{x})+\frac{\theta}{2}R^{\prime}_{d/2-1}(\theta\sqrt{x}) \displaystyle=-1+\frac{1}{2\sqrt{x}}R_{d/2-1}(\theta\sqrt{x})+\frac{\theta}{2}(1-R_{d/2-1}(\theta\sqrt{x})-R_{d/2-1}(\theta\sqrt{x})^{2}). • For all \theta\in\Theta , \partial_{\theta}p_{\theta} and \partial_{\theta}^{2}p_{\theta} verify the integrability conditions. In fact, R_{d}(x)\leq\frac{x}{2(d+1)} . Therefore, \displaystyle|\partial_{\theta}\log p_{\theta}(x)| \displaystyle\leq-\theta+\frac{\theta x}{d}, \displaystyle|\partial_{\theta}^{2}\log p_{\theta}(x)| \displaystyle\leq-1+\frac{\theta}{2d}+\frac{\theta}{2}\left(1+\frac{\theta\sqrt{x}}{d}+\frac{\theta^{2}x}{d^{2}}\right). Moreover, p_{\theta} have exponentially light tails. In fact, leveraging Lemma H.2 , I_{d/2-1}(\theta\sqrt{x})\leq I_{d/2-1}(1)\theta\sqrt{x}e^{\theta\sqrt{x}-1} , and: \displaystyle p_{\theta}(x) \displaystyle=\frac{1}{2}e^{-(\theta^{2}+x)/2}\left(\frac{x}{\theta^{2}}\right)^{d/4-1/2}I_{d/2-1}(\theta\sqrt{x}) \displaystyle\lesssim x^{d/4-1/2}e^{-x/2+\theta\sqrt{x}}. As |\partial_{\theta}\log p_{\theta}(x)| and |\partial_{\theta}^{2}\log p_{\theta}(x)| can be upper bounded by polynomials, the integrability conditions hold. • For all \theta,\theta^{\prime}\in\Theta , \left|\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x)\right|\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}} and \left(\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x)^{2}+\left|\partial_{\theta^{\prime}}^{2}\log p_{\theta^{\prime}}(x)\right|\right)\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}} verify the integrability conditions. In fact, for some constant C(\theta,\theta^{\prime}) : \displaystyle\frac{p_{\theta^{\prime}}(x)}{p_{\theta}(x)} \displaystyle=C(\theta,\theta^{\prime})\frac{I_{d/2-1}\big(\theta^{\prime}\sqrt{x}\big)}{I_{d/2-1}\big(\theta\sqrt{x}\big)}\leq C(\theta,\theta^{\prime})\left(\frac{\theta^{\prime}}{\theta}\right)^{d/2-1}e^{\sqrt{x}(\theta^{\prime}-\theta)}. Therefore: \frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}}\lesssim x^{d/4-1/2}e^{-x/2+\sqrt{x}(\alpha\theta^{\prime}-(\alpha-1)\theta)}. Since |\partial_{\theta}\log p_{\theta}(x)| and |\partial_{\theta}^{2}\log p_{\theta}(x)| can be upper bounded by polynomials, the integrability conditions hold. ∎ H.2 Regularity of non-central Wishart distributions Let d>k\geq 1 . Let v,w\in\mathbb{R}^{d\times k} . We note v_{\theta}=(1-\theta)v+\theta w , and \Omega_{\theta}=v_{\theta}^{\top}v_{\theta} . We consider the family of distributions: P=\{P_{\theta}:=\mathcal{W}(d,k,\Omega_{\theta}),\;\theta\in\mathbb{R}\}. Proposition H.2 . The family of distributions P is regular. Proof. We proceed from Definition C.1 by verifying each point: • \Theta=\mathbb{R} is a right-open set of \mathbb{R} . • For each \theta\in\Theta , \supp(P_{\theta})=\mathbb{S}_{+}^{k}(\mathbb{R}) , the set of symmetric positive semidefinite matrices. • Since d>k , for each \theta\in\Theta , P_{\theta} admits a density ( 39 ) : \text{for all }x>0,\;p_{\theta}(x)=p_{0}(x)e^{-\tr(\Omega_{\theta})/2}{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right), where p_{0} is the density of a central Wishart distribution with degree of freedom d and size k . • For all \theta\in\Theta , \log p_{\theta}(x) is twice differentiable. In fact, from Lemma H.3 , hypergeometric functions of a matrix argument are C^{\infty}(\mathbb{R}^{k\times k}) . • For all \theta\in\Theta , \partial_{\theta}p_{\theta} and \partial_{\theta}^{2}p_{\theta} verify the integrability conditions. In fact, Lemma H.4 ensures that, for all x\in\mathbb{S}_{+}^{k}(\mathbb{R}) , \displaystyle\left|\partial_{\theta}\log p_{\theta}(x)\right| \displaystyle\leq\|v-w\|_{F}^{2}\|v_{\theta}\|_{F}^{2}+\sqrt{k}\|v-w\|_{F}\|x\|_{F}^{1/2}, \displaystyle\left|\partial_{\theta}^{2}\log p_{\theta}(x)\right| \displaystyle\leq 2k\|v-w\|_{F}^{2}\|x\|_{F}. Moreover, p_{\theta} have exponentially light tails. In fact, from Lemma H.6 , {}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)\leq e^{\sqrt{k}\|v_{\theta}\|_{F}\|x\|_{F}^{1/2}} , and: p_{\theta}(x)=p_{0}(x)e^{-\tr(\Omega_{\theta})/2}{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)\leq C_{0}\det(x)^{(d-k-1)/2}e^{-\tr(\Omega_{\theta})/2+\sqrt{k}\|v_{\theta}\|_{F}\|x\|_{F}^{1/2}-tr(x)/2}, therefore ensuring domination by an integrable function. • For all \theta,\theta^{\prime}\in\Theta , \left|\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x)\right|\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}} and \left(\partial_{\theta^{\prime}}\log p_{\theta^{\prime}}(x)^{2}+\left|\partial_{\theta^{\prime}}^{2}\log p_{\theta^{\prime}}(x)\right|\right)\frac{p_{\theta^{\prime}}(x)^{\alpha}}{p_{\theta}(x)^{\alpha-1}} verify the integrability conditions. In fact, from Lemma H.5 , for some constant C(\theta,\theta^{\prime}) : \displaystyle\frac{p_{\theta^{\prime}}(x)}{p_{\theta}(x)} \displaystyle\leq C(\theta,\theta^{\prime})\exp\left(|\theta^{\prime}-\theta|\sqrt{k}\|v-w\|_{F}\|x\|_{F}^{1/2}\right). Since |\partial_{\theta}\log p_{\theta}(x)| and |\partial_{\theta}^{2}\log p_{\theta}(x)| can be upper bounded by polynomials of in the norm of x , the integrability conditions hold. ∎ Here, we state some classical properties of modified Bessel functions of the first kind and the hypergeometric function of a matrix argument {}_{0}F_{1} : Lemma H.1 (Ricatti equation for Bessel quotients) . R^{\prime}_{d}(x)=1-\frac{2d+1}{x}R_{d}(x)-R_{d}(x)^{2} Proof. Modified Bessel functions of the first kind admit the following recurrence relations ( 5 , 11.115 and 11.116) : I_{d-1}(x)-I_{d+1}(x)=\frac{2d}{x}I_{d}(x). I_{d-1}(x)+I_{d+1}(x)=2I_{d}^{\prime}(x). Then, \displaystyle R^{\prime}_{d}(x) \displaystyle=\frac{I^{\prime}_{d+1}(x)}{I_{d}(x)}-\frac{I_{d+1}(x)I_{d}^{\prime}(x)}{I_{d}(x)^{2}} \displaystyle=\frac{I_{d+2}(x)+I_{d}(x)}{2I_{d}(x)}-\frac{I_{d+1}(x)^{2}+I_{d+1}(x)I_{d-1}(x)}{2I_{d}(x)^{2}} \displaystyle=\frac{(I_{d}(x)-\frac{2(d+1)}{x}I_{d+1})+I_{d}(x)}{2I_{d}(x)}-\frac{I_{d+1}(x)(I_{d+1}(x)+\frac{2d}{x}I_{d}(x))}{2I_{d}(x)^{2}}-\frac{1}{2}R_{d}(x)^{2} \displaystyle=1-\frac{2d+1}{x}R_{d}(x)-R_{d}(x)^{2}. ∎ Lemma H.2 . Let d>1,x,y>0 . Then: \frac{I_{d}(y)}{I_{d}(x)}\leq\left(\frac{y}{x}\right)^{d}e^{y-x} . Proof. Let d>1,x,y>0 . We compute the derivative of \log I_{d}(x) : \partial_{x}\log I_{d}(x)=\frac{I_{d}^{\prime}(x)}{I_{d}(x)}=\frac{I_{d-1}(x)+I_{d+1}(x)}{2I_{d}(x)}=R_{d}(x)+\frac{d}{x}\leq 1+\frac{d}{x}. Then, \displaystyle\frac{I_{d}(y)}{I_{d}(x)}=\exp(\log I_{d}(y)-\log I_{d}(x)) \displaystyle=\exp\left(\int_{x}^{y}\partial_{u}\log I_{d}(u)du\right) \displaystyle\leq\exp\left(\int_{x}^{y}\left(1+\frac{d}{u}\right)du\right) \displaystyle\leq\left(\frac{y}{x}\right)^{d}e^{y-x}. ∎ Lemma H.3 ( {}_{0}F_{1} is real-analytic ( 22 ) ) . Let d>k\geq 1 . Let {}_{0}F_{1} be the hypergeometric function of a matrix argument. Then, X\in\mathbb{R}^{k\times k}\mapsto{}_{0}F_{1}(d/2,X) is real-analytic. Equivalently, {}_{0}F_{1}(d/2,\cdot)\in C^{\infty}(\mathbb{R}^{k\times k}) . Proposition H.3 (Integral representation of {}_{0}F_{1} , Theorem 7.4.1. from 39 ) . Let 1\leq k\leq d . If x\in\mathbb{R}^{d\times k} , then: {}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}X^{\top}X\right)=\int_{V_{d,k}}e^{\langle X^{\top},H\rangle}d\mu(H), where V_{d,k}\subset\mathbb{R}^{k\times d} is the Stiefel manifold, and \mu is the normalized uniform measure on V_{n,d} . Lemma H.4 . Let d>2 , x\in\mathbb{S}_{+}^{k}(\mathbb{R}) . Then, \displaystyle\left|\partial_{\theta}\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)\right| \displaystyle\leq\sqrt{k}\|v-w\|_{F}\|x\|_{F}^{1/2}, \displaystyle\left|\partial_{\theta}^{2}\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)\right| \displaystyle\leq 2k\|v-w\|_{F}^{2}\|x\|_{F}. Proof. We have: p_{\theta}(x)=p_{0}(x)e^{-\tr(\Omega_{\theta})/2}{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right). Then, leveraging the integral representation of {}_{0}F_{1} from Proposition H.3 : \displaystyle\partial_{\theta}\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right) \displaystyle=\frac{1}{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)}\partial_{\theta}\int_{V_{d,k}}e^{\langle x^{1/2}v_{\theta}^{\top},H\rangle}d\mu(H) \displaystyle=\frac{1}{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)}\partial_{\theta}\int_{V_{d,k}}\langle x^{1/2}\dot{v}_{\theta}^{\top},H\rangle e^{\langle x^{1/2}v_{\theta}^{\top},H\rangle}d\mu(H) \displaystyle=\mathbb{E}_{H\sim\pi_{X}}[\langle x^{1/2}\dot{v}_{\theta}^{\top},H\rangle]. Then, since H is supported on the Stiefel manifold V_{d,k} : \left|\partial_{\theta}\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)\right|\leq\|H\|_{F}\|\dot{v}_{\theta}\|_{F}\|x\|_{F}^{1/2}\leq\sqrt{k}\|v-w\|_{F}\|x\|_{F}^{1/2}. Also, the second-order derivative writes: \displaystyle\partial_{\theta}^{2}\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right) \displaystyle=\frac{\partial_{\theta}\int_{V_{d,k}}\langle x^{1/2}\dot{v}_{\theta}^{\top},H\rangle e^{\langle x^{1/2}v_{\theta}^{\top},H\rangle}d\mu(H)}{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)}-\left(\frac{\partial_{\theta}\int_{V_{d,k}}\langle x^{1/2}\dot{v}_{\theta}^{\top},H\rangle e^{\langle x^{1/2}v_{\theta}^{\top},H\rangle}d\mu(H)}{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)}\right)^{2} \displaystyle=\frac{\int_{V_{d,k}}\left(\langle x^{1/2}\dot{v}_{\theta}^{\top},H\rangle^{2}+\langle x^{1/2}\ddot{v}_{\theta}^{\top},H\rangle\right)e^{\langle x^{1/2}v_{\theta}^{\top},H\rangle}d\mu(H)}{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)}-\mathbb{E}_{H\sim\pi_{X}}[\langle x^{1/2}\dot{v}_{\theta}^{\top},H\rangle]^{2}. \displaystyle=\mathbb{E}_{H\sim\pi_{X}}[\langle x^{1/2}\dot{v}_{\theta}^{\top},H\rangle^{2}+\langle x^{1/2}\ddot{v}_{\theta}^{\top},H\rangle]-\mathbb{E}_{H\sim\pi_{X}}[\langle x^{1/2}\dot{v}_{\theta}^{\top},H\rangle]^{2} Then, since H is supported on the Stiefel manifold V_{d,k} : \left|\partial_{\theta}^{2}\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)\right|\leq 2k\|v-w\|_{F}^{2}\|x\|_{F}. ∎ Lemma H.5 . Let d>2 , x\in\mathbb{S}_{+}^{k}(\mathbb{R}) and \theta,\theta^{\prime}\in\mathbb{R} . Then, \frac{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta^{\prime}}x^{1/2}\right)}{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)}\leq\exp\left(|\theta^{\prime}-\theta|\sqrt{k}\|v-w\|_{F}\|x\|_{F}^{1/2}\right). Proof. Let d>2 , x\in\mathbb{S}_{+}^{k}(\mathbb{R}) and \theta,\theta^{\prime}\in\mathbb{R} . \displaystyle\frac{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta^{\prime}}x^{1/2}\right)}{{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)} \displaystyle=\exp\left(\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta^{\prime}}x^{1/2}\right)-\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right)\right) \displaystyle=\exp\left(\int_{\theta}^{\theta^{\prime}}\partial_{u}\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{u}x^{1/2}\right)du\right) \displaystyle\leq\exp\left(|\theta^{\prime}-\theta|\sqrt{k}\|v-w\|_{F}\|x\|_{F}^{1/2}\right), giving the desired result. ∎ Lemma H.6 . Let x\in\mathbb{S}_{+}^{k}(\mathbb{R}) and d>1 . Then, {}_{0}F_{1}(d/2,x)\leq e^{\sqrt{k}\|v_{\theta}\|_{F}\|x\|_{F}^{1/2}}. Proof. Let x\in\mathbb{S}_{+}^{k}(\mathbb{R}),d>1,t\in\mathbb{R} . Then, leveraging the integral representation of the {}_{0}F_{1} function from Proposition H.3 : \displaystyle\log{}_{0}F_{1}\left(\frac{d}{2},\frac{1}{4}x^{1/2}\Omega_{\theta}x^{1/2}\right) \displaystyle=\int_{V_{d,k}}e^{\langle x^{1/2}v_{\theta}^{\top},H\rangle}d\mu(H)\leq\sup_{H\in V_{n,d}}e^{\langle x^{1/2}v_{\theta}^{\top},H\rangle}\leq e^{\sqrt{k}\|v_{\theta}\|_{F}\|x\|_{F}^{1/2}}. ∎