Reconsideration of Information-Theoretic Principles—Perspective from the Dual Probability Distribution Authors: Yoshikazu Ohtaki, Tomomi Nakamura, Hiroshi-H. Hasegawa, Tatsuaki Wada Published: 2026 ABSTRACT We reconsider information-theoretic principles, such as the maximum entropy principle/minimum Massieu potential principle, from the perspective of the dual probability distribution. This is introduced through Sanov’s Lemma for the multinomial distribution. The dual correspondence becomes asymptotically manifest. The Massieu potential is rewritten as the Kullback–Leibler divergence between the dual probability distribution and the dual reference distribution. Similarly, the dual potential is rewritten as the cumulant generating function with respect to the dual reference distribution. This perspective gives us new insight into information-theoretic principles. As the dual probability distribution naturally arises in data sampling, we anticipate that this new perspective will play a significant role in data analysis. 1. Introduction In information geometry, probability distributions are treated as points in a parameter space, and the ‘distance’ between them is treated as divergence [1,2]. The well-known Kullback–Leibler (KL) divergence/relative entropy is not symmetric with respect to the exchange of two probability distributions, meaning it does not satisfy the axiom of distance. Symmetry plays a significant role in physics. Attempts to introduce symmetric divergences, such as the Jensen–Shannon divergence [3], are significant. However, Amari et al. focused on the hidden dual structure behind asymmetric divergences instead, revealing the dual flat structure in the Pythagorean theorem [1,2]. The dual probability distributions, also known as ‘statistically dual distributions’ [4], refer to a pair of distributions in which the roles of the parameter and the stochastic variable are interchanged. For example, in a canonical distribution ρExβ=exp−βEx with a constant temperature T, the parameter is the inverse temperature β=1kT, and the stochastic variable is the energy Ex of a state x. We can consider its dual distribution ρβyE, where these roles are swapped: the energy E is treated as the parameter and the inverse temperature β(y) of a state as the stochastic variable y. At the same time, the temperature in a canonical distribution is usually a fixed number that does not fluctuate. In contrast, in a real system, in the case of a finite heat bath, its temperature can fluctuate. In this case, the parameter is given as the average of the fluctuating temperature. The dual probability distribution accounts for the variability of the parameter that is fixed in the thermodynamic limit. In Section 2, we provide a brief review of the information geometry of the exponential family on the n-dimensional simplex Sn, with a given reference distribution. The Massieu potential [5] becomes the cumulant generating function with respect to the reference distribution. The dual parameter vector is generated as an average of the Massieu potential via Legendre transformation, and the dual potential becomes the KL divergence between the probability distribution and the reference distribution. In Section 3, we introduce the concept of a dual probability distribution. According to Sanov’s Lemma [6], the dual potential can be interpreted as an exponent of the sampling distribution generated by the reference distribution. The integral representation of this exponent introduces new stochastic variables that follow a dual probability distribution. This yields a dual-symmetric correspondence asymptotically. The dual potential can be rewritten as a cumulant generating function with respect to the dual reference distribution. The parameter vector can be generated as an average from the dual potential using the inverse Legendre transformation. The Massieu potential can be expressed as the KL divergence between the dual probability distribution and the dual reference distribution. Our derivation is based on Stirling’s approximation. Therefore, the Massieu/dual potential requires modification for finite N. Upon estimating the correction term, we found that the O(logN/N) term arising from Stirling’s formula is negligible, and that the correction to the Massieu/dual potential is a convex function of order O(1/N). In Section 4, we reconsider the maximum entropy principle and its dual in the context of the dual probability distribution. The maximum entropy principle is derived asymptotically from the Pythagorean theorem based on the orthogonality between the canonical geodesic and the iso-energy surface. For the dual principle, the minimum Massieu potential principle, we introduce an explicit dual canonical distribution. The exponent of this distribution is expressed by one dual parameter and a dual energy function. This principle is also derived asymptotically from the Pythagorean theorem based on the orthogonality between the dual canonical geodesic and the dual-iso-energy surface. We expect new insights to play an important role in data analysis, given that dual probability distributions naturally arise in data sampling. Specifically, we intend to apply our argument to the analysis of time series for simple chaotic systems [7], the probability distribution of which evolves according to a gradient flow equation [8,9,10]. These decay modes are known as the Ruelle–Pollicott resonances. Although originally introduced in connection with decay in classical chaotic systems, it has recently attracted attention in relation to irreversibility in quantum many-body chaos [11]. In Section 5, we present the results of applying our discussion to a simple chaotic map system. We simulate the time evolution for several initial states. In particular, we focus on the time evolution of the Massieu potential to clarify its information-geometric meaning. In Section 6, we summarize these duality correspondences and comment. 2. Exponential Family in n-Dimensional Simplex 2.1. Exponential Family in n-Dimensional Simplex The exponential family is a class of probability distributions commonly encountered in information geometry. For simplicity, we will mainly deal with Sn in this section. In data analysis, a probability distribution is represented as a histogram of empirical data. In practice, any probability distribution can be approximated by Sn. A probability distribution in Sn is written as the following representation of the exponential family in information geometry [1,2],(1)pxθ=expθ·δx−ψθp¯x, where a stochastic variable, x=0,1,…,n, n-dimensional parameter vector: θ=θ1, θ2,…,θn and θ·δx≡∑i=1nθiδx,i. And p¯x is a reference distribution, p¯x>0 and ∑xp¯x=1. To satisfy the normalization condition, ∑xpxθ=1,(2)expψθ=Ep¯expθ·δx, where we write the expectation value with respect to distribution px as, Epx·≡∑x=0n·px. Then, the Massieu potential is written as the cumulant generating function with respect to the reference distribution,(3)ψθ=logEp¯expθ·δx=−logp0p¯0. The i-th element of the n-dimensional parameter vector θ is written as,(4)θi=logpip¯ip¯0p0. Since the Massieu potential is a convex function, the Hessian,(5)∂θi∂θjψθ, is the positive definite matrix. 2.2. Legendre Transformation The dual parameters are defined by the Legendre transformation based on the Massieu potential,(6)θi*=∂θiψθ. They are written as the average of δx,i with respect to px,(7)θi*=Epxδx,i=pi. The dual potential is also defined as,(8)φθ*=θ·θ*−ψθ=DKLp:p¯, where the KL divergence with respect to px,(9)DKLp:p¯=∑x=0npxlogpxp¯x, which reduces to the well-known negentropy for the uniform reference distribution up to a constant. 2.3. Divergences In the information geometry, a probability distribution p(θ) in Sn is represented as a point θ in the n dimensional parameter space. The information distance between two probability distributions, p(θA) and p(θB), is given by the Bregman divergence based on the convex function, the Massieu potential ψθ,(10)DψθA:θB=ψθA−ψθB−θA−θB·∇θBψθB. Using the dual potential, the Bregman divergence is rewritten as follows,(11)DθA:θB*=ψθA+φθB*−θA·θB*. We mainly use this divergence without any subscript in this article, since it has a dual symmetric form under the Legendre transformation. The divergence is related to the KL divergence as,(12)DθA:θB*=DKLpθB:pθA, where we used the following cross-entropy with the reference distribution,(13)−EpθBlogpθAp¯=−θA·θB*+ψθA. 3. Dual Probability Distribution 3.1. Dual Probability Distribution We will consider how the parameter vector, θ, fluctuates. Although parameters are usually treated as constants, we will demonstrate that they can be interpreted as the expected value of certain stochastic variables. The dual potential, the KL divergence with respect to px, is considered as the logarithm of the multinomial distribution in N independent trials,(14)−1NlogPMDNpp¯=−1NlogN!N0!N1!…Nn!p¯0N0p¯1N1…p¯nNn=DKLp:p¯+OlogNN, where px=Nx/N. The derivation is based on the Stirling formula for large N [2]. This relation is known as Sanov’s Lemma [6] in the large deviation theory [12]. Using the integral form of the Gamma function,(15)N!=ΓN+1=∫−∞∞eNyey−eydy, we can introduce dual stochastic variables, yx x=0, 1,…,n. We can rewrite the dual potential as the cumulant generating function with respect to the dual stochastic variables up to a constant and a negligible function,(16)φθ*−∆φNθ*=1NlogEP¯yeNθ*·y~+cN, where y~i=yi−y0 i=1,…,n, the dual reference distribution,(17)P¯y=NNN!p¯0NeNy0e∑x{yx+log(Np¯x)−eyx+log(Np¯x)}, and the constant cN and a convex function ∆φN(θ*) defined as follows,(18)cN≡−logp¯0−n+12Nlog2πN+1−logN+1NlogN!,(19)∆φNθ*≡−1N∑x=0nlog(Npx)!−NpxlogNpx−Npx−12log2πN=−1N∑x=0n12logpx+λNx=O1N. where 112N+1<λN<112N is known as Robbins’ bound [13]. We note that ∆φNθ* is a convex function for N large so that φθ*−∆φNθ* is interpreted as a modified dual potential. These derivations are shown in Appendix A. Since the modified dual potential is the cumulant generating function up to a constant, the inverse Legendre transformation gives the following modified parameter vector as averages of the dual stochastic variables,(20)θ−∆θ=∇θ*φθ*−∆φNθ*=EPy|θ*y~, where the dual probability distribution,(21)Py|θ*=eN{θ*·y~−φθ*+∆φNθ*+cN}P¯y. The modified Massieu potential is rewritten as the following scaled KL divergence between Py|θ* and P¯y up to a constant,(22)ψθ−∆ψNθ=EPy|θ*θ*·y~−φθ*+∆φNθ*+cN−cN=1NDKLPy|θ*:P¯y−cN. The Massieu potential can be considered as the information/negentropy of the dual stochastic variables. The dual stochastic variables represent fluctuations in the distribution of the data sample. The order of the deviation is O(1/N). From the Central Limit Theorem, the distribution becomes a normal distribution:(23)Py≅Ny¯,Σ, where y¯x=logpx−logp¯x+1, and Σx,x’=δx,x′/(Npx). Although fluctuation is negligible for large N, it is important for finite N. For example, the inverse of temperature is the typical parameter in the canonical distribution. Its fluctuation is negligible for a large reservoir in the thermodynamical limit. But the temperature can fluctuate in a finite system. 3.2. Divergence The KL divergence between Py|θA* and Py|θB* is rewritten by the modified potentials as,(24)1NDKLPy|θA*:Py|θB*=ψθA−∆ψNθA+φθB*−∆φNθB*−θA−∆θA·θB*, where we used (22) for the modified Massieu potential and the following dual cross-entropy derived from (20) and (21),(25)−1NEPy|θA*logPy|θB*P¯y=−(θA−∆θA)·θB*+φθB*−∆φNθB*−cN. The KL divergence between the dual probability distributions, Py|θA* and Py|θB*, is given as the following modified divergence between θB and θA* as,(26)1NDKLPy|θA*:Py|θB*=DθA:θB*−∆DNθA:θB*, where ∆DNθA:θB* is a Bregman divergence while ∆φNθ* is a convex function with respect to θ*,(27)∆DNθA:θB*≡∆ψNθA+∆φNθB*−∆θA·θB*=O1N. The KL divergence between Py|θA* and Py|θB* asymptotically becomes the dual KL divergence between pθB and pθA in the limit of large N,(28)limN→∞DKLPy|θA*:Py|θB*=DKLpθB:pθA. 3.3. Pythagorean Theorem The Pythagorean theorem for the KL divergences between the dual probability distributions is given as follows,(29)1NDKL[Py|θA*:Py|θC*]−1NDKL[Py|θA*:Py|θB*]−1NDKL[Py|θB*:Py|θC*]=DθA:θc*−∆DNθA:θc*−DθA:θB*+∆DNθA:θB*−DθB:θC*+∆DNθB:θC*=θA−∆θA−θB+∆θB·θB*−θC*=0, where the parameter geodesic is modified. 4. Maximum Entropy Principle and the Dual Principle 4.1. Maximum Entropy Principle We introduce a canonical parameter following the one-parameter geodesic,(30)θcanβ−∆θcanβ=−βE, that is corresponding to the canonical distribution in physics,(31)pθcanβ−∆θcanβ=exp−βEx+F(β)p¯x. The parameter β is the inverse temperature included in the Boltzmann constant, Ex is the n+1-dimensional energy function, Ex=E·δx≡∑i=1nEiδx,i and E=E1,…En where we choose E0=0 for which the free potential is given as the negative Massieu potential,(32)Fβ≡βFβ=−ψθcanβ−∆θcanβ. Note that Fβ is the free energy and Fβ is the dimensionless potential. The orthogonal condition for the Pythagorean theorem in (29) to hold can be rewritten as for θA−∆θA=θcanα−∆θcanα, θB−∆θB=θcanβ−∆θcanβ, θB*=θcan*β=pθcanβ and θC*=θ*=p(θ),(33)α−βE·θcan*β−θ*=α−βE·pθcanβ−E·pθ=0. This implies that the canonical geodesic is perpendicular to the iso-energy surface, as is shown in Figure 1. Under this orthogonal condition, the Pythagorean theorem holds as,(34) 1NDKLPy|θcan*α:Py|θ*=1NDKLPy|θcan*α:Py|θcan*β+1NDKLPy|θcan*β:Py|θ*. The maximum entropy principle is derived under the iso-energy condition. From (34), the following inequality is derived,(35) 1NDKLPy|θcan*α:Py|θ*≥1NDKLPy|θcan*α:Py|θcan*β. By using (20) and (30),(36)αE·θ*−φθ*+∆φNθ*≤αE·θcan*β−φθcan*β+∆φNθcan*β. From the iso-energy condition, we rigorously obtained the following inequality,(37)−φθ*+∆φNθ*≤−φθcan*β+∆φNθcan*β. The maximum entropy principle is derived in the limit of large N,(38)−φθ*≤−φθcan*β. We note that the dual potential is the negative entropy for the uniform reference distribution up to a constant. 4.2. Minimum Massieu-Potential Principle We introduce a dual canonical parameter following the one-parameter dual-geodesic,(39)ϑcan*κ·y~=κϑcan*λ·y~/λ=−κEy~, that is corresponding to the dual canonical distribution,(40)Py|ϑcan*κ=exp−κEy~−φϑcan*κ+∆φϑcan*κ+cNP¯(y). Two dual canonical parameters are connected through a scale transformation in (39), ϑcan,i*κ=κϑcan,i*λ/λ i=1,…,n. However, ϑcan,x*κ= pcan,xϑcanκ in Sn, the normalization of pcan,xϑcanκ is the same as pcan,xϑcanλ due to the following relation between their respective zero-components,(41)ϑcan,0*κ=1−κλ1−ϑcan,0*λ>0. The orthogonal condition in the Pythagorean theorem can be rewritten as for θA−∆θA=θ−∆θ=EPy|θ*[y~], θB−∆θB=ϑcanλ−∆ϑcanλ=EPy|ϑcan*λy~ , θB*=ϑcan*λ, and θC*=ϑcan*κ=κϑcan*λ/λ ,(42)θ−∆θ−ϑcanλ+∆ϑcanλ·ϑcan*λ1−κ/λ=EPy|ϑcan*λEy~−EPy|θ*[Ey~]λ−κ=0. This implies that the canonical dual geodesic is perpendicular to the iso-dual-energy surface, as is shown in Figure 2. Under this orthogonal condition, the Pythagorean theorem holds as,(43) 1NDKLPy|θ*:Py|ϑcan*κ=1NDKLPy|θ*:Py|ϑcan*λ+1NDKL[Py|ϑcan*λ:Py|ϑcan*κ]. The minimum Massieu-potential principle is derived under the iso-dual-energy condition. From the Pythagorean theorem (43), the following inequality is derived,(44) 1NDKLPy|θ*:Py|ϑcan*κ≥1NDKL[Py|ϑcan*λ:Py|ϑcan*κ]. After canceling φϑcan*κ−∆φNϑcan*κ from both sides, the inequality is rewritten as,(45)ψθ−∆ψNθ−θ−∆θ·ϑcan*κ≥ψϑcanλ−∆ψNϑcanλ−ϑcanλ−∆ϑcanλ·ϑcan*κ, where we used (24). From the iso-dual-energy condition in (42), we rigorously obtained the following inequality,(46)ψθ−∆ψNθ≥ψϑcanλ−∆ψNϑcanλ. The minimum Massieu-potential principle is derived in the limit of large N,(47)ψθ≥ψϑcanλ. 5. Application to a Simple Chaotic Map System We will reconsider the time evolution of a simple chaotic map system from the information-geometric viewpoint. We consider the following chaotic map of the unit interval into itself, satisfying the following equation:(48)zt+1 =fzt =1−azta+aif zt∈0,azt−a1−aif zt∈a,1, where 0a, where we choose,(58)θt=0z=−βz−1+a2, so that the initial state lies on the iso-dual-energy surface,(59)θt=0·ϑinv*=11−a2∫a1θt=0zdz=0, since θt=0z is an anti-symmetric function within the interval a,1. We replaced the dot product with the integration. The initial Massieu potential, ψt=0, is determined by the normalization condition,(60)eψt=0=pinv,0+1−pinv,0∫a1eθt=0zdz. After integrating the right-hand side, we obtain,(61)ψt=0=log1+11+a2β1−asinhβ21−a−1=1611+aβ21−a2−…. Then,(62)pt=0,0=11+2aβ(1−a)sinh{β21−a}. In the numerical simulation, we prepared a total of N particles at t=0. A total of pt=0,0N particles are uniformly distributed in the interval [0,a) while the remaining particles are distributed exponentially as exp[−βz] in the interval [a,1). In this case, the following behavior can be expected: if the system moves on the iso-dual-energy surface orthogonal to the dual canonical geodesic line, the Massieu potential smoothly decreases to its minimum value of zero. As shown by the green line in Figure 4, this non-uniform initial distribution causes a large change in the particle number in the interval [0,a) at t=1, resulting in a significant jump in the absolute value of the Massieu potential. This implies that the state has deviated significantly from the iso-dual-energy surface. Although the state subsequently converges to the invariant distribution, its decay profile differs from the one along the dual canonical geodesic, suggesting a different underlying mechanism of decay. 6. Conclusions and Remarks In this paper, we introduce the dual probability distribution of the n-dimensional simplex Sn. According to Sanov’s Lemma for the multinomial distribution, the dual potential can be interpreted as an exponent of the sampling distribution generated by the reference distribution. The integral representation of this exponent introduces new stochastic variables that follow the dual probability distribution. The dual probability distribution yields asymptotically dual-symmetric correspondences. The dual potential can be expressed as a cumulant generating function with respect to the dual reference distribution. The parameter vector is obtained by averaging the dual potential using the inverse Legendre transformation. The Massieu potential is equal to the KL divergence between the dual probability distribution and the dual reference distribution. Information-theoretic principles, such as the maximum-entropy and minimum Massieu-potential principles, can be derived asymptotically from the dual-probability-distribution perspective. For the minimum Massieu potential principle, we introduced a dual canonical distribution explicitly, whose exponent is written using one dual parameter and a dual energy function. The minimum Massieu potential principle is derived asymptotically from the Pythagorean theorem, based on the orthogonality between the dual canonical geodesic and the iso-dual-energy surface. We expect new insights to play an important role in data analysis, given that dual probability distributions naturally arise in data sampling. Specifically, we intend to apply our argument to the analysis of time series for simple chaotic systems [10], the probability distribution of which evolves according to a gradient flow equation [11,12,13]. In this paper, we present the results of applying our discussion to a simple chaotic map system. We simulate the time evolution for two initial states: One is on the dual canonical geodesic, and the other is on the iso-dual-energy surface. The time evolution of the former initial state is comprehensible because it follows the gradient flow equation. However, the latter is difficult to understand because the dual-energy conservation does not hold. Finally, this paper provides comments on future research. Our arguments can be easily applied to the maximum work formulation (the dual formulation) using KL divergences scaled by the canonical parameter (the dual-canonical parameter) [16]. The Pythagorean theorem for the scaled divergences gives orthogonality between the canonical-parameter geodesic and the isentropic surface (the dual canonical-parameter geodesic and the iso-Massieu-potential surface). We expect that the information-geometric perspective gives us a clear understanding of the time evolution of chaotic systems in an isentropic process (iso-Massieu potential process).