Internal Energy, Fundamental Thermodynamic Relation, and Gibbs’ Ensemble Theory as Emergent Laws of Statistical Counting Authors: Hong Qian, Antonio Prados Published: 2024 ABSTRACT AbstractStatistical counting ad infinitum is the holographic observable to a statistical dynamics with finite states under independent and identically distributed N sampling. Entropy provides the infinitesimal probability for an observed empirical frequency ν^ with respect to a probability prior p, when ν^≠p as N→∞. Following Callen’s postulate and through Legendre–Fenchel transform, without help from mechanics, we show that an internal energy u emerges; it provides a linear representation of real-valued observables with full or partial information. Gibbs’ fundamental thermodynamic relation and theory of ensembles follow mathematically. u is to ν^ what chemical potential μ is to particle number N in Gibbs’ chemical thermodynamics, what β=T−1 is to internal energy U in classical thermodynamics, and what ω is to t in Fourier analysis.Keywords: emergent phenomenon, entropy, information, internal energy, probability theory, statistic, thermodynamics 1. Introduction It is a pleasure to be a part of this celebration for Signe Kjelstrup. She has made significant contributions to nonequilibrium thermodynamics, in both theory and applications that include electrochemistry, transport in heterogeneous media, and T. L. Hill’s small systems [1,2,3]. In this work we extend Gibbs’ and Hill’s approach to equilibrium thermodynamics [4] and show that the new logical path via the “crucial step” advocated in [5] is in fact a consequence of a limit theorem [6] in the mathematical theory of probability [7,8]. The results perfectly fit P. W. Anderson’s notion of emergent phenomenon [9]. Sometimes a mathematical transform can provide a fundamental concept beyond just being a technique for solving a problem, and through which a new representation of a natural phenomenon emerges. A case in point is the Fourier transform (FT) that leads to the theory of harmonics in music instruments [10] and the very concept of optical spectrum. FT represents a function of time f(t) in terms of f˜(ω), where ω is introduced as a novel notion, the temporal frequency of a sinusoidal oscillatory component in time [11]. The solutions to a large class of problems in differential calculus involving t can be very efficiently expressed through FT. We show in the present paper that the fundamental notion of internal energy first appeared in the theory of thermodynamics in the 19th century, collectively developed by J. R. von Mayer, W. Rankine, R. Clausius, and W. Thomson among many others [12], is a concept that can be understood, and generalized, in statistical counting. The transformation in question is the Legendre–Fenchel transform (LFT) [13,14], a more refined mathematical formulation of the traditional Legendre transform [15]. When a simple statistical analysis is carried out on a set of data, correlated or not, it is usually supposed that they are from an identical probability distribution. One of the best understood systems that exhibit an invariant probability is the ergodic dynamical system [16]. The ergodic theory of classical Hamiltonian dynamics has been an intense research area in both physics and mathematics for more than a century [17,18]. Even when the data are from seemingly different “objects”, say different individuals within a biological species, it is understood that an ergodic mating or mutational process is behind the statistical practice; and the conclusions drawn are most meaningful in this regard. Such an ergodic stochastic dynamic perspective has transformed cell biology through the notion of phenotypic switching in recent years [19]. 2. Energetic Theory of Statistical Counting Let us consider the repeated statistical samples ad infinitum of a system with finite state space S={0,1,⋯,n}. In the present work we shall restrict our discussion for independent and identically distributed (i.i.d.) samples. More general sampling of Markov data will be published elsewhere. The number counting ν=(ν0,⋯,νn) with ν0+⋯+νn=N and counting frequency ν^=ν/N, not to be confused with the ω in FT above, has a homogeneous degree 1 neg-entropy function with respect to a given probability prior p=(p0,⋯,pn) [8,20]: The Appendix A provides the mathematical origin of the non-negative Φ(ν) as a result of statistical counting. In information theory, it is interpreted as the “surprise” in observing the ν under the assumption p [21,22]. It is a double-edged sword which tells the rareness of ν (or ν^) with respect to p or erroneous model p with respect to empirical ν. The Kolmogorov probability p has two rather different roles in statistical inference and in statistical physics. In the former it has been identified as only half to the other “half of probability theory as it is needed in current applications—the principles for assigning probabilities by logical analysis of incomplete information—is not present at all in the Kolmogorov system” [20]. The application of modern probability to statistical physics involves the limit of sample ad infinitum represented by N→∞ [6]. In this case, the probability p is for all the systems with the same state space S; it is not meant to be realistic for any particular system. It simply provides a “metric” under which each and every particular system has its own representation in terms of its complete information, ν. The Φ(ν) is introduced to further gauge the differences among systems with different ν’s on the same S; it becomes a “theory of everything.” Because the N→∞ limit, there are no uncertainties in ν^; it is a definitive characterization of an i.i.d. statistical distribution with state space S. Therefore, statistical inference is about the mathematical model of a particular system, and statistical physics is about the mathematical representation of all systems with the same S under the supposition of i.i.d. data ad infinitum. The entropy in (1) is an emergent characterization in the limit of N→∞, with the starting point in terms of generative models [9]. It provides the relationship between ν and p in the sampling process. It is an Eulerian degree 1 homogeneous function of ν: Φ(λν)=λΦ(ν). This fits naturally to the fundamental thermodynamic postulate formulated by H. B. Callen [23]. The LFT of Φ as a function of the normalized ν^ then yields [13,14,24]: with corresponding optimal ν^*(u) Note that the second equation in (2b) is obtained when one uses calculus to solve the infimum in (2a); this recovers the traditional Legendre transform. Normalizing ν to ν^ induces a gauge freedom in (2), an arbitrary additive constant to ui. In statistical thermodynamics, the conjugate variable uk introduced in Equation (2) has been interpreted as the internal energy of the state k, in kBT unit [25]; then ν^·u is the mean internal energy of “the statistical system”. In a real-world laboratory working on a particular system, the ν tends to infinity as N→∞ but ν^ converges to the intrinsic property of the statistical system. In statistical inference, the assumed p, as a prior, then is expected to be replaced by the observed, real, posterior ν^ according to conditional probability and/or Bayesian statistical logic [24,25]. This concludes the statistical investigation of the particular system with respect to the type of observations. The neg-entropy function in (1) actually provides a meta-statistical theory for all possible observed ν^, assessing their respective infinitesimal probability (rate) with respect to the prior p (see Appendix A). 3. Maximization of Entropy −Φ Under Constraint by Empirical Mean Value However, The complete counting for the entire state space S in terms of empirical frequencies ν^ is only a gedankenexperiment. The significance of Gibbs’ ensemble theory is in dealing with observations from a small set of real-valued observables g1(i),g2(i),⋯,gJ(i), where i∈S but J≪n. These g’s are random variables on the state space S. In fact, their empirical mean values are linear combinations of the ν^: To fix mathematical notations, we append g0(i)=1 and x0=1, which represent the fact that ν^ is always normalized, and denote (n+1)×(J+1) matrix GJ with elements Equation (3) shows that if all the g’s are linearly independent and J=n, then one can solve the normalized ν^ uniquely from each set of x’s: ν^=xGn−1. We refer to such a set of observables as holographic with full information. In the following discussion, we shall always imagine the (g1,⋯,gJ) as the first J component of a holographic observable (g0, g1,⋯,gn). When JΦ(ν′). This is the origin of the maximum entropy principle (MEP). To understand the limit P(N)(ν)→0, one can also introduce the probability generating function [8]: in which u·ν=u0ν0+⋯+unνn. Then, Equation (A1) becomes The free energy function Ψ is meaningful for all finite N. This is why the partition function is valid even for small systems in Gibbs’ theory of ensembles [13]. The Legendre–Fenchel transform of Ψ(u) is precisely the the right-hand side of (A3): in which the optimal e−ui∝νi/pi. Legendre–Frenchel transform arises in the limit of N→∞ through the Laplace’s method of evaluating asymptotic integrals, or the related Darwin–Fowler method of maximum term. The analysis in this Appendix suggests that a proper interpretation of p=(p0,⋯,pn) in Equation (A1) is not as an intrinsic property, for example, the generative model of data statistics, rather it should be interpreted as a choice of a “gauge” in terms of which a set of counting data is represented: each particular set of data ad infinitum is represented by the energy function u, not p, and the ν^ is gauge invariant via the Boltzmann relation ν^i∝pie−ui—this yields an i.i.d. generative model. Probability is not for generative models, it is for analyzing empirical measurements on random variables. Data Availability Statement The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author. Conflicts of Interest The author declares no conflicts of interest. Funding Statement This research received no external funding. Footnotes Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. References Associated Data Data Availability Statement The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.