How random connectivity shapes the fluctuating dynamics of finite-size neural populations Nils E. Greven, Jonas Ranft, Tilo Schwalger Abstract Mesoscopic models of finite-size neuronal populations are crucial to understand the dynamics of neural networks in the brain, especially their fluctuations and response to stimuli. However, current theories to derive such models are based on the assumption of homogeneous all-to-all (full) connectivity, neglecting the variance in the connectivity of biologically realistic networks with connection probabilities p < 1 p<1 (non-full connectivity). To gain insight into the different fluctuation mechanisms underlying the neural variability of populations of spiking neurons, we derive and analyze a stochastic mean-field model for finite-size networks of Poisson neurons with random connectivity (including non-full connectivity) , external noise and disordered mean inputs. We treat the quenched disorder of the connectivity by an annealed approximation enabling a doubly stochastic description of synaptic inputs for finite network size. A further reduction leads to a low-dimensional closed system of coupled Langevin equations for the mean and variance of the neuronal membrane potentials as well as a variable capturing finite-size fluctuations arising specifically in the case of connectivity disorder . Comparing to microscopic simulations, we find that the mesoscopic model describes the fluctuations and nonlinearities well and outperforms previous mesoscopic models that neglected the variance in the connectivity. The joint effect of connectivity disorder and finite network size can be analytically understood by a softening of the effective nonlinearity and the multiplicative character of spiking noise. The mesoscopic theory shows that quenched disorder can stabilize the asynchronous state, and it correctly predicts large quantitative and non-trivial qualitative effects of connection probability on the variance of the population firing rate and its dependence on stimulus strength. Our theory thus elucidates how disordered connectivity shapes nonlinear dynamics and fluctuations of neural populations at the mesoscopic scale and showcases a useful mean-field method to treat random connectivity in finite-size, spiking neural networks. I Introduction Biological systems often exhibit significant fluctuations and variability in their dynamics, in contexts ranging from gene expression to neural activity in the brain. The possible roles of fluctuations for the dynamics and functions of such systems have thus received increasing attention [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . An important tool to understand the collective dynamics of biological systems consisting of many interacting units are low-dimensional descriptions of the effective dynamics at the mesoscopic or macroscopic scale . Taking fluctuations properly into account can be necessary to establish a correct understanding of the system and can provide crucial insight into the origins of observed biological variability. At the mesoscopic scale, fluctuations naturally arise from the finite number of underlying units. These “finite-size” fluctuations may then be accounted for by stochastic low-dimensional models. In the context of living systems, examples range from Langevin equations for gene regulation [ 8 ] to coupled molecular motors [ 9 , 10 , 11 ] to ecology [ 12 ] . In neuroscience, neural-population models (also known as firing-rate or neural-mass models) such as the Wilson-Cowan model [ 13 ] have been highly successful in describing various cortical phenomena such as oscillations [ 14 ] , multistability [ 15 ] , and nonlinear response properties [ 16 ] . Stochastic versions of neural-population models have been critical for understanding fluctuating neural population dynamics such as metastability [ 17 , 18 , 15 ] , stochastic oscillations [ 19 ] , and response properties of cortical variability [ 20 , 21 ] . Despite their mathematical tractability, however, these models are often phenomenological, lacking a clear link to an underlying finite-size network of spiking neurons. A link to biophysical properties is maintained through bottom-up mean-field approaches [ 22 , 23 , 24 , 25 , 26 , 27 , 28 ] , where the macroscopic or mesoscopic dynamics is analytically derived from a microscopic model. The resulting mean-field dynamics are amenable to mathematical analysis [ 29 , 30 , 31 , 32 , 33 ] and enable efficient and accurate simulations of large spiking neural networks [ 34 , 35 , 36 , 37 ] . Recently, a bottom-up theory for the mesoscopic dynamics of finite-size populations of generalized integrate-and-fire neurons with escape noise has been developed [ 36 , 33 ] . It suggests an efficient multiscale modeling framework of cortical circuits [ 36 , 38 , 39 ] grounded in realistic neuron dynamics [ 40 , 41 ] . However, this and other mean-field theories [ 42 , 43 , 36 , 44 , 45 , 46 , 33 ] for finite-size neural networks typically assume homogeneous, all-to-all (full) connectivity. This assumption can be regarded as an approximation of a non-fully connected random network by its homogeneous mean connectivity while neglecting its quenched variability [ 47 , 36 ] . On the other hand, the role of random connectivity for intrinsically generated fluctuations of synaptic inputs has been extensively studied using dynamical mean-field theories [ 48 , 49 , 29 , 30 , 50 , 51 , 52 , 53 ] , albeit only in the limit of infinitely large networks. Both assumptions–homogeneous full connectivity ignoring quenched variability and infinite network size ignoring fluctuations–are severe limitations for describing biologically realistic, recurrent neural networks. For example, rat barrel cortex contains only on the order of N\sim 100 to N\sim 1000 neurons per cell type per layer [ 54 ] – a population size for which finite-size fluctuations can be significant, as shown in computational models [ 55 , 56 , 57 , 36 , 58 , 59 ] . Furthermore, the local connection probabilities p in mouse visual cortex vary on a wide range ( 0\lesssim p\lesssim 0.7 ) with low average ( p\sim 0.1 ) and larger connection probabilities ( p\sim 0.5 ) for similarly tuned neurons [ 60 ] . How such pronounced deviations from fully connected networks ( p=1 ) affect the mesoscopic dynamics of finite-size neural populations is still poorly understood. Both finite population size ( N<\infty ) and quenched random connectivity (e.g. p<1 ) cause intrinsically generated (endogenous) fluctuations of the recurrent, synaptic input currents. These intrinsic fluctuations act as multiplicative noise whose strength increases with population firing rates. On the other hand, neural variability in the cortex often decreases upon stimulation [ 61 ] – a phenomenon that could be explained, among others, with a model where variability is generated by an external, additive noise [ 21 ] . To clarify the role of intrinsic fluctuations for such cortical phenomena under biologically realistic neuron numbers and connection probabilities requires a mean-field theory that accounts for these different origins of fluctuations. In a homogeneous, fully connected network, as arising under a mean-connectivity approximation, the input fluctuations are common to all neurons. This is because all neurons receive identical recurrent inputs and thus experience the same finite-size fluctuations of the population activity. These “coherent” fluctuations lead to stochastic population equations with finite-size noise of order 1/\sqrt{N} . In contrast, in random networks with connection probability p<1 , neurons no longer receive identical synaptic inputs, and hence input fluctuations become partially decorrelated. Consequently, the assumption of perfectly coherent fluctuations no longer holds, raising questions about the validity of the mean-connectivity approximation for finite-size networks. A mean-field description of random networks (including non-fully connected networks, p<1 ) requires the treatment of fluctuations resulting from a “quenched” (i.e. temporally constant) random synaptic connectivity. Previous approaches to random networks considered the macroscopic limit N\to\infty . Depending on the concrete limiting procedure and network architecture, the recurrent fluctuations seen by different neurons either vanish [ 22 , 31 , 32 , 28 ] or become perfectly independent (“incoherent”) in this limit [ 48 , 50 , 29 , 30 , 51 , 52 , 53 ] unless correlations are induced externally. A famous example is the mean-field theory for sparsely connected networks by Brunel and co-workers [ 29 , 30 ] . In this theory, the neural population dynamics is deterministic and given by a nonlinear Fokker-Planck equation. The effect of incoherent fluctuations of the recurrent synaptic input caused by the random connectivity appears as an additional contribution to the diffusion coefficient in this equation. Although this approach relies on an annealing approximation that neglects temporal correlations in the spike trains caused by the quenched disorder, the nonlinear Fokker-Planck equation has been successful to capture the influence of intrinsic fluctuations on the nonlinear population dynamics (for a treatment of temporal correlations, see e.g. [ 62 ] ). Using this theory, intrinsic fluctuations have been shown to crucially shape nonlinear response properties [ 63 ] , network oscillations [ 29 , 64 ] and multistability [ 56 ] . Even though dynamical mean-field theories for N\to\infty have been used for real neural networks of finite size (see e.g. [ 65 ] ), they cannot describe finite-size fluctuations at the population level. Furthermore, finite-size networks are necessarily outside the sparse limit (i.e. p>0 ) which entails a non-vanishing probability of shared recurrent inputs among neurons. Therefore the assumption of perfectly incoherent noise in dynamical mean-field theories cannot hold fully true for N<\infty . While finite-size extensions of the mean-field theory for sparsely connected networks have been proposed in [ 29 , 30 , 66 ] , low-dimensional neural population models that reveal and take into account the distinct effects of coherent and incoherent fluctuations are currently lacking. In this paper, we consider finite-size networks with random connectivity, allowing arbitrary connection probabilities p\in[0,1] . For these networks, we derive and analyze a simple mesoscopic dynamics that describes incoherent and coherent recurrent fluctuations. To this end, we study a network of Poisson neurons using an annealed approximation [ 67 , 29 ] of the quenched random connectivity. The annealed approximation neglects temporal correlations of incoherent fluctuations but captures the effects of random connectivity and finite network size surprisingly well. In contrast to the case N\to\infty of previous theories [ 29 , 30 ] , we need to treat the fluctuating input as a doubly stochastic process to account for both finite N and random connectivity. This description eventually yields a three-dimensional Langevin equation for the mesoscopic dynamics . We compare this dynamics with a naive mean-field theory in which the variance of the random connectivity is neglected, corresponding to the ad hoc mean-field approximation used in previous studies [ 47 , 36 ] . While the focus of the paper is the Langevin dynamics for a homogeneous population with quenched or annealed random connectivity and common external noise, the theoretical framework can be extended to heterogeneous populations, where the resting potentials or external currents of the neurons exhibit quenched Gaussian disorder. In summary, we present a simple theoretical framework for the effects of various sources of noise and quenched disorder on the neural population dynamics at the mesoscopic scale ( 1\ll N<\infty ). The paper is organized as follows. We introduce the microscopic network model with quenched random connectivity and its annealed approximation in Sec. II . By reducing the annealed network to a mean-connectivity network with incoherent dynamical noise, we derive in Sec. III a mesoscopic population model as a system of three coupled stochastic differential equations. In Sec. IV , we then analyze the mesoscopic model with respect to fixed-point solutions and their stability, the linear response to dynamic stimuli and second-order statistics. In particular, we compare the obtained analytical results to the corresponding statistics of the quenched and annealed microscopic models using extensive simulations. We conclude and discuss our results in Sec. V . The extension to quenched Gaussian disorder as well as longer, detailed calculations are provided in the Appendix. II Microscopic models II.1 Quenched network model At the microscopic level, we consider a random network of N interacting Poisson neurons (also called nonlinear Hawkes processes). While our theory will be developed for general random connectivity with finite variance, we focus on the specific example of binary, non-full connectivity to study the effect of connection probabilities p<1 . The Poisson neurons fire spikes stochastically with conditional intensity \lambda_{i}(t)=\phi(h_{i}(t^{-})) , i=1,\ldots,N , where h_{i}(t) is the input potential of neuron i and \phi is a non-negative hazard function. The dynamics of the input potentials is given by a system of coupled first-order equations with delay, \displaystyle\tau\frac{\mathrm{d}h_{i}}{\mathrm{d}t} \displaystyle=-h_{i}+\mu(t)+\frac{w}{C}\sum\limits_{j=1}^{N}a_{ij}\dot{Z}_{j}(t-d), (1) i=1,\dotsc,N . Here, \tau is the time constant of the low-pass filter dynamics, \mu(t) is an external drive and \dot{Z}_{i}(t)=\sum_{l}\delta(t-t_{i,l}) is the spike train of neuron i with \{t_{i,l}\} being the individual spike times. The presence or absence of synaptic connections between neurons are described by the random adjacency matrix [a_{ij}] with fixed in-degree C=\sum_{j=1}^{N}a_{ij} and elements a_{ij}\in\{0,1\} that are marginally Bernoulli distributed with mean \left\langle a_{ij}\right\rangle=p=C/N representing the connection probability. This random connectivity can be constructed by choosing for each neuron C presynaptic neurons randomly, and the resulting connectivity is fixed (“quenched”) in time. Furthermore, d is the synaptic delay and w=CJ is the total coupling strength with J being the efficacy of a single synapse (i.e. J/\tau is the jump size in millivolt of the postsynaptic potential caused by a single presynaptic spike). Since we focus on inhibitory networks in this paper, we assume that J<0 . As a concrete hazard function, we choose here a sigmoidal function in the form of an error function [ 22 ] : \phi(h)=r_{\text{m}}\Phi(\beta(h-\vartheta)), (2) where \Phi(x)=\left[1+\mathrm{erf}\left(x/\sqrt{2}\right)\right]/2 . Although not essential for the general theory, this choice will allow us to analytically calculate the first and second moments of \phi(h_{i}) . The parameter \beta determines the steepness around the inflection point. Without loss of generality, we choose the position of the inflection point \vartheta to be zero because we can always measure voltages with respect to the potential \vartheta . The sigmoidal shape with an upper limit r_{\text{m}} prevents the Poisson neurons to fire with arbitrarily high rates. While the concave behavior and saturation of the sigmoidal hazard function for h>0 is technically useful, we are in the following mainly interested in dynamical regimes operating in the convex part of the hazard function \phi(h) , i.e. corresponding to mean inputs below the inflection point, h<0 . This is because biologically realistic hazard functions of cortical neurons are typically convex [ 68 , 69 ] . As default parameters of the model, we choose \tau=20 ms, r_{\text{m}}=100 Hz and d=0 ms unless otherwise noted. The aim of this paper is to derive a mesoscopic population model that generates population activities that statistically match the population activities obtained from a microscopic network simulation. We define the population activity A_{N}(t,\Delta t) with respect to a time discretization with time step \Delta t as the total number of spikes per neuron and time step: A_{N}(t;\Delta t)=\frac{\Delta Z(t)}{N\Delta t}. (3) Here, \Delta Z(t) denotes the total number of spikes in the time interval (t,t+\Delta t] . II.2 Mean-connectivity network Figure 1: Response of the population rate variance to a change of the mean stimulus strength in the original, quenched random network (a) and in a corresponding fully-connected, mean-connectivity network that only accounts for the mean synaptic weight (b). Top: Noisy external stimulus \mu(t) whose mean increases at t=0.08 s in a step-like fashion from \mu_{0}=28 mV to \mu_{0}=50 mV; the noise strength \sigma_{\text{ext}}^{2}=1 mV 2 is kept fixed. (a) Schematic of a quenched network with fixed adjacency matrix a_{ij} (left) and the trial-averaged variance of the population firing rate as a function of time (right) for different network sizes (green line: N=5\cdot 10^{4} , blue line: N=10^{3} , in both cases C=100 ). (b) Same for a mean-connectivity network, Eq. ( 5 ), as an ad hoc approximation of the quenched network, where the quenched connectivity a_{ij} is replaced by the average connectivity \langle a_{ij}\rangle . While the mean stationary population rates after the step are equal in both models (measured rates are provided in the right panels) , the variances of the population rate strongly differ (note the different scaling of the y -axis). Other parameters: \beta=5 mV -1 , w=-1 mVs. A major obstacle in deriving population models analytically is the quenched randomness of the connectivity J_{ij}:=Ja_{ij} . A standard mean-field approach to tackle this problem is to neglect fluctuations in the recurrent synaptic input by scaling the synaptic weights inversely proportional to the network size, J_{ij}=w_{ij}/N , where the coupling strength w_{ij} is assumed to be of order 1 . Then, in the limit N\to\infty , the synaptic input N^{-1}\sum_{j}w_{ij}\dot{Z}_{j} converges to the deterministic input \left\langle w_{ij}\right\rangle\langle\dot{Z}_{j}\rangle [ 27 , 28 ] , where \left\langle\cdot\right\rangle denotes the average over both quenched disorder and Poisson noise. In our case, this corresponds to a network where the quenched connectivity w_{ij} (Fig. 1 a, left) is replaced by the mean connectivity \left\langle w_{ij}\right\rangle=NJ\left\langle a_{ij}\right\rangle{\color[rgb]{0,0,0}=CJ}=w (Fig. 1 b, left). Importantly, the mean connectivity effectively yields a fully connected network with homogeneous coupling w . In the limit N\to\infty , the synaptic input is thus not affected by the variance \sigma_{w}^{2}=w^{2}(1-p)/p (4) of the disordered couplings w_{ij} but only by the mean \left\langle w_{ij}\right\rangle . The replacement of all elements of the adjacency matrix a_{ij} with their mean p has been used ad hoc also for finite network size N<\infty [ 47 , 36 ] . However, this ad hoc approach is no longer exact for finite N but must be regarded as an approximation that retains the mean synaptic connectivity \left\langle J_{ij}\right\rangle{\color[rgb]{0,0,0}=pJ} but ignores all higher-order cumulants of the random variables J_{ij} . In the following, we will refer to this effective, fully connected network as the mean-connectivity network , whereas the original model, Eq. ( 1 ), will be referred to as the quenched network (Table 1 ) . Replacing the connection strengths w_{ij} with their means w , we obtain from Eq. ( 1 ) the dynamics of the mean-connectivity network: \tau dh_{i}(t)=\left[-h_{i}+\mu(t)\right]dt+\frac{w}{N}\sum_{j=1}^{N}dZ_{j}(t-d), (5) where dZ_{j}(t)\sim\text{Pois}(\phi(h_{j}(t))dt) are the conditionally independent Poisson increments of the spike count of neuron j . The mean-connectivity network of Poisson neurons, Eq. ( 5 ), is a microscopic model that admits an exact reduction to a mesoscopic model for the total spike count dZ(t)=\sum_{i=1}^{N}dZ_{i}(t) . To see this, we note that the sum of independent Poisson numbers in Eq. ( 5 ) is a Poisson number with mean Nr(t-d)dt , where r(t)=\frac{1}{N}\sum_{i=1}^{N}\phi\bigl(h_{i}(t)\bigr). (6) is the stochastic population rate. Furthermore, because all neurons are driven by the same synaptic input, their input potentials h_{i}(t) coincide (or converge after an initial transient). The common input potential \bar{h}(t) thus obeys the mesoscopic dynamics \tau d\bar{h}(t)=\left[-\bar{h}+\mu(t)\right]dt+\frac{w}{N}dZ(t-d), (7) where now dZ(t)\sim\text{Pois}(N\phi(\bar{h}(t))dt) describes the increment of the total spike count of the population. Equation ( 7 ) describes the mesoscopic dynamics as it only involves the population activity A_{N}(t,\Delta t)=\int_{t}^{t+\Delta t}dZ(t)/(N\Delta t) . Simulations of the quenched and the mean-connectivity network yield similar results for the stationary mean firing rates and mean input potentials. However, the mean-connectivity network may differ drastically compared to the quenched network when it comes to the non-stationary response (linear response function) and the fluctuations (second-order statistics). To illustrate the failure of the mean-connectivity approximation, let us consider the variance of the stochastic population rate r(t) . This variance represents the part of the mesoscopic neural variability (variance of the population activity A_{N}(t,\Delta t) ) that is caused by the rate variability rather than the Poisson spiking noise. It can be measured from population activity data (see Appendix, Sec. C ), or directly computed from a simulated time series of r(t) in our model. A non-vanishing variance of the population rate can occur for two reasons in our model: first, because of intrinsic finite-size fluctuations, and second, because of externally injected common noise. Here, we model the common external noise as a Gaussian white noise (more precisely, \mu(t)=\bar{\mu}(t)+\sqrt{\tau\sigma_{\text{ext}}^{2}}\hat{\zeta}(t) , where \bar{\mu}(t) is the mean stimulus and \hat{\zeta}(t) is a standard Gaussian white noise process). Interestingly, simulations of the mean-connectivity network (Eq. ( 5 ) or ( 7 )) strongly overestimate the population-rate variance of the quenched network by more than one order of magnitude (Fig. 1 , note the different scale of the y axes). We stress that this difference is not an effect of a difference in firing rates: The stationary mean population rates \left\langle r\right\rangle are the same in both networks (e.g., the numerical values of the firing rates at high stimulus amplitudes are approximately equal as indicated in Fig. 1 a,b). In addition to this large quantitative difference, we also observe a qualitatively different response to fast stimulus changes when the network is large ( N=5\cdot 10^{4} , green lines in Fig. 1 ) : When the mean stimulus strength jumps from 28 mV to 50 mV, the variance of the population rate exhibits a small but significant decrease in the quenched network (Fig. 1 a, right). This suppression of variability is in marked contrast to the prediction by the mean-connectivity network, which exhibits instead an increase in the variance (Fig. 1 b, right). This qualitatively different behavior of the population-rate variance in response to a step stimulus is, however, not observed for smaller network size ( N=10^{3} , but otherwise same parameters and similar firing rates; blue lines in Fig. 1 ). How can we understand non-stationary responses and neural variability of mesoscopic variables theoretically? In this paper, we present a “second-order” mesoscopic mean-field theory that also captures second-order statistics and nonstationary responses quantitatively, and explains non-trivial, qualitative phenomena such as the suppression of variability observed in Fig. 1 a. In particular, the explanation of the failure of the mean-connectivity approximation with respect to the stationary variances in Fig. 1 b and its resolution by the “second-order” mean-field theory will be presented towards the end of the paper in Sec. IV.5 . II.3 Annealed network model Figure 2: Schematic illustration of the annealed network and its approximation by a mean-connectivity network with additional noise. (a) In the annealed network model, Eq. ( 8 ), the random connectivity is re-sampled at each spike time resulting in an adjacency matrix a_{ijk} that varies at each spike k . (b) The annealed model can be mapped to a mean-connectivity network with additional noise, Eq. ( 12 ). The network is all-to-all connected with an average connectivity \langle a_{ij}\rangle . Additionally, each neuron is subject to an independent noise that captures the random connectivity. (c) Example trajectories of the input potentials in the mean-connectivity network with additional noise (gray). The distribution P of the input potentials follows a Gaussian distribution with a time-dependent mean \bar{h}(t) and a time-dependent variance \sigma^{2}(t) . The second-order mean-field model ( 20 ) aims to describe the stochastic dynamics of these two mean-field variables. The quenched connectivity causes heterogeneity of neural firing with spatial and temporal correlations which often make a direct mean-field treatment infeasible. Here, we follow a different approach based on a dynamical resampling of the connectivity (“annealing”) at each spike [ 67 , 29 ] (Fig. 2 a). This procedure largely ignores the part of the spatial and temporal correlations that is caused by the quenched connectivity, but leads to a tractable model of a fully connected neural network with additional dynamical noise that, as we will see, retains and captures the main statistical features of the recurrent synaptic input to a neuron (Fig. 2 b). We expect that the annealed approximation is better in dense networks of strongly correlated neurons. In this case, the temporal correlations in the spike input of a given neuron are partially retained upon randomization of the presynaptic neurons. In contrast, in sparse networks, the membrane potentials are independent among neurons, and therefore, the resampling of presynaptic neurons in time destroys the temporal correlations in the presynaptic spike train of a fixed presynaptic neuron. In the annealed approximation, the connections a_{ij} are re-sampled at each spike occurring in the network. Thus, the time-varying adjacency matrix can be written as [\hat{a}_{ijk}] , where the additional index k labels the k -th spike occurring in the network. Then, for a fixed k , the N\times N -adjacency matrix [\hat{a}_{ij}]_{k} is still a random matrix with fixed in-degree, and therefore has independent rows but dependent elements \hat{a}_{ijk} within a given row i . However, the dependence caused by the fixed-in-degree constraint is immaterial for the recurrent inputs in the annealed approximation. This is because only the j_{k} -th column of the matrix [\hat{a}_{ij}]_{k} , which has independent elements, matters for the transmission of the k -th spike, where j_{k} is the index of the neuron that fired the k -th spike, and because of the re-sampling of the adjacency matrix. The latter implies that for different spike times, the columns are independent. From the perspective of the recurrent inputs, we can thus equally well re-sample the elements of the adjacency matrix as independent Bernoulli variables a_{ijk}\sim\text{Ber}(p) with success probability p=C/N . Thus, our model, Eq. ( 1 ), changes in the annealed approximation to \displaystyle\tau dh_{i}(t) \displaystyle=[-h_{i}+\mu(t)]dt+{\color[rgb]{0,0,0}\frac{1}{N}}dY_{i}(t-d), (8) \displaystyle\frac{dY_{i}}{dt}(t) \displaystyle={\color[rgb]{0,0,0}\frac{w}{p}}\sum\limits_{j=1}^{N}\sum_{l}a_{ijk_{j,l}}\delta(t-t_{j,l}) (9) where the index k_{j,l} delivers the location of the l -th spike time t_{j,l} of neuron j within the sequence of all spikes in the network. The quantity Y_{i}(t) is the integrated synaptic input received by neuron i until time t . We refer to this model as the annealed network (Table 1 ) . We note that in simulations of the annealed network, it is not necessary to re-sample the entire adjacency matrix at each spike but only the column that corresponds to the neuron that fired the spike. In other words, whenever a neuron fires a spike, this spike is transmitted independently to postsynaptic neurons with probability p=C/N . By generalizing from binary weights w_{ij}=\frac{w}{p}a_{ij} to an arbitrary connectivity matrix w_{ij} , the annealed network can also be defined for such general connectivity through dynamic resampling, which produces time-dependent synaptic weights w_{ij}(t) . In the specific case of binary synapses, w_{ij}(t_{j,l})=\frac{w}{p}a_{ijk_{j,l}} as in Eq. ( 9 ). Apart from the random connectivity, another source of quenched disorder that can be included in our theory is heterogeneity in the external current \mu_{0} . For the sake of clarity, we will begin our analysis without heterogeneity in \mu_{0} and will defer the straightforward extension to heterogeneity to a brief section in the Appendix (Sec. A ). Table 1: Summary of microscopic network models. Network Equation Connectivity Quenched ( 1 ) random, fixed in time, \left\langle w_{ij}\right\rangle=w , \text{var}(w_{ij})=\sigma_{w}^{2} Mean-connectivity ( 5 ) mean-matched all-to-all, \left\langle w_{ij}\right\rangle=w , \text{var}(w_{ij})=0 Annealed ( 8 ), ( 9 ) random, dynamic (resampling in time), \left\langle w_{ij}(t)\right\rangle=w , \text{var}(w_{ij}(t))=\sigma_{w}^{2} III Derivation of the mesoscopic model III.1 A mean-connectivity network with additional noise that accounts for second-order connectivity statistics The first step of the derivation is a temporal coarse-graining of the annealed model leading to an effective, fully connected network with homogeneous weights (mean-connectivity network) and additional noise. The additional noise will be the crucial difference to the mean-connectivity network discussed in Fig. 1 . To coarse-grain, we integrate Eq. ( 8 ) over a small time step of length \Delta t : \tau\Delta h_{i}(t)=\int_{t}^{t+\Delta t}[-h_{i}(s)+\mu(s)]\,ds+{\color[rgb]{0,0,0}\frac{1}{N}}\Delta Y_{i}(t-d). (10) Here, \Delta h_{i}(t)=h_{i}(t+\Delta t)-h_{i}(t) is the increment of the input potential, \Delta Y_{i}(t)=\sum_{j=1}^{N}w_{ij}(t)\Delta Z_{j}(t-d) is the synaptic input received by neuron i during the time step \Delta t and \Delta Z_{j}(t)=1 if neuron j has a spike during that time step and \Delta Z_{j}(t)=0 otherwise. In the annealed approximation, the time-varying coupling strengths w_{ij}(t) are independent random variables (across time steps and synapses) with mean w and variance \sigma_{w}^{2} . In our specific example, the random variables are binary taking the value w/p with probability p and 0 otherwise. However, the derivation does not hinge upon binary connectivity but applies more generally to any distribution of coupling strengths with finite variance \sigma_{w}^{2}<\infty . The synaptic input \Delta Y_{i}(t) can be regarded as the result of a doubly stochastic process: On the one hand, given the total spike count \Delta Z(t)=\sum_{i=1}^{N}\Delta Z_{i}(t) , each \Delta Y_{i}(t) is the sum of \Delta Z(t) independent random variables. Hence, the conditional mean and covariance are \displaystyle\langle\Delta Y_{i}(t)|\Delta Z(t)\rangle={\color[rgb]{0,0,0}w}\Delta Z(t), \displaystyle\text{cov}(\Delta Y_{i}(t),\Delta Y_{j}(t)|\Delta Z(t))={\color[rgb]{0,0,0}\sigma_{w}^{2}}\Delta Z(t)\delta_{ij}, where \delta_{ij} denotes the Kronecker delta. If N is large such that p\Delta Z(t)\gg 1 , the synaptic input \Delta Y_{i}(t) for a given \Delta Z(t) is approximately Gaussian distributed. Under this condition, we can write \Delta Y_{i}(t)\approx{\color[rgb]{0,0,0}w}\Delta Z(t)+{\color[rgb]{0,0,0}\sigma_{w}}\sqrt{\Delta Z(t)}n_{i}(t), (11) where n_{i}(t)\sim\mathcal{N}(0,1) are independent standard normal random variables. The first term in Eq. ( 11 ) is proportional to the total number of spikes or the mesoscopic population activity and thus corresponds to the input in an effective mean-connectivity network. This term is common to all neurons. In contrast, the second term is independent across neurons and only arises when \sigma_{w}>0 . Specifically, it captures the individual differences of the synaptic inputs in a non-fully connected (“diluted”) network with p<1 . On the other hand, \Delta Z(t) is itself a Poisson random variable with mean Nr(t)\Delta t . From Eq. ( 11 ), or, alternatively, from the law of total expectation and total covariance, we find that the synaptic inputs \Delta Y_{i}(t) exhibit the following mean and covariance for a given stochastic population rate r(t) : \displaystyle\left\langle\Delta Y_{i}(t)\mid r(t)\right\rangle \displaystyle={\color[rgb]{0,0,0}wN}r(t)\Delta t \displaystyle\text{cov}(\Delta Y_{i}(t),\Delta Y_{j}(t)\mid r(t)) \displaystyle={\color[rgb]{0,0,0}\left(w^{2}+\sigma_{w}^{2}\delta_{ij}\right)N}r(t)\Delta t. The last equation shows that pairwise correlations of the synaptic input to different neurons are non-zero for finite N . These correlations arise from the common fluctuations of the first term in Eq. ( 11 ) representing the population activity. The covariance structure of the synaptic inputs \Delta Y_{i}(t) is preserved if their representation by Eq. ( 11 ) is replaced by \Delta\hat{Y}_{i}(t)={\color[rgb]{0,0,0}w}\Delta Z(t)+{\color[rgb]{0,0,0}\sigma_{w}}\sqrt{{\color[rgb]{0,0,0}N}r(t)\Delta t}n_{i}(t). This representation allows to take the temporal continuum limit. Substituting \Delta\hat{Y}_{i}(t-d) for \Delta Y_{i}(t-d) in Eq. ( 10 ), dividing by \Delta t and taking the limit \Delta t\to 0 , we obtain \tau\frac{dh_{i}}{dt}=-h_{i}+\mu(t)+\frac{w}{N}\sum_{j=1}^{N}\dot{Z}_{j}(t-d)+{\color[rgb]{0,0,0}\sigma_{w}\sqrt{\frac{r(t-d)}{N}}}\zeta_{i}(t). (12) Here, we have used that the population activity, Eq. ( 3 ), can be rewritten as A_{N}(t,\Delta t)=\frac{1}{N\Delta t}\int_{t}^{t+\Delta t}\sum_{i=1}^{N}\,dZ_{i}(t^{\prime}), (13) where the spike trains \dot{Z}_{i}(t) have stochastic intensities \phi(h_{i}(t^{-})) . Furthermore, \zeta_{i}(t) are independent Gaussian white noises obeying \langle\zeta_{i}(t)\zeta_{j}(t^{\prime})\rangle=\delta_{ij}\delta(t-t^{\prime}) . Equation ( 12 ) represents an effective, fully connected network with rescaled synaptic efficacy w/N=pJ and additional dynamic noise whose intensity is proportional to the variance \sigma_{w}^{2} of the coupling strengths w_{ij} . Therefore, the dynamics is the same as the mean-connectivity network, Eq. ( 5 ), but with an additional noise term that captures the fluctuations caused by the random connectivity. This representation also reveals the distinct effects of finite network size ( N<\infty ) and random dilution ( p<1 ) on the recurrent synaptic fluctuations. First, the stochastic input spikes from all neurons in the network contribute a common synaptic input current proportional to the population activity (third term on the right-hand side of Eq. ( 12 )), which is stochastic for finite N and thus exposes the coherent part of the synaptic fluctuations. More precisely, the population activity has conditional mean r(t) (given all input potentials h_{i}(t^{-}) , i=1,\dotsc,N ) and finite-size fluctuations of order N^{-1/2} because the spike trains \dot{Z}_{i}(t) are conditionally Poisson (given h_{i}(t^{-}) ) with conditional mean r(t) . In contrast, the last term of Eq. ( 12 ) contains a Gaussian white noise \zeta_{i}(t) that is independent for each neuron, and thus represents the incoherent part of the synaptic fluctuations. This incoherent noise captures the effect of the random connectivity ( \sigma_{w}^{2}>0 ), and, specifically, the random dilution ( p<1 ). It therefore vanishes in the limit of fully connected, homogeneous networks ( p\to 1 , \sigma_{w}^{2}\to 0 ), in which case the mean-connectivity network is recovered. In the following, we assume a large network size N such that a diffusion approximation of the shot noise can be made: noting that \Delta Z(t) is conditionally Poisson with mean and variance equal to Nr(t)\Delta t , the population activity can be approximated for large N as A_{N}(t;\Delta t)\approx\frac{\int_{t}^{t+\Delta t}A(t^{\prime})\,dt^{\prime}}{\Delta t},\quad A(t)=r(t)+\sqrt{\frac{r(t)}{N}}\eta(t), (14) where \eta(t) is a common Gaussian white noise with \langle\eta(t)\eta(t^{\prime})\rangle=\delta(t-t^{\prime}) . In this form, the finite-size fluctuations of order 1/\sqrt{N} become explicit. We note that the diffusion approximation of the population activity, Eq. ( 14 ), can in principle become negative. However, if N is sufficiently large such that Nr(t)\Delta t\gg 1 , the probability that such event happens is negligibly small. III.2 Second-order mesoscopic mean-field dynamics All-to-all uniformly connected networks with independent noise, such as the mean-connectivity network with additional noise derived in the previous section, have been well studied. In particular, in the limit N\to\infty , the population dynamics can be described by a Fokker-Planck equation [ 70 ] , while for finite N , the stationary statistics of the fluctuations has been calculated using linear response theory [ 71 ] . However, despite earlier theoretical approaches [ 29 , 66 ] , a derivation of stochastic population dynamics that captures both finite-size fluctuations and nonstationary dynamics remains challenging. In our setting, this derivation is facilitated by the absence of resets in the Poisson neuron model. To obtain a self-consistent description, we follow a standard two-step procedure: first, we consider the dynamics of the input potentials \tau\frac{\mathrm{d}h_{i}}{\mathrm{d}t}=-h_{i}+f(t)+g(t)\zeta_{i}(t), (15) driven by fixed, time-dependent input signals f(t) and g(t) , and derive the dynamics of the mean and variance conditioned on these inputs. We then derive the dynamics of the stochastic rate r(t) for fixed input signals f and g such that it depends on these input signals only through the conditional mean and variance and an additional colored noise. In the second step, in turn, the input signals will be expressed in terms of the stochastic rate r(t) according to Eqs. ( 12 ), ( 13 ) and ( 14 ): \displaystyle f(t) \displaystyle=\mu(t)+w\left[r(t-d)+\sqrt{\frac{r(t-d)}{N}}\eta(t-d)\right], (16) \displaystyle g(t) \displaystyle={\color[rgb]{0,0,0}\sigma_{w}\sqrt{\frac{r(t-d)}{N}}} This step will close the system for the conditional mean and variance and the colored noise because the stochastic rate depends itself only on these variables. In this way, we will obtain a self-consistent mean-field description for the annealed model, Eq. ( 12 ), as detailed in the following. In view of Eq. ( 16 ), we also note that the conditioning on the input signals f and g will allow for the interpretation of the mean-field variables at time t as conditional averages given the history of the noise \eta(t^{\prime}-d) , the stochastic rate r(t^{\prime}-d) and the external input \mu(t^{\prime}) for t^{\prime}0 . The imperfect averaging leads to finite-size fluctuations \xi(t) – a third effect caused by the combination of both finite network size and random connectivity. This effect vanishes when N\to\infty or \sigma_{w}^{2}=0 ( p=1 ) because the noise strength is proportional to \sqrt{G/N} and G vanishes for \sigma^{2}=0 . If we had neglected the dispersion of the input potentials due to the random connectivity, i.e. assuming \sigma^{2}\equiv 0 , we would have arrived at a first-order MF theory (MF1) with only one equation for the mean \bar{h}(t) : \displaystyle\tau\frac{\mathrm{d}}{\mathrm{d}t}\bar{h} \displaystyle=-\bar{h}+\mu(t)+w\left[r(t-d)+\sqrt{\frac{r(t-d)}{N}}\eta(t)\right] \displaystyle r(t) \displaystyle=\phi\bigl(\bar{h}(t)\bigr). (21) Here, we have used that, for \sigma=0 , the population transfer function F(\bar{h},\sigma^{2}) simplifies to the transfer function of a single Poisson neuron \phi(\bar{h})=F(\bar{h}(t),0) and that G(\bar{h},0)=0 . Theoretical predictions of MF1, Eq. ( 21 ), can always be obtained from the corresponding result of MF2 by taking the limit p\to 1 while keeping the network size N=C/p and the coupling strength w=JpN constant. While the MF1 dynamics, Eq. ( 21 ), still captures finite-size effects through the noise term, it does not describe the effect of randomness in the connectivity \sigma_{w}^{2}>0 when p<1 . In fact, MF1 precisely corresponds to the mean-connectivity approximation that underlies the assumption of full connectivity made in previous finite-size theories [ 42 , 43 , 47 , 36 , 44 , 45 , 46 , 33 ] . This can be seen by applying the diffusion approximation of the Poissonian shot noise for large N , Eq. ( 14 ), to the exact mesoscopic dynamics, Eq. ( 7 ), of the mean-connectivity network, Eq. ( 5 ), which recovers MF1. Thus, by comparing simulations of the microscopic models (quenched and annealed networks) with MF1 and MF2, we will thus be able to judge the mean-connectivity approximation used in previous studies [ 47 , 36 ] and the correction for the effect of random connectivity ( p<1 ) given by the 2nd-order MF theory. In the limit N\to\infty , the MF2 theory , Eq. ( 20 ), converges to either a one- or a two-dimensional, deterministic dynamics depending on whether the connectivity is dense or sparse. In the dense limit, where p is kept constant and, hence, C=pN\to\infty , the variance \sigma^{2} as well as the fluctuations N^{-1/2}\eta and N^{-1/2}\xi vanish, leaving a one-dimensional deterministic firing-rate model. This case is equivalent to the large- N limit of the 1st-order MF theory and can thus neither capture finite-size nor p<1 effects . In contrast, in the sparse limit, where C is kept constant and, hence, p=C/N\to 0 , the noise terms N^{-1/2}\eta and N^{-1/2}\xi vanish but the variance \sigma^{2} does not. The result is a two-dimensional, deterministic mean-field dynamics for the mean and variance of the input potentials, \displaystyle\begin{aligned} \tau\frac{\mathrm{d}}{\mathrm{d}t}\bar{h}&=-\bar{h}+\mu(t)+wr(t-d)\\ \tau\frac{\mathrm{d}}{\mathrm{d}t}\sigma^{2}&=-2\sigma^{2}+\frac{w^{2}}{\tau C}r(t-d),\end{aligned} (22) where r(t)=F(\bar{h}(t),\sigma^{2}(t)) . This sparse connectivity limit is similar to the classical model of Amari [ 22 ] . While it captures the effect of random connectivity, it cannot describe finite-size effects and hence second-order statistics caused by finite-size fluctuations. In the following, we will refer to the dynamics in the sparse limit, Eq. ( 22 ), in short as MFsparselim . The effects described by the three population dynamics MF1, MF2 and MFsparselim for different parameter regimes are summarized in Table 3 (Appendix, Sec. J ). III.2.1 Functions F and G for an error-function nonlinearity For the error function nonlinearity, Eq. ( 2 ), the functions F and G can be calculated explicitly as (see appendix, Sec. H ) \displaystyle F(h,\sigma^{2})=r_{\text{m}}\Phi\left(\beta_{\text{eff}}(\sigma^{2})h\right), (23) \displaystyle G(h,\sigma^{2})=r_{\text{m}}^{2}\Phi\left(\beta_{\text{eff}}(\sigma^{2})h\right)-2r_{\text{m}}^{2}T\Big(\beta_{\text{eff}}(\sigma^{2})h,\beta_{\text{eff}}(2\sigma^{2})h\Big) (24) with T being the Owen T-function and \beta_{\text{eff}}(\sigma^{2})=\frac{\beta}{\sqrt{1+\beta^{2}\sigma^{2}}} being the effective steepness of the sigmoidal nonlinearity at the population level. Because \beta_{\text{eff}}(\sigma^{2})\leq\beta_{\text{eff}}(0)=\beta , the effective nonlinearity that governs the population dynamics has a reduced steepness compared to the the single-neuron nonlinearity (Fig. 3 a) and this reduction is caused by the strictly positive spread \sigma>0 of the membrane potentials in a non-fully connected random network. IV Analysis of the mesoscopic model The mesoscopic dynamics MF2 , Eq. ( 20 ), allows us to calculate and analyze the first- and second-order statistics of mesoscopic quantities such as the stochastic population rate r(t) , the population-averaged input potential \bar{h}(t) and the population activity A_{N}(t,\Delta t) . We will mainly focus on the second-order statistics, the analysis of which will be based on a linearization of the dynamics about the fixed points of the noiseless systems. Therefore, we will first calculate the fixed points and their stability. Although not the focus of the paper, this analysis will also yield approximations for first-order statistics including the stationary mean population activity and the linear response of the mean population activity to time-dependent external stimuli. IV.1 Stationary mean population activity We first calculate the fixed-points \bar{h}(t)=\bar{h}_{0} , \sigma^{2}(t)=\sigma_{0}^{2} and \xi(t)=0 of the deterministic system when the noise terms in Eq. ( 20 ) are set to zero and the external drive is constant, \mu(t)=\mu_{0} . In the following, we assume an inhibitory network ( w<0 ) balanced with a positive external drive \mu_{0}>0 . In this case, the system exhibits a unique fixed point as shown in the Appendix, Sec. I . If the fluctuations are sufficiently small and the fixed point is stable, the quantity r_{0}:=F(h_{0},\sigma_{0}^{2}) then provides an approximation of the mean stationary population activity (or equivalently, the mean stationary firing rate of neurons). At equilibrium, the input potential h_{0} is given by the fixed-point equation \frac{\bar{h}_{0}-\mu_{0}}{w}=F\left(\bar{h}_{0},\frac{w(1-p)}{2\tau pN}\left(\bar{h}_{0}-\mu_{0}\right)\right), (25) (see appendix, Sec. B ) . Furthermore, the fixed points of the MF1 are obtained by setting p=1 in Eq. ( 25 ) corresponding to setting \sigma_{0}^{2}=0 in the sigmoidal function F(h_{0},\sigma_{0}^{2}) . The solutions for both MF1 and MF2 can be determined graphically (Fig. 3 a) : For MF1, \sigma_{0}^{2}=0 and therefore the sigmoidal function F(h_{0},0) is steeper compared to MF2, for which \sigma_{0}^{2} is strictly positive. While the fixed-points can be obtained numerically, for the analytical theory we use the approximations \displaystyle r_{0} \displaystyle\approx-\frac{\mu_{0}}{w}, (26) \displaystyle\bar{h}_{0} \displaystyle\approx\Phi^{-1}\left(-\frac{\mu_{0}}{wr_{\text{m}}}\right)\sqrt{\beta^{-2}+\sigma_{0}^{2}} (27) \displaystyle\sigma_{0}^{2} \displaystyle\approx-\frac{w\mu_{0}(1-p)}{2\tau Np} (28) for stimuli 0<\mu_{0}<-wr_{\text{m}} , where the firing rates are low ( 00.023r_{\text{m}} . The fixed-point solutions yield theoretical predictions for the stationary first-order statistics such as the stationary mean firing rate \left\langle r\right\rangle , mean membrane potential \left\langle\bar{h}\right\rangle and mean membrane-potential spread \left\langle\sigma^{2}\right\rangle . However, we postpone the discussion of these predictions to the Appendix B , because our main focus is the second-order statistics. IV.2 Stability of the fixed-point solutions Figure 3: Analysis of the deterministic mean-field dynamics. (a) Graphical solution of the fixed point equation ( 25 ), here illustrated for \mu_{0}=5 mV and w=-0.7 mVs. Solid lines: function F(\bar{h},\sigma^{2}(\bar{h})) for MF1 (orange, \sigma^{2}(\bar{h})\equiv 0 ) and MF2 (blue, \sigma^{2}(\bar{h}) as in Eq. ( 25 )). Black dashed line: left-hand side of Eq. ( 25 ). Cyan dash-dotted line: F with the second argument fixed at the fixed-point value \sigma_{0}^{2} . (b) (d,w) -phase diagram with Hopf bifurcation lines (onset of oscillations). Bifurcation lines of MF2 and MFsparselim explain stable asynchronous activity and oscillatory instability. The heat map shows the standard deviation of the population rate r(t) in the quenched network ( p=0.1 ) to indicate the presence of oscillations. Hopf bifurcations, calculated from Eq. ( 30 ), are displayed for MF1 (orange line), MF2 (blue line) and MFsparselim (brown line). (c) Sample trajectories of population rate r(t) for quenched network (green), MF1 (orange), MF2 (blue) and MFsparselim (brown) with d=1 ms. (d)-(f) Linear response of the time-dependent mean population rate (“rate susceptibility”) to a weak modulation of the external current \mu(t) for quenched network (green circles), MF1 (orange), MF2 (blue) and MFsparselim (dashed brown). The MF theories are given by Eq. ( 34 ), where p is set to 1 for MF1, and the sparse limit N\to\infty , pN=C=\text{const} is taken for MFsparselim. For each panel, top and bottom plots show the amplitude |\tilde{\chi}_{r}| and the phase \phi_{r} of the susceptibility, respectively. Dashed horizontal line: \phi_{r}=-\pi/2 . (d) p=0.1 ( C=100 ), d=0 ; (e) p=0.4 ( C=400 ), d=1\, ms; (f) p=0.95 ( C=950 ), d=0 . Other parameters unless otherwise indicated: N=1000 , w=-1 mVs, \beta=5 mV -1 , \mu_{0}=10 mV, \sigma_{\text{ext}}=0 . In the presence of a non-zero transmission delay d>0 , the fixed-point solution of the mean-field model may become unstable, leading to an oscillatory regime, see e.g. [ 73 ] . For such an instability to correctly predict oscillatory behavior of our microscopic network model, our mean-field theory must accurately describe the recurrent fluctuations in the network. The onset of oscillations can be understood by a linear stability analysis. To this end, we linearize the noiseless system around the fixed point. Small deviations of the mesoscopic variables, \delta\bar{h}=\bar{h}(t)-\bar{h}_{0} , \delta\sigma^{2}=\sigma^{2}(t)-\sigma_{0}^{2} , obey the linearized dynamics \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\boldsymbol{X}(t)=\operatorname{T}\boldsymbol{X}(t)+\operatorname{W}\boldsymbol{X}(t-d), (29) where we introduced the deviation vector \boldsymbol{X}(t)=\left(\delta\bar{h}(t),\delta\sigma^{2}(t),\xi(t)\right)^{T} and the matrices \displaystyle\operatorname{T}=-\frac{\text{diag}(1,2,1)}{\tau},\quad\operatorname{W}=\left(\frac{w}{\tau},\frac{w^{2}\left(1-p\right)}{\tau^{2}pN},0\right)^{T}\boldsymbol{L}^{T} with \boldsymbol{L}^{T}=(F_{h},F_{\sigma},N^{-1/2}) . Here, F_{h} and F_{\sigma} denote the partial derivatives of F(h,\sigma^{2}) with respect to h and \sigma^{2} at the fixed point (exact and approximate expressions for F_{h} and F_{\sigma} are given in the Appendix D ). We note that any perturbation in \xi exponentially decays to zero with a time constant \tau . It is therefore sufficient to look for solutions of the form \boldsymbol{X}(t)=e^{\lambda t}\left(\hat{h}(\lambda),\hat{\sigma}(\lambda),0\right)^{T} with some constant parameter \lambda . The sign of the real part of \lambda then determines the stability of the fixed point; in particular, if \text{Re}(\lambda)>0 the fixed-point is unstable. As shown in Appendix E , the parameter \lambda must satisfy the characteristic equation \displaystyle\lambda\tau \displaystyle=-1+w{F}_{h}e^{-\lambda d}+\frac{w{F}_{h}\hat{F}_{\sigma}e^{-2\lambda d}}{2+\lambda\tau-\hat{F}_{\sigma}e^{-\lambda d}}, (30) where \hat{F}_{\sigma}:=w^{2}\frac{1-p}{\tau C}F_{\sigma} . To find the onset of oscillations, we look for a Hopf bifurcation, at which \lambda=i\omega with a frequency \omega\in\mathbb{R}\setminus\{0\} . Imposing such purely imaginary values of \lambda in Eq. ( 30 ) allows us to find the “Hopf boundary” of the oscillatory phase in the parameter space. In the following, we investigate the stability depending on the transmission delay d and the coupling strength w (Fig. 3 b) . In the (d,w) -space, we can see that the bifurcation lines of the noiseless systems significantly differ between 2nd-order and first-order MF dynamics. For sufficiently strong coupling strength, the 1st-order MF dynamics exhibits a Hopf boundary at small values of d , predicting oscillations already at small delays. In contrast, the 2nd-order theory predicts an oscillation onset for much larger delays. Interestingly, MF2 predicts that a stronger coupling strength w requires a larger delay to enable oscillations, whereas the first-order model requires a smaller delay. We note that the bifurcation line obtained for MFsparselim is close to MF2. The bifurcation lines qualitatively explain the presence or absence of large noisy oscillations in the mean-connectivity and random network models (Fig. 3 b,c). Furthermore, the simulations of the MF dynamics indicate that the Hopf bifurcation is supercritical: varying the delay, the variance of the population rate r changes continuously at the bifurcation and no bistability between fixed point and limit cycle is observed. In conclusion, our stability analysis of MF2 shows that connectivity disorder leads to a stabilization of the non-oscillatory stationary state. In particular, the disordered connectivity leads to incoherent fluctuations of the recurrent input, which flattens the effective nonlinearity of the macroscopic dynamics and thus changes the stability properties of the fixed point. IV.3 Linear response of the mean activity We now analyze how the network responds to a small, time-dependent perturbation of the external current, \mu(t)=\mu_{0}+\mu_{1}(t) by calculating the linear response function [ 74 ] . To this end, we linearize our mesoscopic mean-field dynamics , Eq. ( 20 ), around the fixed point and obtain a generalization of Eq. ( 29 ): \displaystyle\frac{\mathrm{d}}{\mathrm{d}t}\boldsymbol{X}(t)=\operatorname{T}\boldsymbol{X}(t)+\operatorname{W}\boldsymbol{X}(t-d)+\boldsymbol{M}(t)+\operatorname{B}\boldsymbol{\zeta}(t). (31) Here, the vector \boldsymbol{X}(t) represents again the deviation from the fixed point as in the previous section and the three-dimensional vector \boldsymbol{M}(t) represents a general small perturbation to the system. Furthermore, the last term represents the finite-size noise, where \displaystyle\operatorname{B}=\begin{pmatrix}\frac{w}{\tau}\sqrt{\frac{r_{0}}{N}}&0\\ 0&0\\ 0&\sqrt{\frac{2G_{0}}{\tau}}\end{pmatrix},\quad\boldsymbol{\zeta}(t)=\begin{pmatrix}\eta(t)\\ \zeta(t)\end{pmatrix} and G_{0}=G(\bar{h}_{0},\sigma_{0}^{2}) . In the linearized dynamics we neglected terms of order O(1/N) and higher. In the Fourier domain, the solution reads \tilde{\boldsymbol{X}}(\omega)=\tilde{\operatorname{\chi}}(\omega)\left(\tilde{\boldsymbol{M}}(\omega)+\operatorname{B}\tilde{\boldsymbol{\zeta}}(\omega)\right), (32) where we introduced the susceptibility matrix \tilde{\operatorname{\chi}}(\omega)=\left[i\omega\operatorname{I}_{3}-\operatorname{T}-\operatorname{W}e^{-i\omega d}\right]^{-1}. (33) Here, \operatorname{I}_{3} denotes the 3\times 3 identity matrix and the tilde notation \tilde{f}(\omega) for a function f(t) denotes its Fourier transform, i.e. \tilde{f}(\omega)=\int_{-\infty}^{\infty}f(t)e^{-i\omega t}\,dt . The explicit expressions for the elements of \tilde{\operatorname{\chi}} are given in the appendix, Eq. ( 63 ). The susceptibility matrix provides the response of the ensemble mean \left\langle\boldsymbol{X}(t)\right\rangle of the deviation from the fixed point of the full nonlinear system to an infinitesimally small perturbation \boldsymbol{M}(t) in the Fourier domain via the relation \left\langle\tilde{\boldsymbol{X}}(\omega)\right\rangle=\tilde{\operatorname{\chi}}(\omega)\tilde{\boldsymbol{M}}(\omega) . Similarly, the response of the ensemble-averaged firing rate \langle r(t)\rangle=F(h_{0},\sigma_{0}^{2})+\delta r(t) is given in the Fourier domain by \tilde{\delta r}(\omega)=\boldsymbol{L}^{T}\tilde{\operatorname{\chi}}(\omega)\tilde{\boldsymbol{M}}(\omega) . For an external stimulus \mu(t)=\mu_{0}+\mu_{1}(t) , the perturbation is \tilde{\boldsymbol{M}}(\omega)=(\tilde{\mu}_{1}(\omega)/\tau,0,0)^{T} . In this case, we then have \tilde{\delta r}(\omega)=\tilde{\chi}_{r}(\omega)\tilde{\mu}_{1}(\omega) , where we introduced the rate susceptibility \tilde{\chi}_{r}(\omega)=\frac{1}{\tau}\left[F_{h}\tilde{\operatorname{\chi}}_{11}(\omega)+F_{\sigma}\tilde{\operatorname{\chi}}_{21}(\omega)\right]. (34) Here, we used that \tilde{\chi}_{31}\equiv 0 . The rate susceptibility is in general complex-valued with absolute value |\tilde{\chi}_{r}(\omega)| and phase \phi_{r}(\omega) . A straightforward interpretation is that the system driven by a sinusoidal external input modulation \mu_{1}(t)=\epsilon\sin(\omega t) with small amplitude \epsilon will respond with a sinusoidal rate modulation with amplitude \epsilon|\tilde{\chi}_{r}(\omega)| and a phase shift \phi_{r}(\omega) . For all models studied here, the amplitude decreases for high frequencies with a power law with exponent -1 (Fig. 3 d-f) . The MF2 theory agrees very well with the quenched microscopic network , in contrast to the MF1 theory which massively overestimates the amplitude at high frequencies. The quenched network shows a slightly smaller amplitude than predicted by the 2nd-order MF theory , but is still reasonably well approximated by the latter. While the amplitude vanishes at low frequencies as expected, the phase shift approaches -\frac{\pi}{2} in the limit of large frequencies (Fig. 3 d-f) . We can understand this behavior analytically by considering the limits of Eq. ( 34 ). For \omega=0 we obtain \tilde{\chi}_{r}(0)=\frac{F_{h}}{1-wF_{h}-\frac{w^{2}}{2\tau}F_{\sigma}\left(\frac{1}{C}-\frac{1}{N}\right)}, (35) which is real-valued and has therefore a phase zero. The amplitude in the low-frequency limit is approximately the negative inverse of the coupling strength -w^{-1} , under the condition that 1\ll wF_{h} and C large. Similarly, for large frequencies we find asymptotically \tilde{\chi}_{r}(\omega)\sim-i\frac{F_{h}}{\tau}\cdot\frac{1}{\omega},\qquad\omega\to\infty. (36) In this limit, the phase converges to -\frac{\pi}{2} and the amplitude follows a power law \omega^{-1} scaled by F_{h}/\tau . The different values for the derivative F_{h} explain the difference between the MF1 and the MF2 theory . To understand this difference analytically, we use the following approximation for the slope of the transfer function at the fixed point (see Appendix D ): F_{h}\approx\frac{r_{\text{m}}Q\left(-\frac{\mu_{0}}{wr_{\text{m}}}\right)}{\sqrt{2\pi\left(\beta^{-2}+\sigma_{0}^{2}\right)}}, (37) where \sigma_{0}^{2} is given by Eq. ( 28 ). Furthermore, we introduced the non-dimensional function Q(x):=\exp\left[-\frac{1}{2}\Phi^{-1}\left(x\right)^{2}\right] (38) defined on the interval [0,1] . This function has an inverted-U shape and is symmetric with respect to x=\frac{1}{2} (a graph of Q is shown in Fig. 6 c, dotted lines). Equation ( 37 ) clearly shows that the factor F_{h} increases with the connection probability p because \sigma_{0}^{2} decreases with p , cf. Eq. ( 28 ). The MF1 theory corresponding to the mean-connectivity network (without additional noise) is obtained by setting p=1 in Eq. ( 28 ). In this case, the dispersion \sigma_{0}^{2} vanishes, and hence the slope F_{h} is maximized. Therefore, the MF1 and MF2 theories predict the same power-law behavior of the rate susceptibility with \omega at large frequencies , but with a higher magnitude for the 1st-order model (Fig. 3 d) . With increasing connection probability p the susceptibility approaches the 1st-order MF theory. The low-frequency limit is in both mesoscopic models almost independent of p as the low-frequency limit is roughly given by -w^{-1} . Simulations of the annealed network network (not shown in the figure) match the predictions of the MF2 theory while the quenched network model has slightly lower amplitude and slightly lower phase for intermediate frequencies. Overall, the MF2 theory yields a strongly improved prediction compared to the MF1 theory . The synaptic delay d appears in \tilde{\chi}_{r} only via the complex exponential in Eq. ( 33 ). On the other hand, the delay does not influence the low-frequency limit and the high-frequency asymptotics, Eqs. ( 35 ) and ( 36 ) of the response, respectively. For intermediate frequencies the complex exponential causes a modulation and a peak in the amplitude as a function of \omega (Fig. 3 e) . Simulations of the annealed network model perfectly confirm (not shown in figure) the prediction of the 2nd-order MF theory , whereas the quenched network model has a less pronounced peak at a lower frequency. The susceptibility of MFsparselim fits well with the quenched network for small or medium connection probability, p=0.1 and p=0.4 (Fig. 3 d, e). For larger p , however, there is a noticeable discrepancy for MFsparselim, while MF1 and MF2 have a good match with the quenched network (at p=0.95 , see Fig. 3 f). In marked contrast to MF1 and MFsparselim, the predictions of MF2 are good over the entire range of connection probabilities p . IV.4 Power spectral density Figure 4: Power spectral densities of the population activity and population rate. a) Power spectrum of A_{N}(t;\Delta t) for quenched (green circles) and annealed network (red squares) and 1st- and 2nd-order theory (orange and blue lines, Eqs. ( 65 ) and ( 42 ), respectively). b) Same for the power spectrum of r(t) (theory: Eq. ( 41 )) Parameters: w=-1 mVs a) N=500 , p=0.07 , \mu(t)=\mu_{0}=10 mV, \sigma_{\text{ext}}=0 , \Delta t=0.1 ms . Now we turn to the second-order statistics of the stationary dynamics. We assume that the external stimulus \mu(t) is a stationary process with given mean \mu_{0} and power spectral density S_{\mu\mu}(\omega) . We begin with the power spectrum of the population activity A(t) and population rate r(t) . This statistics has been calculated analytically using linear-response theory for finite-size networks with uniform full connectivity [ 71 , 47 ] . In a similar fashion, we here derive the power spectrum via a linearization of our 2nd-order MF dynamics. To this end, we start with the power spectral density matrix \operatorname{S}(\omega) of the mean-field state vector \boldsymbol{X}(t) (cf. Eq. ( 29 )) defined via the relation \left\langle\tilde{\boldsymbol{X}}(\omega)\tilde{\boldsymbol{X}}^{*}(\omega^{\prime})\right\rangle=2\pi\operatorname{S}(\omega)\delta(\omega-\omega^{\prime}) (39) where ∗ denotes the conjugate transpose of a matrix. Using Eq. ( 32 ) for the linearized system, we find the following approximation for \operatorname{S}(\omega) describing weak fluctuations around the fixed points (see Appendix, Sec. G ): \operatorname{S}_{ij}(\omega)=\frac{1}{\tau^{2}}\left[S_{\mu\mu}(\omega)+\frac{w^{2}r_{0}}{N}\right]\tilde{\chi}_{i1}(\omega)\tilde{\chi}_{j1}^{*}(\omega)\\ +\frac{2G\left(\bar{h}_{0},\sigma_{h,0}^{2}\right)}{\tau}\tilde{\chi}_{i3}(\omega)\tilde{\chi}_{j3}^{*}(\omega). The matrix \operatorname{S} gives us the (cross-) spectral densities between \bar{h} , \sigma^{2} and \xi . For the power spectral densities of the population activity A(t) , we use the linear approximation \tilde{A}\approx\tilde{r}+\sqrt{\frac{r_{0}}{N}}\tilde{\eta},\quad\tilde{r}\approx F_{h}\tilde{h}+F_{\sigma}\widetilde{\sigma^{2}}+\frac{1}{\sqrt{N}}\tilde{\xi}. (40) which yields approximate expressions for the power spectrum of the stochastic population rate r(t) , S_{rr}(\omega)\approx F_{h}^{2}S_{11}(\omega)+F_{\sigma}^{2}S_{22}(\omega)+\frac{1}{N}S_{33}(\omega)+\\ 2F_{h}F_{\sigma}\mathfrak{Re}(S_{12}(\omega))+2\frac{F_{h}\mathfrak{Re}(S_{13}(\omega))+F_{\sigma}\mathfrak{Re}(S_{23}(\omega))}{\sqrt{N}}, (41) and of the population activity A(t) : S_{AA}(\omega)\approx S_{rr}(\omega)+\frac{r_{0}}{N}+2\sqrt{\frac{r_{0}}{N}}\mathfrak{Re}(S_{r\eta}(\omega)). (42) The cross spectral density S_{r\eta} that appears here can be calculated using Eq. ( 40 ) and ( 32 ): S_{r\eta}(\omega)\approx\frac{w}{\tau}\sqrt{\frac{r_{0}}{N}}\left[F_{h}\tilde{\chi}_{11}(\omega)+F_{\sigma}\tilde{\chi}_{21}(\omega)\right]. In Fig. 4 , we compare the theoretical predictions, Eqs. ( 41 ) and ( 42 ), to the corresponding power spectra obtained from simulations of the quenched and annealed network. In all cases, the power spectra of the population activity display a pronounced trough at low frequencies (Fig. 4 a) . The MF1 theory severely underestimates the observed low-frequency power, whereas the MF2 theory matches well the microscopic model under the annealed approximation. Nonetheless, even with this marked improvement over MF1, the annealed network’s low‐frequency power remains lower than that of the quenched network, exposing a limitation of the annealed approximation. The discrepancy of the power spectra of quenched and annealed networks is caused by the partial elimination of the effect of the single-neuron auto-correlations on the recurrent input when applying the annealing (i.e. the permanent resampling of connectivity). We note that the elimination of temporal correlations only concerns the incoherent part of the recurrent input, as expressed by the white noise in Eq. ( 12 ). In contrast, for finite N , the coherent part of the recurrent input fluctuations (finite-size fluctuations of population activity) still contains effects of spike-train auto-correlations, even in the annealed network. Finally, we mention that we have not included MFsparselim in Fig. 4 because its stationary dynamics is characterized by a deterministic equilibrium without any fluctuations and a vanishing power spectrum. IV.5 Variances Figure 5: Stationary variance of the population rate and mean input potential for varying connection probability p . (a) The number of neurons is fixed to N=1000 (hence C=1000p ). (b) The in-degree is fixed to C=100 (hence N=100/p ). Left panels of (a) and (b): Variance of the population rate r(t) (Symbols: simulations of microscopic networks as indicated in the legend; orange and blue solid lines: full theory Eq. ( 43 ) and Eqs. ( 68 ); black dashed line: analytical approximation for the variances based on the 2nd-order MF theory, Eq. ( 46 ), ( 47 ) and Eqs. ( 45 ), ( 49 ) for \text{var}(r) and \text{var}(\bar{h}) , respectively; dash-dotted line: MFsparselim. ) Right panels of (a) and (b): Same as left panels but for variance of \bar{h} . Parameters: \mu_{0}=10 mV, w=-1 mVs, \sigma_{\text{ext}}=0 mV, \beta=5 mV -1 The stationary variances of r(t) and \bar{h}(t) can be calculated by standard methods for the linearized system in the absence of synaptic delay, d=0 (see Appendix, Sec. G ). Furthermore, we assume that the external stimulus \mu(t) has mean \mu_{0} and white Gaussian fluctuations of strength \sigma_{\text{ext}}^{2} as in Sec. II.2 . Under these assumptions, one eventually obtains \text{var}(r)\approx F_{h}^{2}\sigma_{11}+F_{\sigma}^{2}\sigma_{22}+2F_{h}F_{\sigma}\sigma_{12}\\ +\frac{2F_{h}}{\sqrt{N}}\sigma_{13}+\frac{2F_{\sigma}}{\sqrt{N}}\sigma_{23}+\frac{1}{N}G(\bar{h}_{0},\sigma^{2}_{0}). (43) where \sigma_{ij}=\left\langle X_{i}(t)X_{j}(t)\right\rangle are the elements of the covariance matrix of the state vector \boldsymbol{X}(t) . In particular, \sigma_{11} yields the variance of the population-averaged input potential \bar{h}(t) . Approximate analytical expressions for \sigma_{ij} can be calculated using the linearized system, Eq. ( 31 ), and are given in the appendix, Sec. G . Our theory, Eq. ( 43 ), agrees well with microscopic simulations (Fig. 5 and 6 ), discussed in more detail below. In general, we find a good quantitative match between the theory and simulations of the annealed network. While, in the case of quenched random connectivity, the theory again exhibits quantitative deviations, our 2nd-order MF theory still captures the parameter dependence of the variance qualitatively and yields a much better quantitative prediction than the 1st-order MF theory. Again, MFsparselim without any noise in the external input deterministically converges into an equilibrium with zero variance. IV.5.1 Dependence on the connection probability Which connection probability causes the largest variability in our network model? To address this question, we analyze the variances of the population rate r and the average input potential \bar{h} as a function of p (Fig. 5 ). To avoid a trivial dependence through the mean firing rate, we vary p such that the mean firing rates remain roughly unchanged. According to Eq. ( 26 ), a constant rate can be achieved by keeping the coupling strength w=NpJ constant. In an experimental setting, where the synaptic efficacy J is the basic physiological parameter, this constraint could be realized in two different scenarios, either by fixing the network size N and changing the synaptic efficacy J such that Jp=\text{const.} , or by fixing both the number of input connections C=Np and the synaptic efficacy J . In the first scenario with constant N , we find that the variance of the rate r(t) increases with connection probability in microscopic network simulations (Fig. 5 a, left), although the variance of the input potential \bar{h} decreases (Fig. 5 a, right). MF1 cannot explain this behavior because the model equation ( 21 ) does not depend on the parameter p when w is fixed and N is constant. Thus, for all p\in[0,1] , the variance predicted by the 1st-order MF theory corresponds to the fully connected case p=1 , i.e. to the mean-connectivity network. In contrast, the dependence on p of the variances measured from simulations of the annealed network are quantitatively well reproduced by the 2nd-order MF theory, Eq. ( 43 ). The 2nd-order theory also captures the variances of the quenched network qualitatively. The quantitative agreement is also reasonably good, however, we find that the rate variance is overestimated and the variance of the population-averaged input potentials is underestimated by the annealed network. Nevertheless, the prediction of variances by the 2nd-order theory yields a significant improvement over the 1st-order MF theory. In order to gain an analytical understanding, we use the following explicit approximations of the variances (see Appendix, Sec. G ): \text{var}(\bar{h})\equiv\sigma_{11}\approx\frac{\frac{-w\mu_{0}}{2\tau N}+\frac{\sigma_{\text{ext}}^{2}}{2}}{(1-wF_{h})},\quad\text{var}(r)\approx F_{h}^{2}\text{var}(\bar{h}). (44) In the last approximation, we have used Eq. ( 43 ) and the fact that, for the parameters considered in this study, the variance of r is vastly dominated by the first term in that equation. By contrast, the other terms in Eq. ( 43 ) are small enough to be ignored. Equation ( 44 ) reveals that an increasing slope F_{h} decreases the variance of \bar{h} in an inhibitory network ( w<0 ). Intuitively, this effect can be understood by the increase of the effective leakage (1-wF_{h})/\tau in the linearized dynamics of \bar{h} which dampens the input fluctuations. In contrast, the variance of the population rate can increase with increasing slope because the quadratic prefactor F_{h}^{2} can overcompensate the decrease of \text{var}(\bar{h}) . To make this argument more clear in the theoretical discussion, we further assume that -wF_{h}\gg 1 (which holds for our parameter setting). Under this assumption, the variance of the population-averaged input potentials becomes inversely proportional to F_{h} , \text{var}(\bar{h})\approx\left(\frac{\mu_{0}}{2\tau N}+\frac{\sigma_{\text{ext}}^{2}}{-2w}\right)\frac{1}{F_{h}}. (45) In turn, the quadratic factor F_{h}^{2} due to the conversion from variance of \bar{h} to variance of r leads effectively to a linear dependence of \text{var}(r) on F_{h} , \text{var}(r)\approx\left(\frac{\mu_{0}}{2\tau N}+\frac{\sigma_{\text{ext}}^{2}}{-2w}\right)F_{h}. (46) The variance formulas ( 45 ) and ( 46 ) share a common factor that represents the sum of two contributions to the variability. The first term of the sum corresponds to the intrinsically generated variability caused by finite-size fluctuations of order \sqrt{r_{0}/N} in the mesoscopic MF dynamics , Eq. ( 20 ). We recall that the dependence of these fluctuations on the firing rate r_{0}=-\mu_{0}/w , arises from the Poisson spiking noise at the microscopic level. Thus, the Poisson property that the variance scales proportionally to the mean causes finite-size fluctuations that increase with the mean stimulus strength \mu_{0} (see below). We emphasize that this contribution to the variability is a clear finite-size effect that vanishes in the limit N\to\infty . In contrast, the second term of the sum in Eqs. ( 45 ) and ( 46 ) represents externally generated variability proportional to the variance of the common external stimulus. Figure 6: Variance of the stochastic population rate r(t) for varying mean external stimulus \mu_{0} . (a) Moderate-size network of N=10^{3} neurons whose inputs exhibit internally generated, finite-size noise but no external noise, \sigma_{\text{ext}}=0 . Different in-degree C are encoded by color . Left: Response to a step stimulus, where \mu_{0} jumps from 28 mV to 50 mV at time t=0.04 s. Blue/orange line: simulations of MF2 for C=100 ( p=0.1 ) and C=1000 ( p=1 ), the latter ( p=1 ) equivalent to MF1 or the mean-connectivity approximation of the former ( p=0.1 ). Black dotted: corresponding theory for the stationary variances, Eq. ( 43 ). Gray dashed: stimulus onset. Inset: magnified data for C=100 . Symbols represent stationary variances of network simulations and theory, see description of right panel. Right: Stationary variance of r(t) vs. mean stimulus strength \mu_{0} . Symbols: simulations for C=100 of quenched network (green circles), MF1 (orange triangles), and MF2 (blue diamonds). Solid lines: full MF1 (orange) and MF2 theory (blue). Black dashed: corresponding analytical approximations, Eq. ( 47 ); dash-dotted: MFsparselim . The gray dashed line marks the border of the region of interest \mu_{0}\leq-wr_{\text{m}}/2 where the hazard function at the fixed point, \phi(\bar{h}_{0}) is convex. (b) Analogous to (a) for a much larger network ( N=5\cdot 10^{4} ) with negligible internally generated finite-size noise but with additional external noise \sigma_{\text{ext}}^{2}=1 mV 2 . The connection probability is varied by different values of C (encoded by color) . Orange line: common MF1 theory or mean-connectivity network. Dash-dotted lines: MFsparselim. (c) Slope \hat{F}_{h} vs. mean stimulus \hat{\mu}_{0} . Dotted: p=1 corresponding to the function Q , solid line: p=0.7 . Gray dashed as in (a),(b) right. (d) Same for the product \hat{\mu}_{0}\hat{F}_{h} . (c), (d): N=1000 , \hat{\beta}=500 (equivalent to \beta=5 mV -1 in (a), (b)). For constant N , the p -dependence in both formulas, Eq. ( 45 ) and ( 46 ), is solely contained in the slope F_{h} of the transfer function. As discussed in Eq. ( 37 ) above, the slope F_{h} is a monotonously increasing function of p . The slope dependence thus explains the increase of the variance of the population rate (Fig. 5 a left) and the decrease of the variance of the mean input potential (Fig. 5 a right), respectively. In the second scenario with constant C and varying population size N=C/p , we observe different dependencies of the variances on p compared to the first scenario with constant N (Fig. 5 b). In simulations of microscopic networks and 2nd-order MF dynamics , the rate variance increases supralinearly with p and the variance of the mean input potential \bar{h} exhibits a non-monotonic behavior with a maximum inside the range 00 ). In this case, the rate variability is purely externally generated. In both cases, we find that the first-order MF theory largely overestimates the variance of the population rate in the quenched and annealed network, whereas the 2nd-order MF theory, Eq. ( 43 ), correctly predicts the magnitude of the rate variance (Fig. 6 a,b). Because the 1st-order MF dynamics is the exact MF dynamics for the mean-connectivity network, this large deviation explains our observation in Fig. 1 . Furthermore, in the two different scenarios, we find a different monotonicity of the rate variance if we restrict ourselves to the biologically interesting range where the fixed point is in the convex part of the sigmoidal hazard function, \bar{h}_{0}<0 and r_{0}0 until it reaches a maximum at \hat{\mu}_{0}^{*}=-\mu_{0}^{*}/(wr_{\text{m}})=0.5 , which corresponds to the upper bound r_{\text{m}}/2 of the biologically relevant dynamical range. For p<1 , the factor \hat{F}_{h} is the quotient of the symmetric function Q and an increasing function, which shifts the maximum of \hat{F}_{h} to a value <1/2 (Fig. 6 c, solid line). Hence, \hat{F}_{h} exhibits a non-monotonic behavior in the biologically relevant range 0<\mu_{0}<\mu_{0}^{*} . Besides the shift of the maximum to smaller values of the mean stimulus strength \hat{\mu}_{0} , the maximum also decreases when the connection probability is lowered from p=1 to smaller values. This decrease is a consequence of dividing the function Q(\hat{\mu}_{0}) by the square-root term that is strictly larger than one for p<1 . Importantly, \hat{F}_{h} can decrease by a large factor if the single-neuron transfer function is steep, i.e. \hat{\beta}\gg 1 , which explains the drastic decrease of the rate variance from 1st- to 2nd-order MF theory (Fig. 6 a,b, where \hat{\beta}=500 , corresponding to \beta=5 mV -1 as in Fig. 1 ). The behavior of \hat{F}_{h} just described explains the variance of the population rate for an infinitely large network with external noise (second scenario, Fig. 6 b). In fact, the prefactor of \hat{F}_{h} in Eq. ( 47 ) becomes independent of \hat{\mu}_{0} in the limit when N=C/p\to\infty , and hence the dependence on \hat{\mu}_{0} is fully captured by \hat{F}_{h} . In particular, the stark difference in magnitude of \text{var}(r) between the 1st- and 2nd-order MF prediction, as well as the qualitative response of the rate variability (suppression or amplification) to increasing stimuli, is explained by how the monotonicity and magnitude of the slope of the population transfer function \hat{F}_{h} varies with p . In contrast, in the first scenario when \sigma_{\text{ext}}=0 and N=C/p is finite, Eq. ( 47 ) tells us that the variance of the population rate is proportional to \hat{\mu}_{0}\hat{F}_{h}=\frac{\hat{\mu}_{0}}{\sqrt{1+\frac{\hat{\beta}^{2}(1-p)}{2\hat{\tau}C}\hat{\mu}_{0}}}Q(\hat{\mu}_{0}). The right-hand side is a product of a monotonically increasing function of \hat{\mu}_{0} and the function Q(\hat{\mu}_{0}) , which is maximized at \hat{\mu}_{0}^{*}=1/2 . Therefore, the maximum of \hat{\mu}_{0}\hat{F}_{h} is shifted to values \hat{\mu}_{0}>1/2 for all p\in[0,1] (Fig. 6 d). As a consequence, the variance of the population rate \text{var}(r) is a monotonically increasing function of the mean stimulus strength in the biologically relevant range 0\leq\hat{\mu}_{0}\leq 1/2 (Fig. 6 a right, \mu_{0}<50 mV), and thus an increase of the mean stimulus strength always increases the rate variability in the absence of external common noise (Fig. 6 a left). To keep the arguments simple, we studied here two clear-cut cases – one with and another without external noise. According to our theory, Eq. ( 46 ), the rate variance is a sum of two terms, where each term on its own corresponds to one of these cases. In general, the theory also applies to the mixed case, where the rate variance is the sum of externally and intrinsically induced variability. For example, in the simulations of Fig. 1 , we also compared two networks of size N=1000 and N=50000 but the external common noise was present in both cases. Nonetheless, the mesoscopic case of N=1000 with external noise can still be qualitatively understood by the limit case \sigma_{\text{ext}}=0 because the finite-size fluctuations dominate over the external noise. Lastly, we remark that as \hat{\mu}_{0}\to 0 , e.g. for strongly inhibition-dominated networks, the population rate variance decreases for both the mean-connectivity network (MF1) and MF2 in both cases as can be seen from Fig. 6 c,d. V Conclusions In this work, we set out to derive a stochastic population dynamics for finite-size networks of spiking neurons that correctly accounts for fluctuations at the mesoscopic scale when the connectivity is random – as in the case of non-full connectivity ( p<1 ) . While such regime is biologically highly relevant, to our knowledge it has not yet been systematically investigated within the frameworks of previously developed mean-field theories [ 29 , 30 ] . The low-dimensional, stochastic population model derived here allows one to study specifically how the combination of quenched disorder and stochastic neuronal firing affects the mesoscopic activity of finite-size neuronal populations. We thus hope to close an important gap in the use cases of simplified descriptions of networks and to provide the foundations for a mean-field modeling framework of neural variability at the mesoscopic population level. Based on a Poisson assumption for neural firing and a “dynamically annealed” description of the connectivity, we approximated the recurrent input to each neuron by its (population) mean, a coherent fluctuating part shared among all neurons, and an individually fluctuating contribution reflecting the connectivity disorder. Eventually, we obtained three coupled stochastic differential equations (SDEs) that describe the evolution of the mean input potential \bar{h} within the population, the input potential’s variance \sigma^{2} , as well as a colored noise \xi in the population rate that vanishes in both limits N\to\infty and p\to 1 . Our theory thus goes beyond previous, deterministic mean-field dynamics for first- and second-order cumulants [ 22 ] or pseudo-cumulants [ 53 ] that similarly captured the quenched variability of synaptic weights but assumed the macroscopic limit N\to\infty . The analysis of the new “second-order” mean-field (MF2) dynamics showed that the quenched connectivity disorder for p<1 has a drastic effect on the neural variability, the response to time-dependent stimuli, and the stability of the network in biologically relevant regimes. Specifically, our main findings with regard to the effects of a finite connectivity 00 ) caused by the quenched random connectivity with finite C=pN leads to an effective broadening of the neuron transfer function, but the stationary mean firing rate remains rather unaffected (Figs. 3 a, 8 b) . (ii) In the presence of synaptic delays, the stability properties of fixed points of the dynamics and locations of bifurcations can change considerably. In particular, we showed that the connectivity disorder can stabilize networks in regimes where a “first-order” mean-field (MF1) theory corresponding to a mean-connectivity approximation would predict oscillatory dynamics (Fig. 3 b,c). (iii) The population-rate response to high-frequency stimuli and (iv) the variance of the population rate are significantly lower than predicted by the MF1 theory (Fig. 3 d,e and 5 ). Moreover, (v) in the presence of shared noise due to common external input, an increase in the mean external input can actually lead to a decrease of the variance of the population rate (Figs. 1 and 6 ) – an effect which again is not captured in the MF1 theory. Overall, we provided simple analytical explanations for the effects of finite connectivity and finite network size in terms of the slope of the population transfer function and the multiplicative character of the finite-size noise. In particular, we found that a decreasing connection probability p lowers the slope F_{h} through an increase of the membrane potential variance \sigma^{2} and that, at finite network size N , the neural variability has a rate-dependent contribution that is proportional to the product of mean stimulus strength (or firing rate) and the slope of the transfer function (Fig. 6 d, Eq. ( 46 )). Of note, our mesoscopic approach differs from previous theoretical studies that focussed on how quenched random connectivity shapes microscopic dynamics [ 48 , 75 , 72 , 76 ] . Neural networks with quenched disorder can display a wide repertoire of dynamical states even under homogeneous random connectivity [ 76 ] , as considered here. In particular, inhibition-dominated networks operating near the edge of chaos produce rich single-neuron dynamics that may support complex computations [ 77 , 76 ] . Importantly, this critical dynamical state refers to the microscopic dynamics, as characterized by broad distributions of pairwise correlations [ 77 , 76 ] and neuronal time scales [ 78 ] . The annealed approximation underlying our theory cannot reproduce these microscopic features. However, this limitation is not problematic for our mesoscopic description of the coarse-grained population activity. As already noted in [ 76 ] , the mesoscopic population activity is largely unaffected by the critical dynamics at the microscopic scale. We have furthermore shown here that, at the mesoscopic scale, the annealed approximation captures surprisingly well both the dynamics and their fluctuations across the full range of connection probabilities. To derive a simple mean-field model with fluctuations that highlights the effects of finite connectivity and finite size, we made several simplifications. In particular, we used (i) an annealed approximation of the quenched random connectivity, (ii) only one cell type (single population), and (iii) a first-order Poisson model of the spiking-neuron dynamics. In the following, we will briefly discuss these simplifications and possible extensions. In our “dynamically annealed” approximation, we assumed that each presynaptic spike would be randomly distributed among all neurons according to the connection probability p , independent of the actual, fixed (quenched) synaptic connectivity matrix of a given network. A systematic comparison of our theory with simulations of quenched networks showed that our MF2 theory captures reliably the behavior of quenched networks with respect to the main findings described above. Deviations of the annealed network (from the quenched network) and, correspondingly, the MF2 dynamics, are noticeable for several statistics, though. However, these deviations were considerably smaller than the respective deviations of the mean-connectivity network and the corresponding MF1 dynamics used in previous studies [ 47 , 36 , 79 ] . The remaining deviations of the annealed network can most likely be attributed to the neglect of temporal correlations in the recurrent input caused by the annealing, i.e. the incessant resampling of the connections in time. This explanation would be consistent with the underestimation of the low-frequency power by the annealed network which we observed in Fig. 4 . For infinitely large, sparsely connected networks, a proper account of temporal correlations can, in principle, be obtained through a self-consistent treatment of the auto-correlations of the recurrent fluctuations [ 75 , 62 ] , which typically adds low-frequency power [ 62 ] . Such self-consistent theory would lead to colored noise rather than white noise for the incoherent fluctuations in Eq. ( 12 ). However, how to build a stochastic, dynamical population model for finite-size, random networks that accounts for temporal correlations in a self-consistent manner is an open theoretical problem which goes beyond the scope of the present study. Figure 7: Variance of r and \bar{h} for a network of LIF neurons with a rectified linear hazard function \phi(h)=\beta_{\text{lin}}[h-\theta_{\text{lin}}]_{+} and a post-spike reset to h_{\text{reset}} as a function of connection probability p . The LIF network behaves qualitatively similar to the Poisson network without reset in this paper (cf. Fig. 5 ). Variances are calculated from network simulations of quenched (green) and annealed (red) networks for (a) N=1000 and (b) C=100 for different connection probabilities p . The rectified linear hazard rate was matched both by value and slope of the single neuron transfer function Eq. ( 2 ) at a reference point h_{\text{ref}}=-0.25 mV. Parameters: h_{\text{reset}}=-15 mV, \beta_{\text{lin}}\approx 91.32 mV -1 , \theta_{\text{lin}}\approx-0.37 mV, w=-1 mVs, \mu_{0}=10 mV . Interestingly, perhaps not surprisingly, the comparison of our theory with simulations of “dynamically annealed” networks shows that it can be considered almost exact for the annealed case. One may ask whether a biological realization of such a “dynamically annealed” network, where our theory becomes exact, might actually exist. A possible origin of such randomly distributed spikes might be the probabilistic vesicle release: A fully connected network where synapses would transmit a spike with probability p would be an exact realization of the system we describe by our approximation. Thus, the annealed network model may also be regarded as a simple caricature of a spiking neural network with probabilistic synaptic transmission. Another type of annealing occurs in networks with full connectivity but dynamic synapses (short-term synaptic plasticity). Recently, such dynamically changing networks have also been reduced to low-dimensional Langevin dynamics based on a similar Poisson neuron model [ 58 , 59 ] . For the sake of a systematic and thorough development of the theory, we here began with the case of one population corresponding to a single neuron type; especially, we focused on the case of an inhibitory network. However, our theory suggests a straightforward extension to multiple populations. Cortical networks typically consist of several cell types and are generally described as excitatory-inhibitory networks that are, in the simplest case, organized as two coupled populations. Rate models of such two-population E-I networks remain a popular tool to describe network dynamics up to this day, whether to characterize inhibition stabilization in cortical networks [ 80 ] or spatial patterns of oscillatory activity observed in motor cortex [ 79 , 81 ] . It will certainly be worthwhile to investigate how the combined effects of random connectivity and network size we began to explore here may shape the more complicated dynamics of networks that comprise more than one population. Furthermore, a multi-population extension would offer the possibility to model specific cortical microcircuits, such as canonical circuit models [ 82 ] or cortical column models with multiple interneuron types [ 41 ] , on the mesoscopic scale, which accounts for fluctuations. To develop the theory, we used Poisson neurons based on a one-dimensional dynamics for the membrane potential of each neuron. While a Poisson model neglects spike-history effects such as neuronal refractoriness, it allowed us to derive a low-dimensional, closed set of mean-field equations for the first- and second-order cumulants. We expect that in regimes where the neuron’s firing is Poissonian, similar observations can be made in more biophysically grounded models of spiking networks. While this is beyond the scope of this work, it would certainly be worthwhile to systematically investigate the effects described here in simulations of such networks. The main difference of our Poisson model compared to more realistic integrate-and-fire (IF) models is the absence of a reset mechanism (in fact, with reset, our model would be equivalent to a leaky IF model with escape noise [ 31 , 83 , 33 , 28 , 84 ] ). While the resetting in IF neurons captures refractoriness, it significantly complicates the population dynamics. Specifically, the population dynamics of IF neurons is usually infinite dimensional, represented by partial differential equations [ 29 , 84 ] , integral equations [ 23 ] , infinite systems of SDEs [ 85 , 66 ] , or, in the case of finite N , stochastic versions of these [ 66 , 36 , 33 , 86 ] . In principle, it seems possible to carry over our theory to these population equations by again using the same annealed approximation. It would be interesting to study how well the annealed approximation works for neurons with refractoriness. One may speculate that similar second-order mean-field theories may provide a quantitatively correct account for networks e.g. of generalized integrate-and-fire neurons in the presence of finite connectivity where effective first-order mean-field theories have already proven to be quantitatively precise for fully connected networks [ 36 , 79 , 33 ] . While a systematic analysis of the effect of resets is beyond the scope of this paper, we present here some first simulation results. We consider a quenched network, in which the input potential is reset to h_{\text{reset}} after each spike. With reset, the single-neuron hazard rate no longer requires an upper bound, and we therefore adopt a rectified linear hazard rate [ 28 ] . As an example, we repeat the simulations from Fig. 5 with reset. The results (Fig. 7 ) show that the variances of the population rate r and mean input potential \bar{h} as a function of the connection probability p behave qualitatively similarly to the case without reset (Fig. 5 ). Specifically, for fixed N , the variance of r increases with p , while the variance of \bar{h} decreases. For fixed C , the variance of r increases, and the variance of \bar{h} exhibits a maximum. Importantly, the annealed approximation now matches the quenched network much more closely than in the absence of reset. This aligns with our earlier explanation that discrepancies between the quenched network and the annealed approximation arise from neglecting temporal correlations in the incoherent recurrent input due to annealing. The reset reduces the enhanced low-frequency power of the model without reset, effectively whitening the spike-train power spectra and thus reducing temporal correlations in the input. In other words, there are less temporal correlations to be removed by the annealing, leading to a better fit. For simulations and concreteness, we made some further specific modeling choices: For example, we studied random networks with fixed in-degree, i.e. an identical number of presynaptic connections, and chose a specific transfer function (or f-I curve) \phi(h) that relates the instantaneous firing rate to the intensity or input current h . While our specific choice of the transfer function also enabled us to analytically evaluate the effect of a finite variance in the input currents on the population rate, the reduction of the spiking network dynamics to a low-dimensional system of coupled SDEs should remain valid for arbitrary choices of the transfer function, albeit probably more difficult to compute. The assumption of fixed in-degree allowed us to assume that the mean input current as well as the strengths of the fluctuating parts are identical among neurons, but in practice we observed that a purely random Erdős-Réniy network shows almost identical statistics as those described in the previous sections for networks with fixed in-degree. Here, we showed how to derive low-dimensional stochastic models that describe fluctuations in a complex, biological model system, namely a finite-size spiking neural networks with random connectivity. Already in the simple case of one population and Poissonian spiking, we found highly non-trivial effects such as the non-monotonic dependence of the variance of the recurrent input on connection probability (Fig. 5 b) and the weak suppression of variability by external stimuli (Fig. 6 b). Thus, even though a single population and Poisson spiking dynamics may be too simple to quantitatively model certain phenomena in cortex (such as a stronger suppression of variability [ 61 ] ), the fact that our MF2 theory explains non-trivial fluctuation effects demonstrates the power of the approach as a proof of principle. Importantly, with the biological extensions discussed above, we believe that our theory may pave the way for a useful modeling framework for variability in cortical circuits at the mesoscopic population level. VI