RSMA-enabled satellite-terrestrial communication networks: Performance analysis Hong-Nhu Nguyen, Quang Sang Nguyen, Tan N Nguyen ABSTRACT This paper investigates the performance of a satellite-terrestrial communication system employing rate-splitting multiple access (RSMA) to accommodate multiple users. The study focuses on assessing the system’s ergodic sum rate and symbol error rate (SER) under shadowed-Rician fading conditions. By deriving closed-form expressions, we analyze the influence of various power allocation strategies on overall system performance. The results, substantiated by numerical simulations, demonstrate that RSMA yields significant improvements over conventional non-orthogonal multiple access (NOMA) with respect to spectral efficiency and interference management. These findings offer valuable insights for the development of more reliable and efficient satellite-terrestrial networks. 1 Introduction Hybrid Satellite–Terrestrial Networks (HSTNs) are increasingly vital in contemporary wireless systems, providing extensive coverage and dependable connectivity in remote and under-served regions. These networks represent a pivotal technology in the progression of next-generation communication systems, including 6G. HSTNs provide a comprehensive framework for addressing both technological advancements and practical challenges inherent, thereby establishing themselves as a foundational component for the seamless integration of terrestrial and non-terrestrial networks in the 6G era and beyond [ 1 ]. With the swift rise in data traffic and the demand for enhanced spectral efficiency, traditional multiple-access methods encounter considerable obstacles in accommodating the growing user base and effectively managing interference [ 2 , 3 ]. To improve spectral efficiency, orthogonal multiple access (OMA) is commonly utilized in satellite communication systems, where distinct frequency, time, or code resources are allocated to different users to prevent interference [ 4 ]. Nonetheless, OMA is limited in its spectral utilization, especially when faced with a high number of users [ 5 ]. To address this drawback, non-orthogonal multiple access (NOMA) has been developed, enabling multiple users to utilize the same resources through power-domain multiplexing [ 6 , 7 ]. Although NOMA enhances spectral efficiency by facilitating successive interference cancellation (SIC) at the receiver, it also presents challenges like error propagation, issues of fairness, and heightened decoding complexity [ 8 , 9 ]. Recently, Rate-Splitting Multiple Access (RSMA) has emerged as a favorable alternative, encompassing both OMA and NOMA [ 10 ]. Unlike NOMA, which depends on SIC for separating signals, RSMA divides transmitted messages into common and private segments, enabling more adaptable interference management [ 11 , 12 ]. The common message is available for all users to decode, while private messages are decoded separately, striking a balance between interference mitigation and spectral efficiency [ 13 ]. Research indicates that RSMA surpasses both OMA and NOMA regarding sum rate, user fairness, and resilience to imperfect channel state information (CSI) [ 14 – 16 ]. However, although RSMA has been thoroughly studied in terrestrial networks, its application in satellite-terrestrial communication is still largely uncharted, particularly under realistic fading conditions like shadowed-Rician fading [ 17 ]. 1.1 Literature review Since its introduction, RSMA has gained significant research attention, with numerous studies focusing on resource allocation optimization and precoder design to enhance user rate fairness, maximize sum rate, and improve bit error rate (BER) and symbol error rate (SER) performance. For instance, the work in [ 18 ] explored a joint optimization framework for rate-splitting precoding, subcarrier assignment, and power allocation to ensure fair rate distribution among users in multi-cast transmissions. To further analyze BER performance under finite constellation and specific coding rates, a transceiver architecture was proposed, incorporating low-density parity-check (LDPC) encoding, modulation, and RS-based signal splitting at the transmitter, along with maximum a posteriori (MAP) detection, LDPC decoding, and SIC at the receiver. Additionally, in [ 19 ], researchers examined half-duplex cooperative communication (HCC) for RSMA users, demonstrating that even with partial CSI, jointly optimizing the precoder, time allocation, and relay node selection could still achieve a higher weighted sum rate (WSR) compared to non-cooperative communication (NCC). Existing studies can therefore be broadly categorized into two main methodological directions. Regarding the first methodology, which avoids the use of a precoder, the majority of research concentrates on utilizing the inherent advantages of rate-splitting techniques to enhance capacity. For example, the study in [ 20 ] introduced fixed and cognitive rate splitting strategies employing fixed and cyclic single-carrier transmissions to improve the fairness of pairing nearby users with distant users while also reducing their overall outage probability in delay-constrained transmission scenarios. In [ 21 ], a novel reconfigurable intelligent surface (RIS) rate-splitting framework was created utilizing an offline control mechanism to minimize outage occurrences for user communication. In [ 22 ], the application of rate-splitting for Low Earth Orbit (LEO) satellite communication (SatCom) was investigated with the objective of maximizing the sum rate of LEO SatCom systems. This was achieved through the simultaneous optimization of the power budget across multiple beams, RSMA power allocation for users within each beam, and subcarrier user assignment. Additionally, measures were implemented to ensure that interference temperature constraints to GEO SatCom were strictly maintained. Specifically, the research presented in [ 23 ] examined the outage probability and ergodic capacity approximations for MIMO systems while accounting for ULA and imperfect CSI, alongside a simulated assessment with various types of phase-shift keying (PSK) and quadrature amplitude modulation. In the context of downlink multiple-input single-output (MISO) networks featuring cell-edge users supported by reconfigurable intelligent surfaces and LRW precoders, the outage probability was extensively analyzed with a two-layer hierarchical resource sharing approach employing an on-off control method in [ 24 ] and a singular RS layer integrated with discrete phase shifters in [ 25 ]. 1.2 Motivations and contributions While RSMA has been thoroughly examined within multi-antenna and massive MIMO frameworks, its application to multi-user satellite-terrestrial communication systems remains insufficiently explored [ 26 , 27 ]. Additionally, existing research on RSMA often assumes the availability of perfect CSI, which is difficult to obtain in practical satellite networks due to significant mobility and beam alignment challenges [ 28 ]. To tackle this concern, this paper offers an in-depth analysis of RSMA-based satellite-terrestrial communication systems, focusing on ergodic sum rate and SER results under realistic fading conditions. The main contributions of this study are summarized as follows: Ergodic Sum Rate Analysis: We establish a closed-form integral expression for the ergodic sum rate in RSMA-driven satellite-terrestrial networks. A high-SNR approximation is included to provide a further understanding of system behavior at elevated transmit power levels. SER Evaluation: We evaluate the SER performance of both common and private messages during RSMA transmission. The influence of power allocation and modulation techniques (e.g., BPSK, QPSK) on SER is explored. Impact of Power Allocation and Fading Conditions: We investigate how the power coefficient for the common stream affects system performance, particularly under shadowed-Rician fading. Comparison with NOMA: We perform numerical simulations to juxtapose RSMA and traditional NOMA, emphasizing RSMA’s benefits in spectral efficiency and interference handling. From an implementation standpoint, RSMA generally entails lower and more predictable receiver complexity than power-domain NOMA in satellite downlinks. Each user decodes the common stream once, cancels it via a single SIC step, and then decodes its private stream, so the SIC depth per terminal remains essentially constant. In contrast, NOMA typically requires successive decoding of multiple users’ signals (especially at strong users), which increases decoding latency and can amplify error propagation. While RSMA requires optimization of splitting and power allocation, such processing can be centralised (e.g., at the gateway), whereas NOMA additionally relies on user ordering/pairing decisions and related signalling overhead. The outcomes of this research provide critical insights that are necessary for improving RSMA-based satellite-terrestrial networks. These findings directly contribute to the development of more efficient and robust next-generation satellite communication systems. 1.3 Organization & notations The remainder of this paper is organized as follows: Section 2 introduces the system model, explaining the satellite-terrestrial channel characteristics and the RSMA transmission framework. Section 3 focuses on the derivation of the ergodic sum rate and its implications for system performance. Section 4 analyzes the SER, considering different power allocation strategies. Section 5 presents numerical results, comparing RSMA with conventional multiple access schemes and discussing key observations. Finally, Section 6 concludes the paper and suggests directions for future research. For clarity, the key notations used throughout this paper are summarized in Table 1 . 10.1371/journal.pone.0357013.t001 Table 1 Summary of Key Notations. Notation Description K , Q Number of satellite antennas and terrestrial users, respectively S , U q The GEO satellite and the q -th user 𝐡 U q , 𝐠 U q Channel gain and Shadowed-Rician channel vector for U q 𝐰 U q Transmit beamforming vector for U q P S , ϱ S Total transmit power and transmit SNR at the satellite S x c , x q Common message and private message for U q a c , a q Power allocation coefficients for x c and x q γ ¯ c , q , γ ¯ p , q SINR for decoding the common and private messages at U q δ q Scaling factor including path loss and antenna gain δ q = ϱ S L S U q ϑ S ϑ ( θ U q ) 𝒜 q Effective channel gain component, 𝒜 q = δ q ‖ 𝐠 U q ‖ F 2 ψ q Auxiliary fading parameter defined as ψ q = β q − δ q Δ q Auxiliary parameter defined as ∑ l = 1 K j l + K R c , q , R p , q Ergodic rate of the common and private streams at U q S c , q , S p , q Symbol error rate (SER) of the common and private messages at U q I i ( · ) , 1 F 1 ( · ) Modified Bessel function of the first kind and confluent hypergeometric function ℬ ( · ) , γ ( · ) , G p , q m , n [ · ] Beta function, lower incomplete Gamma function, and Meijer’s G-function Q ( x ), ( · ) t Gaussian Q-function and Pochhammer symbol 2 System model 2.1 System description Fig 1 illustrates the considered downlink RSMA-enabled satellite–terrestrial system. A GEO satellite S equipped with K antennas simultaneously serves Q single-antenna terrestrial users { U q } q = 1 Q . The satellite superposes a common stream x c and private stream { x q } with power fractions a c and { a q } , respectively, under the total power constraint a c + ∑ q = 1 Q a q = 1 . The effective channel to user U q is h U q = g U q † w U q L S U q ϑ S ϑ ( θ U q ) , where g U q follows a shadowed-Rician model, w U q is selected according to MRT, and L S U q and ϑ ( θ U q ) capture the free-space loss and satellite beam gain, respectively. Each user decodes x c first while treating all private streams as interference, performs SIC to remove x c and then decodes its own private stream in equation (4) , leading to the SINRs in equations (5) and (6) . Unless otherwise stated, we adopt ideal (benchmark) CSI at the transmitter for MRT beamforming and analysis tractability, as discussed in this section. 10.1371/journal.pone.0357013.g001 Fig 1 An illustration of a downlink RSMA-enabled satellite-terrestrial communication system. Modern satellite systems often use multi-beam technology to enhance coverage and capacity. For geosynchronous Earth orbit (GEO) satellites, these beams are usually generated by array-fed reflectors, which offer better efficiency compared to direct radiating arrays. The beam pattern remains fixed, reducing the need for complex onboard processing. The signal received by each user depends on the satellite’s beamforming strategy, propagation conditions, and power allocation. The channel gain between the satellite and the q th user is given by h U q = g U q † w U q L S U q ϑ S ϑ ( θ U q ) , (1) where ϑ S represents the satellite antenna gain, and g U q denotes the K × 1 Shadowed-Rician channel vector linking K antennas at S . The notation ( . ) † indicates the conjugate transpose, and w U q is the K × 1 vector of transmit weights. The transmit beamforming vector w U q ∈ ℂ K × 1 is chosen based on the maximum ratio transmission (MRT) principle, such that w U q = ‖ g U q ‖ ‖ g U q ‖ F , where ‖ . ‖ F indicates the Frobenius norm. Additionally, L S U q = 1 K B T W ( c 4 π f c d S U q ) 2 represents the instantaneous free space loss [ 29 ], where 1 K B T W serves as a scaling factor, with K B = 1.38 × 10 − 23 J · K designated as the Boltzmann constant, T as the noise temperature of the receiver, W as the bandwidth of the carrier, c as the speed of light, f c as the frequency of the carrier, and d S U q as the distance separating S and U q . Furthermore, the beam gain ϑ ( θ U q ) of the satellite can be expressed as ϑ ( θ U q ) = ϑ U q ( I 1 ( ρ ¯ U q ) 2 ρ ¯ U q + 36 I 3 ( ρ ¯ U q ) ρ ¯ U q 3 ) , (2) where I i ( . ) is the first-kind Bessel function with order i , θ U q is the antenna gain at U q , θ U q is the angular separation and ρ ¯ U q = 2.07123 sin θ U q sin θ U q 3 d B in which sin θ U i 3 d B represents the 3 dB beam-width. Furthermore, it is assumed that the ideal CSI g U q is accessible to the satellite user. This can be achieved by using training symbols for the satellite connection, together with an existing feedback link or spectrum manager (which acts as an intermediary between the two systems) for the terrestrial interference link (Perfect CSI is assumed to obtain a tractable analytical benchmark and to highlight the intrinsic RSMA gains under shadowed-Rician satellite fading. In practice, CSI acquisition in satellite–terrestrial links is affected by estimation errors, feedback quantization, and latency. For GEO multi-beam downlinks, the relatively slow variation of the effective channel and the availability of pilot-assisted estimation with a return link (or a spectrum manager) can make the CSI quality sufficiently high to approximate the perfect-CSI benchmark. For more dynamic scenarios (e.g., LEO or rapid beam tracking), imperfect CSI models are required; robust RSMA designs under CSI uncertainty constitute an important extension and will be considered in future work.) [ 30 ]. 2.2 The signal processing at transceivers In this system, the satellite uses RSMA to transmit signals to multiple users at the same time. The transmitted signal consists of two parts: a common message that is decoded by all users and private messages intended for individual users. To balance performance, the satellite allocates power to these messages using a power coefficient a c for the common message, while the remaining power is distributed among the private messages. The transmitted signal can be expressed as follows: x = P S ( a c x c + ∑ q = 1 Q a q x q ) , (3) Here P S is the total transmit power at S, which x c is the common message, allocated with power a c P S and x q is the private message for user q , allocated with power a q P S . It is important to note that a c + ∑ q = 1 Q a q = 1 . The signal obtained by the q th user can be articulated as y U q = h U q x + n U q = g U q † w U q L S U q ϑ S ϑ ( θ U q ) a c P S x c ⏟ C o m m o n M e s s a g e + g U q † w U q L S U q ϑ S ϑ ( θ U q ) a q P S x q ⏟ D e s i r e d P r i v a t e M e s s a g e + ∑ j = 1 , j ≠ q Q g U q † w U q L S U q ϑ S ϑ ( θ U q ) a j P S x j ⏟ I n t e r f e r e n c e + n U q ⏟ A W G N , (4) Here n U q represents the additive white Gaussian noise (AWGN) characterized by a mean of zero and a variance denoted by σ q 2 . To decode the received signal, each user follows a two-step process. 2.3 Decoding the common message Each user first decodes the common message x c while treating all private messages as interference. The signal-to-interference-plus-noise ratio (SINR) for decoding the common message at the user q is: γ ¯ c , q = a c δ q P S ‖ g U q ‖ F 2 δ q P S ( 1 − a c ) ‖ g U q ‖ F 2 + σ q 2 = a c A q ( 1 − a c ) A q + 1 , (5) where ϱ S = P S · σ q 2 is the transmit signal-to-noise ratio (SNR), while A q = δ q ‖ g U q ‖ F 2 and δ q = ϱ S L S U q ϑ S ϑ ( θ U q ) . 2.4 Decoding the private message Once the common message is successfully decoded and removed from the received signal, each user proceeds to decode its own private message. The remaining interference comes from private messages intended for other users. The SINR for decoding the private message is given by γ ¯ p , q = a q A q A q ∑ j = 1 , q ≠ j Q a j + 1 . (6) The efficiency of this process depends on the power allocation and channel conditions. By carefully choosing a c and a q , RSMA ensures a balance between spectral efficiency and interference management, improving overall system performance. Having established the signal model and the RSMA decoding procedure, we briefly discuss key practical implementation considerations before introducing the channel model used for performance analysis. Practical deployment of RSMA in satellite-terrestrial networks entails several implementation challenges. First, multi-antenna transmission relies on sufficiently accurate and timely CSI to support beamforming; in operational systems, CSI is typically acquired via pilot-assisted estimation and delivered through a feedback/coordination mechanism, and it may be affected by latency, quantization, and beam pointing errors. In this work, we adopt ideal CSI as a benchmark to enable MRT beamforming and to keep the analysis tractable, while acknowledging that imperfect CSI modeling and robust design are important extensions. Second, RSMA requires adaptive configuration of the common/private power split ( a c , { a q } ) and, in general, the allocation of the common rate across users, which can impose additional control-plane signaling and scheduling overhead, particularly when users experience heterogeneous channel conditions. Third, although the receiver-side processing in RSMA has relatively bounded complexity—each user decodes the common stream, removes it via SIC, and then decodes its private stream—non-ideal SIC and practical receiver impairments can still lead to residual interference and performance loss. Finally, satellite system constraints (e.g., payload processing capability, feeder-link limitations, and beam management) may restrict the update rate and granularity of precoding and power control, making low-complexity or gateway-centric implementations especially relevant. Motivated by these practical considerations, the following subsection formalizes the satellite channel statistics used in our performance analysis. 2.5 Terrestrial channel model To facilitate the computation of system performance metrics, we assume that the channel coefficients are independent and identically distributed (i.i.d.). Under this assumption, the probability density function (PDF) of the channel coefficient g q ( k ) , for all q ∈ Q , representing the link from the satellite’s k th antenna to the q t h user, can be expressed as follows: f | g q ( k ) | 2 ( x ) = α q e − β q x 1 F 1 ( m q ; 1 ; ϖ q x ) , x ≥ 0 , (7) Where: α q = ( 2 b q m q / ( 2 b q m q + Ω q ) ) m q / 2 b q , β q = 1 / 2 b q , ϖ q = Ω q / ( 2 b q ) ( 2 b q m q + Ω q ) in which Ω q , 2 bq , and mq represent the average power of the line-of-sight (LOS) component, the average power of the multipath components, and the fading severity parameter, respectively. 1 F 1 ( · ; · ; · ) denotes a confluent hypergeometric function of the first kind, as referenced in [ 31 , Eq. (9.210.1)]. For the purposes of this paper, we assume integer values for the Shadowed-Rician fading severity parameter m q . This assumption facilitates a more straightforward analysis of the channel’s statistical characteristics and their influence on system performance metrics. Accordingly, equation (7) can be reformulated as follows: f | g q ( k ) | 2 ( x ) = α q e − ( β q − ϖ q ) x ∑ t = 0 m q − 1 ζ q ( t ) x t , x ≥ 0 , (8) where ζ q ( t ) = ( − 1 ) t ( 1 − m q ) t ϖ q t / ( t ! ) 2 and ( . ) t denotes the Pochhammer symbol. Drawing upon the findings presented in [ 32 ], the PDF of A q under independent and identically distributed (i.i.d.) Shadowed-Rician fading can be expressed as follows: f A q ( x ) = ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 Λ q ( K ) δ q Δ q x Δ q − 1 e − ( ψ q δ q ) x , (9) where Λ q ( K ) = α q K ∏ l = 1 K ζ q ( j l ) ∏ u = 1 K − 1 B ( ∑ p = 1 u j p + u , j u + 1 + 1 ) , (10) In this context, Δ q = ∑ l = 1 K j l + K , ψ q = β q − δ q , (11) The Beta function B ( . , . ) is referenced from [ 31 , Eq. (8.384.1)]. To obtain the CDF of γ i , we directly utilize the result from [ 31 , Eq. (3.351.1)]: F A q ( x ) = ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 Λ q ( K ) ψ q Δ q γ ( Δ q , ψ q x δ q ) . (12) where γ ( . , . ) is the lower incomplete Gamma function. Using [ 31 , Eq. (8.352.6)], (12) can be simplified as F A q ( x ) = 1 − ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p e − ψ q x δ q x p . (13) Remark 1: This formulation captures the statistical properties of the channel and facilitates a comprehensive understanding of the system’s performance metrics under the designated fading conditions. 3 Integral expression of ergodic sum rate In terms of the instantaneous SINR, the ergodic rate of the common/private stream for the q th user can be given as [ 33 , 34 ] R a , q = 𝔼 [ log 2 ( γ ¯ a , q + 1 ) ] = 1 ln 2 ∫ 0 ∞ [ 1 − F γ ¯ a , q ( x ) ] 1 + x d x , (14) where R a , q is normalized in the unit of bps/Hz. a = c denotes the common stream, whereas a = p denotes the private stream. The common stream carries information intended to be decoded by all users at the receiver side. To guarantee that the shared stream can be interpreted by every user, the ergodic rate of the shared stream for each user must meet R c = min ( R 1 , c , R 2 , c , . . . , R Q , c ) , (15) ∑ q = 1 Q C q = R c . (16) In the satellite-terrestrial communication system that employs RSMA, the ergodic sum rate for the q th user is the total of the common rate and the private rate, which can therefore be expressed as R s u m = ∑ q = 1 Q ( C q + R p , q ) = min ( R 1 , c , R 2 , c , . . . , R Q , c ) + ∑ q = 1 Q R p , q . (17) Proposition 1: The ergodic sum rate may be accurately expressed as follows: R s u m ≃ min q ∈ { 1 , 2 , . . . , Q } 1 ln 2 ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × [ B 1 p G 1 , 2 2 , 1 ( ψ q B 1 δ q | − p − p , 0 ) − B 2 p G 1 , 2 2 , 1 ( ψ q B 2 δ q | − p − p , 0 ) ] + 1 ln 2 ∑ q = 1 Q 1 ln 2 ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × [ B 3 p G 1 , 2 2 , 1 ( ψ q B 3 δ q | − p − p , 0 ) − B 4 p G 1 , 2 2 , 1 ( ψ q B 4 δ q | − p − p , 0 ) ] , (18) B 1 = [ a c + ( 1 − a c ) ] − 1 , B 2 = ( 1 − a c ) − 1 , B 3 = [ a q + ( 1 − a c − a q ) ] − 1 and B 4 = a q − 1 . Proof: First, substituting (5) into (14) we have the ergodic rate for decoding common messages is given by R c , q = 𝔼 [ log 2 ( γ ¯ c , q + 1 ) ] = 𝔼 [ log 2 ( a c A q ( 1 − a c ) A q + 1 ⏟ X + 1 ) ] = 1 ln 2 ∫ 0 ∞ [ 1 − F X ( x ) ] 1 + x d x = 1 ln 2 ∫ 0 a c 1 − a c 1 1 + x [ 1 − F A q ( x a c − ( 1 − a c ) x ) ] d x . (19) By changing the variable t = x a c − ( 1 − a c ) x and with the help of [ 35 ], R c , q can be given by R c , q = 1 ln 2 ∫ 0 ∞ [ 1 t + B 1 − 1 t + B 2 ] [ 1 − F A q ( t ) ] d t , (20) where B 1 = [ a c + ( 1 − a c ) ] − 1 and B 2 = ( 1 − a c ) − 1 . Plugging (13) into (20) , R c , q is rewritten as R c , q = 1 ln 2 ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × ∫ 0 ∞ [ 1 t + B 1 − 1 t + B 2 ] e − ψ q t δ q t p d t = 1 ln 2 ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × [ ∫ 0 ∞ t p e − ψ q t δ q t + B 1 d t − t p e − ψ q t δ q t + B 2 d t ] . (21) To solve the integrals in (21) , we use the following transformation using the Meijer G-function [ 36 ] e − ψ q t δ q = G 0 , 1 1 , 0 ( ψ q t δ q | − 0 ) . (22) By substituting (22) into (21) , R c , q can be expressed as R c , q = 1 ln 2 ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × [ ∫ 0 ∞ x p G 0 , 1 1 , 0 ( ψ q t δ q | − 0 ) t + B 1 d t − x p G 0 , 1 1 , 0 ( ψ q t δ q | − 0 ) t + B 2 d t ] , (23) Building upon [ 31 , Eq. (7.811.5)] the ergodic rate of the common stream can be accurately derived as R c , q = 1 ln 2 ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × [ B 1 p G 1 , 2 2 , 1 ( ψ q B 1 δ q | − p − p , 0 ) − B 2 p G 1 , 2 2 , 1 ( ψ q B 2 δ q | − p − p , 0 ) ] . (24) Similarly, by using (6) , the ergodic rate of the private message can be expressed as R p , q = 𝔼 [ log 2 ( γ ¯ p , q + 1 ) ] . Solving R p , q using steps similar to (23) yields R p , q = 1 ln 2 ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × [ B 3 p G 1 , 2 2 , 1 ( ψ q B 3 δ q | − p − p , 0 ) − B 4 p G 1 , 2 2 , 1 ( ψ q B 4 δ q | − p − p , 0 ) ] , (25) where B 3 = [ a q + ( 1 − a c − a q ) ] − 1 and B 4 = a q − 1 . Using (25) and (24) into (17) , we get (18) . This completes the proof of Proposition 1 . High SNR Region: When ϱ S → ∞ , the law of large numbers may be applied to estimate γ ~ c , q * , ∞ → a c 1 − a c and γ ~ p , q * , ∞ → a q 1 − a c − a q . The respective ergodic rate boundaries of x c and x q at U q can be derived as R c , q ∞ ≃ log 2 ( 1 + a c / 1 − a c ) , (26a) R p , q ∞ ≃ log 2 ( 1 + a q / 1 − a c − a q ) . (26b) By substituting (26b) and (26a) into (17) , the ergodic sum rate under high SNR circumstances may be represented as R s u m ∞ ≃ R c , q ∞ + ∑ q = 1 Q R p , q ∞ = log 2 ( 1 1 − a c ) + ∑ q = 1 Q log 2 ( 1 − a c 1 − a c − a q ) . (27) Remark 2: The high-SNR approximation reveals that the ergodic sum rate grows logarithmically with transmit power, while the choice of a c critically affects the balance between common and private message rates. This emphasizes the need for optimal power allocation in RSMA-based satellite-terrestrial networks. 4 SER analysis The SER for a given modulation scheme can be expressed as [ 37 , Eq. (37)] S a , q = r 𝔼 { Q ( v γ ¯ a , q ) } = r v 2 2 π ∫ 0 ∞ e − v 2 x x F γ ¯ a , q ( x ) d x , (28) where a ∈ { c , p } represents either the common message ( c ) or the private message ( p ), Q ( x ) = 1 2 π ∫ x ∞ e − t 2 · 2 d t is the Gaussian Q-function, r and v are modulation-specific constants (e.g., for BPSK, r = 1, v = 2; for QPSK, r = 2, v = 1) in [ 38 ] and γ ¯ a , q is the SINR for decoding either the common or private message at user q . To analyze the SER, the derived SINR expressions are substituted into above equation and apply statistical methods to evaluate the expected value over the fading distribution. 4.1 SER of the common message Proposition 2: Since all users need to successfully decode the common message, its SER is determined by the worst-performing user in the network. Using probability theory and integral approximations, The closed-form approximate expression of SER for the common message is approximated as S c , q ≈ r v 2 2 π [ 2 π v Φ ( v a c 2 ( 1 − a c ) ) − π a c 2 W ( 1 − a c ) × ∑ w = 1 W 1 − ξ w 2 e − v S ( ξ w ) 2 − ψ q G ( ξ w ) δ q G ( ξ w ) p S ( ξ w ) ] , (29) where W is a trade-off parameter controlling complexity and accuracy, Φ ( . ) is the Error function, G ( t ) = t + 1 ( 1 − a c ) ( 1 − t ) , S ( t ) = a c ( t + 1 ) 2 ( 1 − a c ) and ξ w = cos ( 2 w − 1 2 W π ) . Proof: Submitting (5) into (28) , we have S c , q is given by S c , q = r v 2 2 π ∫ 0 ∞ e − v 2 x x F γ ¯ c , q ( x ) d x = r v 2 2 π ∫ 0 a c 1 − a c e − v 2 x x [ 1 − F A q ( x a c − ( 1 − a c ) x ) ] d x . (30) From (13) , let’s rewrite the result in (30) as S c , q = r v 2 2 π ∫ 0 a c 1 − a c e − v 2 x x [ 1 − ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × e − ψ q ( x a c − ( 1 − a c ) x ) δ q ( x a c − ( 1 − a c ) x ) p ] d x = r v 2 2 π [ ∫ 0 a c 1 − a c e − v 2 x x d x ⏟ I 1 − ∑ j 1 = 0 m q − 1 ⋯ ∑ j K = 0 m q − 1 ∑ p = 0 Δ q − 1 Λ q ( K ) Γ ( Δ q ) p ! ψ q Δ q − p δ q p × ∫ 0 a c 1 − a c e − v 2 x x e − ψ q ( x a c − ( 1 − a c ) x ) δ q ( x a c − ( 1 − a c ) x ) p d x ⏟ I 2 ] . (31) For the first integration of (31) , using [ 31 , Eq. (3.361.1)], I 1 is given by I 1 = 2 π v Φ ( v a c 2 ( 1 − a c ) ) , (32) where Φ ( . ) is the Error function. For the second integral, setting t = 2 ( 1 − a c ) x a c − 1 results in x = a c ( t + 1 ) 2 ( 1 − a c ) . Hence, I 2 is given as I 2 = a c 2 ( 1 − a c ) ∫ − 1 1 e − v S ( t ) 2 − ψ q G ( t ) δ q S ( t ) G ( t ) p d t , (33) where G ( t ) = t + 1 ( 1 − a c ) ( 1 − t ) and S ( t ) = a c ( t + 1 ) 2 ( 1 − a c ) . Using the Gaussian-Chebyshev quadrature method in [ 39 , 40 ], we can obtain I 2 as I 2 ≈ π a c 2 W ( 1 − a c ) ∑ w = 1 W 1 − ξ w 2 e − v S ( ξ w ) 2 − ψ q G ( ξ w ) δ q S ( ξ w ) G ( ξ w ) p , (34) where ξ w = cos ( 2 w − 1 2 W π ) . Plugging (34) and (32) into (31) , we will acquire (29) . The proof is completed. 4.2 SER of private messages For private messages, the SER depends on the ability of each user to successfully decode its intended signal while treating interference from other users as noise. In a like manner, employing (6) , the SER for private messages can be represented as S p , q = r 𝔼 { Q ( v γ ¯ p , q ) } . By following a process similar to that in (29) to deduce S p , q , we arrive at S p , q ≈ r v 2 2 π [ 2 π v Φ ( v a q 2 ( 1 − a c − a q ) ) − π a q 2 W × 1 ( 1 − a c − a q ) ∑ w = 1 W 1 − ξ w 2 e − v S ~ ( ξ w ) 2 − ψ q G ~ ( ξ w ) δ q G ~ ( ξ w ) p S ~ ( ξ w ) ] , (35) where G ~ ( t ) = t + 1 ( 1 − a c − a q ) ( 1 − t ) and S ~ ( t ) = a q ( t + 1 ) 2 ( 1 − a c − a q ) . 4.3 High SNR region At high transmit power levels, the SER expressions simplify significantly. In the high-SNR regime ( ϱ S → ∞ ), the SER can be approximated as S c , q = r Q ( v a c 1 − a c ) , (36a) S p , q = r Q ( v a q 1 − a c − a q ) . (36b) Remark 3: This shows that increasing a c improves the reliability of the common message, but excessive allocation can degrade private message performance. 5 Results and discussions In this section, numerical simulations are presented to validate the derived formulas. The parameters for Shadowed-Rician fading can be found in Table 2 , while the details for the numerical outcomes are consolidated in Table 3 . We consider a baseline scenario involving two users ( Q = 2). Furthermore, the equivalent noise power at U q was determined as σ q 2 = N 0 + 10 log ( B W ) + N F [dBm] as referenced and the respective power allocation coefficients are a 1 = 0.6 ( 1 − a c ) and a 2 = 0.4 ( 1 − a c ) . 10.1371/journal.pone.0357013.t002 Table 2 Table of Satellite Channel Parameters [ 41 – 43 ]. Shadowing b q m q Ω q Average shadowing (AS) 0.251 5 0.279 Heavy shadowing (HS) 0.063 1 0.0007 10.1371/journal.pone.0357013.t003 Table 3 Main system parameters. Monte Carlo simulations 10 6 iterations Satellite type GEO Antennas of satellite K = 2 Number of users Q = 2 Power allocation of common data a c = 0.4 Carrier frequency f c = 2 [GHz] Carrier bandwidth W = 15 [Mhz] Receiver noise temperature T = 300 ∘ K Speed of light c = 3 × 10 8 [m/s] Distance from satellite to U q d S U q = 35786 [Km] Antenna gain at satellite ϑ S = 53.45 [dB] Antenna gain at satellite and U q ϑ U q = 4.8 [dB] Angular separation θ U q = 0.8 ∘ 3dB beamwidth θ U q 3 d B = 0.3 ∘ Bandwidth B W = 10 [MHz] Noise figure N F = 10 [dBm] Thermal noise power density N 0 = − 174 [dBm/Hz] Fig 2 presents the ergodic sum rate as a function of transmit power P S for different values of the power coefficient a c under both AS and HS conditions. The results show that the simulation ergodic sum rate increases with P S , aligning with the analytical expression in (18) . The high SNR region in (27) closely matches the exact analysis at high power levels, validating its accuracy. Under AS conditions, the system achieves a significantly higher sum rate than in HS, demonstrating the impact of severe fading on performance. The choice of a c also plays a crucial role, as a higher a c initially improves the sum rate but may lead to diminishing gains at higher P S . This suggests that optimizing a c is essential for maximizing system performance in satellite-terrestrial networks. 10.1371/journal.pone.0357013.g002 Fig 2 Ergodic sum rate versus P S for power allocation factors a c , with Q = 2, K = 2 and { a 1 , a 2 } = { { 0.36 , 0.24 } , { 0.12 , 0.08 } } . Fig 3 clearly demonstrates that the ergodic sum rate increases substantially with the number of satellite antennas ( K ). The results confirm that increasing K provides improvements of sum rate, driven by superior beamforming gains and more effective interference mitigation. Moreover, it is evident that ergodic sum rates under AS conditions consistently outperform those under HS, owing to reduced fading effects. These findings underscore the decisive influence of channel conditions on system performance and make a compelling case for deploying additional antennas in RSMA-based satellite-terrestrial communication systems. 10.1371/journal.pone.0357013.g003 Fig 3 The ergodic sum rate versus K with increasing number of antennas at satellite, for Q = 2, a c = 0.4 , { a 1 , a 2 } = { 0.36 , 0.24 } and P S = − 5 dBm. Fig 4 compares the ergodic sum rate of RSMA, NOMA and OMA as the transmit power P S increases under AS and HS scenarios. The results show that RSMA consistently achieves a higher sum rate than NOMA at all power levels. This is because RSMA effectively manages interference by splitting messages into common and private parts, allowing for more flexible decoding at the receiver. The performance gap is more noticeable at lower P S , where interference plays a bigger role. As P S increases, all schemes improve, with OMA being lower at low P S (from −30–0 dBm) but becoming the highest from 0 dBm on-wards due to the absence of interference saturation. It is also worth noting that OMA outperforms RSMA in the high-transmit-power region in Fig 4 . This behaviour can be explained by the different interference characteristics of the considered schemes. At low and moderate transmit powers, the system performance is strongly affected by noise, and RSMA benefits from its ability to partially decode and manage interference through the common stream. Hence, RSMA achieves a higher ergodic sum rate than NOMA and OMA in this operating region. However, as the transmit power increases, the noise term becomes less significant, and the performance of RSMA becomes increasingly limited by the residual inter-user interference among the private streams. This leads to an interference-saturation effect in the high-SNR regime. By contrast, OMA allocates orthogonal time/frequency resources to different users, thereby avoiding intra-resource inter-user interference. Although this orthogonalization causes a resource-partitioning loss, the absence of inter-user interference enables OMA to achieve a higher sum rate than RSMA at sufficiently high transmit powers under the considered power allocation and channel settings. This observation does not contradict the advantages of RSMA; rather, it highlights that the relative performance of RSMA, NOMA, and OMA depends on the operating SNR regime, interference level, and resource-allocation strategy. 10.1371/journal.pone.0357013.g004 Fig 4 Comparison between RSMA and NOMA for the ergodic sum rate versus P S , with K = 3 and a c = 0.8 , a 1 = 0.12, a 2 = 0.08 and Q = 2. Fig 5 illustrates the variation of the ergodic sum rate changes with the power coefficient a c for the common stream. The results indicate that increasing a c initially improves the sum rate, but beyond a certain point, the performance starts to decline. This happens because allocating too much power to the common stream reduces the power available for private streams, which limits overall system efficiency. The optimal value of a c depends on system conditions, such as the number of users and channel fading. The results also show that the sum rate is higher under AS than HS, as stronger fading in HS reduces signal quality. These findings highlight the importance of properly adjusting a c to balance common and private stream power for the best system performance. 10.1371/journal.pone.0357013.g005 Fig 5 The ergodic rate versus power coefficient ( a c ) , with Q = 2, K = 2, m q = 1 , b q = 0.063 , Ω q = 0.0007 and P S = − 10 dBm. Fig 6 illustrates the ergodic sum rate of the proposed RSMA scheme versus P S for Q = 2, 3, and 4 users under both HS and AS conditions. It can be observed that the ergodic sum rate decreases slightly as the number of users Q increases. This trend is theoretically justified since the system operates in an overloaded regime. In such scenarios, the spatial degrees of freedom are limited, which prevents the MRT beamforming from completely eliminating inter-user interference among the private streams. Furthermore, because the common stream’s rate is strictly bounded by the user experiencing the worst channel conditions, adding more users increases the probability of encountering a deeper fade, thereby compressing the common rate floor. Nevertheless, the system avoids severe outage and maintains stable connectivity across all users, confirming the scalability of the RSMA transmission framework under massive access setups. 10.1371/journal.pone.0357013.g006 Fig 6 Ergodic sum rate versus P S different numbers of users Q ∈ { 2 , 3 , 4 } with K = 2 and a c = 0.4 under HS and AS environments. Fig 7 presents the SER versus the transmit power P S under different fading conditions. The results show that as P S increases, the SER decreases for all cases, which is expected because higher transmit power improves signal strength and reduces detection errors. However, the rate of improvement varies depending on the shadowing conditions. The SER is consistently lower under AS compared to HS since stronger fading in HS degrades the signal quality. The results also confirm that the analytical expression closely follows the simulation data, validating the accuracy of the derived model. These findings emphasize the importance of considering fading effects in system design, especially in low-power scenarios where SER is more sensitive to channel variations. 10.1371/journal.pone.0357013.g007 Fig 7 The SER versus P S , with Q = 2, K = 4, { a 1 , a 2 } = { 0.48 , 0.32 } and a c = 0.2 . Finally, Fig 8 illustrates the SER versus the power coefficient a c for the common stream under BPSK and QPSK modulation schemes. The results show that as a c increases, the SER consistently decreases. This trend indicates that allocating more power to the common stream improves the overall detection accuracy, reducing symbol errors. Additionally, the SER for BPSK is lower than for QPSK at the same a c , which aligns with the fact that BPSK is more robust to noise and fading compared to QPSK. The SER under AS is also lower than under HS, showing that severe fading conditions significantly degrade performance. These findings highlight the importance of adjusting a c appropriately and considering modulation type to enhance reliability in satellite-terrestrial communication systems. 10.1371/journal.pone.0357013.g008 Fig 8 The SER versus power coefficient ( a c ) , with Q = 2, K = 6, m q = 5 , b q = 0.251 , Ω q = 0.279 and P S = − 20 dBm. 6 Conclusion This paper investigated the performance of an RSMA-enabled satellite-terrestrial communication system under shadowed-Rician fading. Closed-form analytical expressions were derived for the ergodic sum rate and symbol error rate, providing useful insight into the effects of power allocation, fading severity, and the number of satellite antennas on system performance. The numerical results verified the accuracy of the proposed analysis and showed that RSMA can achieve notable performance gains over conventional NOMA in terms of spectral efficiency and interference management, particularly under challenging fading conditions. Nevertheless, it is important to emphasize that the analytical framework developed in this work relies on several idealized benchmark assumptions. In particular, perfect CSI is assumed to enable tractable MRT beamforming and closed-form performance analysis. In practical satellite-terrestrial systems, CSI may be imperfect due to channel estimation errors, limited or quantized feedback, feedback delay, beam pointing mismatch, and CSI aging. Such imperfections may reduce the accuracy of beamforming and power allocation, thereby degrading the achievable ergodic sum rate and SER performance. In addition, the present analysis assumes ideal SIC at the receivers. In practical implementations, SIC may be affected by decoding errors, receiver impairments, and residual interference. Residual SIC can limit the effectiveness of common-stream cancellation and may increase the interference observed during private-stream detection, leading to performance degradation, especially in interference-limited operating regimes. Moreover, the considered analytical model does not explicitly capture dynamic satellite mobility. While the GEO-based benchmark provides a useful reference case, more dynamic satellite scenarios, such as LEO systems, may experience time-varying propagation distances, Doppler shifts, rapid beam movement, and frequent channel variations. These effects can make CSI acquisition, beam tracking, and adaptive resource allocation significantly more challenging. Therefore, the results presented in this paper should be interpreted as a theoretical benchmark for RSMA-enabled satellite-terrestrial networks under idealized operating conditions. Future work will extend the proposed framework by incorporating imperfect CSI, residual SIC, hardware impairments, and dynamic satellite mobility. Robust RSMA transmission strategies, adaptive beam tracking, and low-complexity power allocation algorithms will also be investigated to improve the applicability of the proposed framework to practical satellite-terrestrial deployments.