Sea surface suspended concentrations prediction based on pre-trained transformer Authors: Tao Su, Leilei Wang, Yu Song, Zhongli Bai Published: 2025 ABSTRACT AbstractSea surface suspended sediments (SSSC) monitoring is an important branch of marine engineering, which plays an important role in port transportation, marine habitat ecology, and the environment. However, understanding and monitoring the dynamics of suspended sediments remain challenging due to complex natural and anthropogenic interactions. Existing approaches, such as using suit data or satellite data to construct numerical models, face limitations in sampling irregular spaces or lacking generalization capabilities. Besides, unlike natural language processing (NLP) and computer vision (CV), the amount of suit data or satellite data is small, which make it difficult for some data-driven deep models to achieve excellent performance. To address these challenges, we propose a pre-trained transformer model that leverages an NLP pre-trained model trained on billions of tokens for SSSC reconstruction. Specifically, we design input embedding layers to project remote sensing image data into dimensions required by a specific pre-trained model. During the fine-tuning phase, we freeze the self-attention and feed-forward layers of the residual block in the pre-trained model, allowing knowledge transfer to the SSSC task. Experimental results demonstrate that our proposed pre-trained transformer models achieve state-of-the-art performance on the SSSC task. This result may indicate that parameters trained on massive data in large models have the potential to enhance performance on other tasks.Keywords: Deep learning, Sea surface suspended sediments, TransformerSubject terms: Electrical and electronic engineering, Computer science Introduction Seabed sediment concentration prediction is an important field and plays an important role in seabed engineering1,2, global water hazards3, and ecological environment monitoring4. The main component of seabed sediment is sand, which is mainly produced by tides and waves, but is also produced by trawling and marine development5–7, such as the extraction of oil and natural gas, the installation of submarine cables, tunnels, offshore power generation, etc. Suspended sediment in the water body absorbs heavy metals, total nitrogen, total phosphorus, and other pollutants8,9. This suspended sediment not only affects the optical properties of the water body, such as transparency, turbidity, and color, but also influences the reproduction and growth of aquatic life, especially algae10,11. Furthermore, the accumulation of sediment in large quantities can lead to water hazards and have adverse effects on river management, biological habitats, and the ecological environment12. Therefore, it is crucial to predict the accumulation of sea surface suspended sediment13,14. In past decades, researchers have traditionally relied on turbidity meters to collect suit samples for sea surface suspended sediments concentration (SSSC) in coastal and estuarine environments. However, this approach is time-consuming and limited in spatial resolution. With the development of the satellite remote sensing technology, satellite remote sensing has emerged as a valuable tool for monitoring SSSC. Satellite remote sensing offers numerous advantages over traditional suit sampling methods, including extensive spatial coverage, high spatial resolution, and cost-effectiveness. The ocean color satellite sensors such as SeaWiFS, MODIS, MERIS, and GOCI are specifically designed with spectral bands optimized for observing bio-optical properties in aquatic environments. Notably, these sensors with higher spatial resolution15 can effectively monitor suspended particulate matter (SPM) concentrations in areas characterized by higher SSSC variability. In recent years, an increasing number of researchers have turned to remote sensing satellite data for their studies16–18. For example, the Landsat series of remote sensing satellite data sets are widely used due to their public accessibility, high spatial resolution (30 meters) and spanning several decades (from the 1970s to the present). Researchers focus on reconstructing of sea surface suspended sediments concentration (SSSC) using irregular spatiotemporal and potentially scarce ocean remote sensing data. They have studied some approaches to evaluate the sediment dynamics in coastal areas. These methods can be categorized into two types: traditional mathematical modeling approaches19–22 and data-driven algorithmic models23–25. The traditional SSSC prediction method mainly involves point sampling to obtain the sediment concentration from the continental source to the continental margin and establish a suitable numerical model to predict it. For example, Qiu et al.19 proposed an empirical model for estimating SSSC in the Yellow River estuary, which utilized Landsat 8 Operational Land Imager (L8/OLI) band ratio B4/B3 to tune the SSSC retrieval mode. According to the study conducted by Li et al.20, it was observed that low turbidity water bodies exhibit a strong positive correlation between SPM and Remote Sensing Reflectance (Rrs) in the green and red bands. Additionally, in high turbidity water bodies, the correlation between SPM and Rrs in the Near-Infrared (NIR) band was found to be further enhanced. Building upon these findings, Li et al. developed a three-band SSSC retrieval model. Zhou et al.21 utilized the backscattering coefficient (bb532) from satellite Remote Sensing Reflectance (RSR) to represent the turbidity and obtained the final concentration of surface suspended sediments. These mathematical statistical models used for remote sensing monitoring of SSSC typically assume a specific functional relationship between the uncertainty of water body information and remote sensing information. Then, these models estimate the parameters by transforming them into multivariate linear functions22. However, these approaches often require the sample population to conform to specific distribution characteristics, such as normal distribution. In practice, the spatiotemporal autocorrelation of water parameters may not necessarily meet these assumptions, which may result in lower prediction accuracy, poor generalization performance of the model and limited application. Nowadays, deep learning methods have shown remarkable success in computer vision (CV) and natural language processing (NLP). As a result, some researchers have started embracing deep learning techniques in the field of satellite remote sensing. By leveraging the power of deep learning, they aim to enhance the analysis and interpretation of satellite remote sensing data, enabling more accurate and efficient extraction of valuable information from these datasets. The application of deep learning in satellite remote sensing has the potential to significantly improve various tasks23–29, including image classification, object detection, land cover mapping, and change detection. For instance, Wei et al.30 employ a quantum bee colony algorithm based on Boltzmann’s selection and dimensional dynamic update strategy to optimize artificial intelligence (AI) models (BDQABC-ANN) and exhibited stronger predictive capabilities for SSSC. Xiaoming et al.31 utilized generative adversarial network (GAN) to generate more samples and employed deep convolutional neural networks (CNNs) to predict seabed sediment classification. In the study conducted by Wei et al.32, it was observed that the shallow CNNs were often more effective in extracting relevant features and achieving accurate predictions and outperformed the deep CNN architecture when dealing with small datasets. Although these deep learning methods have achieved excellent performance on satellite remote sensing data, however, we can find that due to the small sample size of the dataset, it is difficult to train an effective model to complete the corresponding task. To address these issues, this paper proposes a pre-trained Transformer model, which utilizes parameters in the NLP pre-trained Transformer model from billions of tokens to facilitate model training for downstream tasks, especially in few shot tasks for SSSC. Specifically, we design input embedding layers to project remote sensing image data into dimensions required by a specific pre-trained model. During the fine-tuning phase, we freeze the self-attention and feed-forward layers of the residual block in the pre-trained model, allowing knowledge transfer to the SSSC task. Then, we conduct experimental verification using the Bay of Biscay dataset, which involves natural movements and artificial operations of sediment generation and can evaluate the performance of the SSSC model effectively. Finally, the results demonstrate that our proposed methods can achieve state-of-the-art performance on the SSSC task. The layout of this paper is arranged as follows. In section II, we introduce the Bay of Biscay dataset. Section III describe the proposed pre-trained Transformer framework in detail. In section IV, we explain the experiment results. Section V analyzes and discusses the experiment result. This paper is concluded in Section VI. Materials The study aimed to predict the sea surface suspended sediments concentration for small dataset. In particular, the Bay of Biscay is selected as the areas to investigate SSSC. The Bay of Biscay is located in the northeastern part of the North Atlantic Ocean, bordering France to the east and Spain to the south. It covers an area of 194,000 km, a total volume of about 332,000 cubic kilometers, an average depth of 1715 m, and a maximum depth of 5120 m. Sediment fluxes at the Bay of Biscay are often affected by natural and anthropogenic interactions such as sea breezes, tides, trawling and fishing. Sediments mainly include three different types, which can be classified according to their mean grain size, including mud (m ), sand (from 63 to m), and gravel (mm ). In this work, we mainly consider the first two, adding the concentrations of mud and sand to obtain the total suspended sediment concentration at the sea surface (in kg m ). The suit data were collected using a Nortek profiler and turbidity meter installed on a bottom mooring at a depth of 23 m33. The site is located near the southern coast of Brittany (at one station of latitude and longitude ), close to the city of ”Le Croisic”. The measurements were conducted from November 25, 2007, to January 31, 2008. Additionally, simultaneous SSC measurements were performed by the AWAC profiler, covering the entire water column over a 20-meter range. The AWAC profiler was calibrated with a turbidity meter. The satellite data34 were derived from the MERIS satellite sensor dataset available from 2007 to 2011 and daily sampled. To optimize the utilization of the dataset, we have followed the approach employed in previous studies35 and utilized the MARS-MUSTANG model to simulate oceanic sedimentary fluxes. As shown in Fig. 1, we introduce the relationship between the bathymetry and the suspended sediment dynamics. Specifically, Fig. 1a illustrates the isobaths of the Bay of Biscay, providing a visual representation of the depth contours in the study area. This information is essential as bathymetry plays a crucial role in influencing the circulation patterns and sediment transport processes within the bay. Furthermore, Fig. 1b shows the spatial distribution of the mean SSSC (Sea Surface Sediment Concentration) within the study area. (a) Displays the bathymetry of the Bay of Biscay using black lines to represent the isobaths at depths of 40, 70, 100, and 130 m. (b) We present the mean spatial distribution of the log10 SSSC (g/L) derived from the MARS-MUSTANG hydro sedimentary model. In particular, the remote sensor can only work in the near visible range, and it is impossible to observe the sea surface under cloudy conditions, which will result in a part of the data being missing. To better reconstruct the sea surface field based on observation data, we use the Observing System Simulation Experiment (OSSE)36 as a benchmark experimental framework. This remote sensor dataset contains 1436 samples, and some data are partially missing due to weather conditions. After data preprocessing, the total number of the training set are 895, and the number of the valid set are 1, and the number of the test data set are 100. Methods Deep learning (DL) techniques have gained wide popularity in CV and NLP due to their impressive performance, especially the large language model (LLM), like GPT237, SAM38. Researchers have also started using DL methods to tackle the problem of Sea Surface Suspended Concentration (SSSC) and have achieved remarkable results. Unlike CV and NLP, where abundant data is available to train complex models capable of performing various tasks, SSSC datasets are limited, making it challenging to achieve excellent performance. To overcome this challenge, we leverage a pre-trained Transformer model (GPT2), originally trained on billions of NLP tokens, and adapt it for SSSC tasks. Specifically, we utilize its NLP-derived parameters without retraining on new tokens, focusing instead on fine-tuning with our small dataset. This approach, termed Pretrained Transformer, transfers knowledge across domains efficiently. Model structure In this section, we present the framework of pretrained transformer models, as illustrated in Fig. 2. Our approach involves leveraging parameters from NLP pretrained transformer models for sea surface suspended sediments concentration (SSSC) prediction. Specifically, the architecture retains the self-attention and feedforward layers of the residual blocks from the pre-trained model. Then, we construct an input embedding layer and normalization to ensure that the SSSC data is suitable for the input of the transformer model. During the fine-tuning process, we specifically fine-tune the position embedding and layer normalization layers. This allows these components to adapt to the specific SSSC task. By fine-tuning these layers, the model can better capture the positional and contextual information relevant to the SSSC prediction. Model architecture for the sea suspended sediment concentration (SSSC) tasks, where pre-trained parameters are transferred. The self-attention and feedforward layers in the transformer blocks are frozen, while only the embedding layer, normalization layers, and output layer require training. Transformer Transformer39 has been widely used and has achieved excellent performance in both CV and NLP, especially when applied to some large models. Here, we focus on the principle of multi-head attention mechanisms. As shown in Fig. 3, the Scaled Dot-Product Attention consists of queries and keys with a dimension of , and values with a dimension of . It involves computing the dot products between the query and all the keys, and each dot product is divided by the square root of . Subsequently, a softmax function is applied to obtain weights assigned to the corresponding values. Importantly, the attention function is performed simultaneously on a set of queries, which are organized into a matrix Q. Similarly, the keys and values are arranged into matrices K and V, respectively. Scaled dot-product attention. The matrix of outputs is computed as follows: The multi-head attention mechanism has been shown to offer superior performance compared to the single-head attention mechanism, as Fig. 4. It facilitates the learning of distinct representations of queries, keys, and values by utilizing separate linear projections. These different projections enable the model to capture diverse aspects of the input data. In the multi-head attention mechanism, scaled dot product attention is applied in parallel to the distinct sets of queries, keys, and values generated from the linear projections. This simultaneous processing of multiple perspectives or contexts enhances the model’s ability to capture comprehensive information from the input. Multi-head attention. The resulting output values from each attention head are then concatenated and projected to obtain the final output. By leveraging different linear projections and performing scaled dot product attention in parallel, multi-head attention enables the model to capture a richer and more nuanced understanding of the input, leading to improved performance in various tasks. Specifically, the multi-head attention mechanism can be expressed by the following formula: where, , the projections matrices , , , , and h is the attention layers. In the transformer model, each layer consists of a fully connected feed-forward network (FFN) that is applied independently to every location in the input. This network comprises two linear transformations, with a Rectified Linear Unit (ReLU) activation function applied between them. The inclusion of this feed-forward network (FFN) allows the model to adapt to specific features and dependencies present at different locations in the data. This capability enhances the model’s ability to learn and represent complex data features, consequently contributing to the improvement of its overall capabilities. The matrix of outputs is computed as follows: Pretrained transformer Building complex models with excellent performance often requires a large amount of training data to mitigate problems such as underfitting or overfitting. However, compared to domains like CV or NLP, the availability of data is often limited in the field of remote sensing. When dealing with small remote sensing datasets, training complex models with a large number of parameters can be challenging. The limited data may not sufficiently capture the underlying patterns or provide enough variation for the model to generalize well. As a result, the performance of these models may suffer. To address this issue, this paper employs transfer learning strategy. Transfer learning involves leveraging pre-trained models on larger datasets from related domains (e.g., CV or NLP) and fine-tuning them on the remote sensing data. This approach allows the model to benefit from the knowledge learned from abundant data sources. Hence, we propose a pretrained transformer model to deal with SSSC prediction. Specifically, to effectively utilize transformer models in remote sensing data, several steps can be taken, including Input Embedding, Normalization, Patching, Fine-tuning. 1. Input embedding When applying NLP pre-trained models to remote sensing tasks, it becomes necessary to redesign and train the input embedding layer to transform the raw remote sensing data into a suitable input format for the transformer model. This step ensures that the model can effectively process and comprehend the unique characteristics of remote sensing data. To project remote sensing image data into the dimensions required by a specific pre-trained model, linear probing can be used. Linear probing involves adding a linear layer on top of the pre-trained model’s embedding layer. This linear layer reduces the dimensionality of the remote sensing image data and aligns it with the dimensions expected by the pre-trained model. By using linear probing, the number of parameters required for training is reduced. This reduction is particularly advantageous when working with limited remote sensing data, as it helps mitigate the risk of overfitting and enhances the model’s ability to generalize to unseen examples. The sea surface field dataset we used includes 1430 days of image data with a size of 6464. The data are first normalized using instance normalization to standardize scales across features. Patching is applied to divide each 6464 sample into 16 patches of size 1616, converting spatial data into a sequence of patches. This is inspired by Vision Transformers (ViT) to retain local spatial information. We use Token Embedding, Positional Embedding, and Temporal Embedding to embed patch features, patch positions, and times, respectively, ensuring that the output feature dimensions conform to the 768 embedding dimension expected by the GPT-2 model. The output is a sequence of embedding vectors fed directly into the frozen Transformer blocks. This process ensures compatibility with the pre-trained architecture while minimizing trainable parameters. 2. Normalization The data normalization block is a valuable step in ensuring consistent and standardized scales in the input data for remote sensing tasks. This normalization block plays a crucial role in facilitating knowledge transfer and improving the model’s ability to learn effectively. The normalization process involves computing the mean and variance of the input time series data. By normalizing the data using these statistics, the different features or channels of the remote sensing data are treated equally during training. This ensures that no particular feature dominates the learning process due to differences in scale or magnitude. Normalizing the data helps to mitigate issues related to varying scales, improves convergence during training, and allows the model to learn more efficiently. Moreover, it aids in reducing the impact of outliers and noise in the data, ultimately enhancing the robustness of the model. 3. Patching Vision Transformer (VIT)40 employs patching methods to capture local spatial information within images effectively and achieves excellent performance. Hence, we also incorporate patching to convert spatial remote sensing data into sequence patches, enabling the transformer model to operate on sequential inputs and retain spatial information. Patching allows the transformer model to handle larger input history timeframes while maintaining a consistent token length. By dividing the input image into smaller patches, the model can capture local details and spatial relationships effectively. Patching helps reduce information redundancy by allowing the model to process smaller spatial regions individually, reducing computational complexity and memory requirements. 4. Fine-tuning Fine-tuning on specific remote sensing data follows an initial pre-training or transfer learning phase using large datasets from related fields. This step involves training a transformer model on a remote sensing dataset to adapt it to the specific features and patterns present in the data. Fine-tuning helps model specialization and achieve better performance on remote sensing tasks. Specially, we utilize two strategies to help model to transfer learning in new remote sensing data. Frozen pretrained block: we choose to retain the position embedding layer and self-attention block in the pre-trained model in our architecture. These components contain extensive knowledge learned from pretrained language models. To preserve this knowledge and prevent it from being overwritten during fine-tuning, you decide to freeze the self-attention block. This approach ensures that the pre-trained knowledge captured in these blocks remains intact and the model can effectively utilize the learned attention mechanism. Freezing the self-attention module while fine-tuning helps to strike a balance between leveraging pre-trained knowledge and allowing the model to be specialized for remote sensing tasks. It enables the model to benefit from the rich representation learned in the pre-training stage while adapting to the specific characteristics of remote sensing data during fine-tuning. Positional embeddings and layer normalization: we choose to fine-tune the position embedding and layer normalization layers during the fine-tuning process, which allows these components to adapt to specific downstream tasks and new patterns present in the data. By fine-tuning the position embedding layer, the model can learn task-specific position information relevant to the target task. This is especially important in situations where the position of an element in the input sequence is important to the current task. Similarly, fine-tuning layers and normalization layers allow the model to adapt the normalization parameters to the specific data distributions and patterns observed in downstream tasks. This fine-tuning process enables the model to better capture and exploit the statistical properties of the data during training. By combining fine-tuning of position embeddings and layer normalization layers, pre-trained knowledge can be effectively exploited while allowing these components to adapt to the specific requirements and patterns of the target task. This approach has the potential to enhance model performance on downstream tasks by tailoring these components to the specific data distributions and patterns encountered in new tasks. Evaluation metric In this work, to evaluate the performance of different methods, we adopt global root mean square error (RMSE) and global reconstruction score (R-score). These metrics provide insights into the method’s capture of the real data and variance rate. Descriptions of these evaluation metrics are as follows: Root mean square error (RMSE): RMSE is a widely used metric to evaluate the accuracy of interpolation methods. It measures the square root of the average squared difference between the predicted and true values. A lower RMSE indicates higher accuracy because it represents the overall difference between the predicted and actual values. where is the prediction of model, the is the true result, and n is the number of samples. Reconstruction score (R-score): the R-score is a metric that evaluates the rate of variance captured by a method. It quantifies the extent to which the predicted values preserve the variability and patterns present in the real data. A higher R score indicates that the interpolation method captures a greater proportion of the true data variance, meaning that the original pattern is better preserved. where SSR is used to estimate the error from the mean, reflecting the sum of squared deviations of the correlation between the independent variable and the dependent variable. SSE is the error between the estimated value and the true value, which reflects the degree of model fitting. By considering RMSE and R-score, we can evaluate the performance of different methods. The RMSE metric focuses on the accuracy of the predicted values, while the R-score provides insights into the method’s ability to capture variance and patterns in real data. These evaluation metrics help to compare different methods and determine which method performs better in terms of accuracy and data variability preservation. Results Our experimental results are summarized as follows. Specifically, we verified it with different methods on the same dataset to demonstrate the effectiveness of our proposed method. Computational setup The configuration of the workstation we employ in these experiments are as follows: CPU: i5-12600KF, RAM: 64G, GPU: RTX 4070s 12G. We utilized the python language to run all programs, and pytorch framework was used to run all deep learning methods. We use the pre-trained model GPT2(3)41 for experiments, where 3 represents the use of the first 3 layers. The patch size is set to 16, so that each 6464 sample can be divided into 16. The learning rate is set to 0.0001 and the batch size is set to 48. Dropout is set to 0.2, and L2 regularization is set to 0.005 to avoid overfitting. Specifically, the early stopping strategy is used to stop the training phase and avoid overfitting, where a counter is used to stop the training process after ten epochs when no loss degradation is observed on the valid set. Metrics and evaluation To effectively evaluate the effectiveness of our proposed method, different methods for SSSC prediction are also compared in this study, including optimal interpolation (OI), EOF-based interpolation (DinEOF), analog data assimilation (AnDA) and variation-based Neural network based on the sub-formula (4DVarNet). Here’s a brief overview of these methods: Optimal interpolation (OI)42: optimal interpolation is a widely used data interpolation technique that combines observations with a background field to produce an optimal analysis. It minimizes the difference between observations and the analysis while considering the background error covariance. OI is often employed for spatial interpolation of data, including SSSC. EOF-based interpolation (DinEOF)43: EOF-based interpolation utilizes Empirical Orthogonal Functions (EOFs) to represent the dominant modes of spatial variability in the data. By projecting the data onto the EOFs, missing or sparse data can be interpolated based on the spatial patterns captured by these modes. Analog data assimilation (AnDA)44: simulated data assimilation refers to the process of assimilating simulated or synthetic observations into a model to improve its state estimation. This approach allows for assessing the performance and sensitivity of data assimilation methods under controlled conditions. 4DVarNet35: 4DVarNet is a variation-based neural network method that combines the principles of four-dimensional variational data assimilation (4DVar) and deep learning. It formulates the data interpolation problem as an optimization problem and utilizes neural networks to learn the mapping between the observations and the analyzed state. Each method has its own approach and assumptions and comparing them helps to assess the strengths and limitations of different interpolation and data assimilation techniques. The different interpolation methods are evaluated according to two standard metrics: the global root mean square error (RMSE) and a global reconstruction score (R-score), in terms of rate of variance of the true data captured by the interpolation. Results In Table 1, we show the average RMSE and R-score values of different methods on the dataset in this experiment. From the results, we can find that OI performs the worst among all tested methods with an RMSE of 0.176. There is only a small improvement using DINEOF and AnDA, 0.167 and 0.162 respectively. Compared with the three traditional methods, the three deep learning methods have shown greater improvements, with the RMSEs of AE-4DVarNet and GE-4DVarNet being 0.142 and 0.104 respectively. After full fine-tuning, RMSE = 0.095 and R-score = 96.8, which are lower than our method (RMSE = 0.054, R-score = 98.4). This confirms that freezing the self-attention/feedforward layer can prevent overfitting and preserve pre-training knowledge, which is especially important for small datasets. Table 1RMSE and R-score results for all methods over the SSSC validation period.ModelsMSER-scoreOI0.17690.4DinEOF0.16791.3AnDA0.16291.9AE-4DVarNet0.14293.7GE-4DVarNet0.10496.6GPT2(3)0.09596.8Pretrained transformer GPT2(3)0.05498.4 RMSE and R-score results for all methods over the SSSC validation period. To further illustrate the reconstruction of the baseline scheme of the SSSC, we visualize the ability of different methods to retrieve the frontal structure of the SSSC field in Fig.5. From these results, we can find that, compared with other methods, the pre-trained transformer GPT2(3) model we proposed has better performance and reconstructs the main frontier structure of the real scene. Specifically, we can know that most methods are adept at effectively restoring the major boundary edges and may struggle to precisely reconstruct contours at different concentrations. However, our method demonstrates the ability to approximately restore a consistent contour trend at different concentrations. This capability is of great significance when it comes to accurately assessing SSSC at high-precision spatial scales. Reconstruction of SSSC gradient fields: reconstruction obtained by OI, AnDA, DINEOF, AE-4DVarNet, GE-4DVarNet, and pretrained transformer on test data. Furthermore, the daily time series of RMSE for different methods is visualized in Fig. 6, aiming to evaluate the models’ temporal capabilities. Notably, the OI and DinEOF methods exhibit elevated RMSE values during November (Fig. 6). This occurrence can be attributed to the increased cloud cover during the winter season, resulting in a higher rate of data loss in the later period. Additionally, the impact of stronger winds, waves, and rainfall during winter significantly affects sediment concentrations. The dynamic fluctuations in sediment concentration, combined with the substantial amount of missing data, offer a plausible explanation for the observed peak in RMSE during November. From Fig.6, it is evident that our proposed GPT2(3) method exhibits less sensitivity to cloud coverage compared to other methods. The GPT2(3) consistently maintains a small RMSE, indicating its proficiency in dynamically reconstructing changes in sediment concentration. In contrast, other methods display larger RMSE values that gradually increase over time. Moreover, these methods encounter challenges in addressing the issues posed by missing data and temporal variations. The capacity of our proposed pre-trained transformer GPT2(3) method to mitigate the impact of such fluctuations enhances its ability to accurately restore the spatiotemporal characteristics of the global SSSC field. Daily reconstructed RMSE of SSSC in different models. Reconstruction of time patterns Besides spatial pattern reconstruction, we conduct further analysis on the ability of the interpolation field to capture the temporal characteristics of SSSC variations. To assess this, we compared the time series of the true SSSC with the interpolated time series at three selected stations, namely SaO, Gas, and Con, as shown in Fig. 7. Through comprehensive evaluation, our results demonstrate that our proposed pre-trained transformer GPT2(3) method achieves the highest correlation with the true values, providing evidence of its superior performance. These results may indicate that the robustness and efficacy pre-trained transformer GPT2(3) in capturing both spatial and temporal characteristics of SSSC variations. The method not only achieves accurate spatial pattern reconstruction but also successfully captures important temporal dynamics. Time series of the SSSC reconstruction for different models in three selected stations: SaO, Gas, and Con. The green series refers to the true state. We ablated the impact of model depth by varying the number of GPT-2 layers used (2, 3, 6, and 12) as shown in Table 2. The results indicate that 3 layers achieve the optimal trade-off, with an RMSE of 0.054 and R-score of 98.4%. Shallower models (2 layers) underfit (RMSE=0.089), while deeper models (6 and 12 layers) overfit (RMSE=0.071 and 0.083, respectively) due to increased parameter complexity on a small dataset. Table 2RMSE and R-score of GPT-2 backbone across different layer configurations.ModelsMSER-R-scoreGPT2(2)0.08997.4GPT2(3)0.05498.4GPT2(6)0.07197.6GPT2(12)0.08397.2 RMSE and R-score of GPT-2 backbone across different layer configurations. Discussion The study aims to evaluate the effectiveness of different models for retrieving gap-free series of SSSC fields from irregularly sampled satellite measurements. From these results may indicate that deep learning methods have significant improvements compared to traditional methods. This may be because traditional methods rely on EOF decomposition to study SSS dynamics, resulting in relatively poor retrieval capabilities. Deep learning methods, using a data-driven approach, have better data representation capabilities, thereby achieving better performance. Moreover, compared with different methods, our proposed pre-trained transformer GPT2(3) method achieve state-of-the-art performance on the SSSC task. It can be found from the results that compared with the traditional OI method, the average relative gain of the log10(SSSC) value of our proposed pre-trained transformer model is 70% in terms of RMSE. These findings provide evidence for the potential of leveraging parameters from large models trained on billions of tokens to aid in various tasks. Furthermore, the employed freezing strategy demonstrates its efficacy in harnessing the capabilities of the large model to address related tasks. Simultaneously, fine-tuning enables the model to acquire knowledge about the specific characteristics of remote sensing data. Despite overall strong performance, the model struggles in two key scenarios: (1) regions with extreme sediment concentrations (e.g., sudden spikes from storm events), where the model tends to underestimate peaks due to smoothing effects; and (2) areas with prolonged cloud cover, where missing data leads to increased uncertainty in reconstruction. These cases highlight the need for incorporating meteorological data and robust gap-filling techniques in future work. Conclusion In this study, we address the problem of data-driven interpolation of satellite-derived SSSC fields. The OSSE setting in the Bay of Biscay is employed to valid our proposed pre-training transformer GPT2(3). Extensive experiment results demonstrate that our proposed method can significantly improve the reconstruction of SSSC dynamics when compared to the traditional Optimal Interpolation (OI) method. Specifically, we achieve a relative gain of up to 70% in terms of root mean square error (RMSE). Besides, this research also reveals the superior performance of large models in various downstream tasks, highlighting the versatility of Transformer models. In the future, building on these findings, we aim to further enhance the performance of our method by leveraging parameter-efficient fine-tuning techniques, thereby enabling its more effective utilization in SSSC-related tasks. Author contributions T.S.: Participated in the research by assisting in certain aspects of data collection, analysis and model construction. L.W.: played an important role in organizing and guiding the research, led the research direction, and was responsible for coordinating and collaborating with other authors to promote the progress of the whole research project. He was deeply involved in the key aspects of model design and experimental protocol development, and also played a key role in the integration of the research results and the writing of the paper. Y.S.: Participate in the research process and provide support in model training, data analysis and experimental validation. Z.B.: Participated in the research process, worked with other authors to solve the problem of predicting suspended sediment concentration at the sea surface, and contributed to the data processing or model optimization. Data availability The datasets used and/or analysed during the current study available from the corresponding author on reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.