SEMT: Static-Expansion-Mesh Transformer Network Architecture for Remote Sensing Image Captioning
Khang Truong, Lam Pham, Hieu Tang, Jasmin Lampert, Martin Boyer, Son Phan, Truong Nguyen · 2025
Read SEMT: Static-Expansion-Mesh Transformer Network Architecture for Remote Sensing Image Captioning by Khang Truong, Lam Pham, Hieu Tang, Jasmin Lampert, Martin Boyer, Son Phan, Truong Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12845.txt.
Introduction
Seven researchers wrote a computer vision paper in two thousand twenty-five on writing captions for satellite images. Captions turn complex overhead imagery into plain text. That helps with tasks such as watching the environment, judging disaster damage, and planning cities, where image volumes are huge.
Their network, SEMT, is built on the transformer design used in modern language models. It combines three ideas: a static expansion step, attention with a memory store, and a mesh form of the transformer. They tested the models on two standard caption data sets. Their best model beat prior systems on most scoring measures, which they take as a sign of real-world promise.
Catalog record
| CREATOR | Khang Truong, Lam Pham, Hieu Tang, Jasmin Lampert, Martin Boyer, Son Phan, Truong Nguyen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-12845
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SEMT: Static-Expansion-Mesh Transformer Network Architecture for Remote Sensing Image Captioning Khang Truong, Lam Pham, Hieu Tang, Jasmin Lampert, Martin Boyer, Son Phan, Truong Nguyen Abstract Image captioning has emerged as a crucial task in the intersection of computer vision and natural language processing, enabling automated generation of descriptive text from visual content. In the context of remote sensing, image captioning plays a significant role in interpreting vast and complex satellite imagery, aiding applications such as environmental monitoring, disaster assessment, and urban planning. This motivates us, in this paper, to present a transformer based network architecture for remote sensing image captioning (RSIC) in which multiple techniques of Static Expansion, Memory-Augmented Self-Attention, Mesh Transformer are evaluated and integrated. We evaluate our proposed models using two benchmark remote sensing image datasets of UCM-Caption and NWPU-Caption. Our best model outperforms the state-of-the-art systems on most of evaluation metrics, which demonstrates potential to apply for real-life remote sensing image systems. Items — vision language, remote sensing image, image captioning, transformer. I INTRODUCTION The rapid growth of remote sensing technologies has resulted in the accumulation of vast amounts of satellite imagery, offering opportunities for Earth observation and analysis. However, the effective interpretation and extracting meaningful insights from remote sensing images remain a challenging task. Remote sensing image captioning, which aims to automatically generate descriptive textual information from images, has emerged as a promising solution to bridge the gap between visual data and human understanding. In recent years, significant progress has been made in the field of remote sensing image captioning, particularly through the development of deep learning models with transformer-based architectures. Indeed, various deep learning models such as [ 1 ] , [ 2 ] , [ 3 ] , [ 4 ] , [ 5 ] , [ 6 ] , [ 7 ] , [ 8 ] , [ 9 ] , [ 10 ] , [ 11 ] have been proposed and demonstrated remarkable success in the task of remote sensing image captioning (RSIC). These network architectures can be separated into two main groups. The first group [ 2 , 3 , 4 ] utilized pre-trained models that were trained on large-scale image datasets. These models are then finetuned on target datasets of remote sensing images. In other words, this approach leverages transfer learning techniques to reduce training time and the cost of network construction, but still achieves the potential and competitive models. However, this approach results in high model complexity, as it leverages part or all of entire large pre-trained models. Instead of leveraging the pre-trained models, the second group [ 5 , 6 , 7 , 8 , 9 , 11 ] focuses on exploring advances of network architecture that shows potential to improve the model performance. For examples, the proposed model in [ 5 ] explored a CNN-based network in the encoder component to capture local and global features of remote sensing images. Similarly, a CNN-based architecture was proposed in [ 9 ] to capture multi-scale contextual features. Meanwhile, [ 8 , 7 , 11 ] made efforts to construct innovative attention layers to effectively capture distinct features from remote sensing images. Although these recently published models proposed for the RSIC task have achieved promising performance, these models mainly were constructed basing on encoder-decoder architectures, especially traditional transformer-based networks. This motivates us to evaluate recently innovative transformer architectures that is potential to further improve the RSIC task performance. In other words, we are inspired by the second approach, focusing on exploring innovative and advanced network architecture to construct an effective model for RSIC task. In particular, three techniques of Mesh Transformer [ 12 ] (Mesh Trans.), Memory-Augmented Attention (Mem. Att.) [ 12 ] and Static Expansion (Stat. Exp.) [ 13 ] are evaluated and integrated to construct our proposed models. The proposed models are evaluated on two benchmark datasets of UCM-Caption [ 14 ] and NWPU-Caption [ 1 ] , and then compared with the state-of-the-art models. II Proposed Transformer Based Network Architecture Fig. 1: The high-level architecture of the proposed models with mesh-based transformer architecture The high-level architecture of the proposed networks is shown in the Fig. 1 that includes 4 main components: CNN-based Backbone, Word Embedding, Encoder, and Decoder. In particular, the input images \mathbf{X} are first fed into the CNN-based Backbone to extract image feature maps \mathbf{E_{1}} . Then, the image feature maps go through the Encoder component, generating \{\mathbf{E_{2},E_{3},....E_{I}}\} as the outputs of encoder blocks in the Encoder component. Meanwhile, the input captions are fed into the Word Embedding to extract word embeddings \mathbf{D_{1}} . The word embeddings \mathbf{D_{1}} then go through the Decoder component. As the proposed model is based on the Mesh Transformer architecture [ 12 ] , all outputs from encoder blocks in the Encoder component and the output of CNN-based Backbone, \{\mathbf{E_{1},E_{2},....E_{I}}\} , are sent to all decoder blocks in the Decoder component. The output of each decoder block in Decoder component is denoted by \mathbf{D_{i}} . TABLE I: Hyper-parameters set for the proposed models Parameters Values Training epoch number 20 Learning rate 10^{-4} Learning rate decay ratio (per epoch) 0.95 Batch size 500 The longest length of captions ( L ) 53 Word embedding dimension ( E ) 768 Number of encoder blocks 4 Number of decoder blocks 4 II-A Word Embedding As shown in Fig. 1 , the Word Embedding component receives the captions (e.g., each caption is an array of discrete words), each of caption presents the fixed length L which is the length of the longest caption in the evaluating dataset. Then, the captions are fed into the positional encoding, generating word feature maps each of which is denoted by \mathbf{D_{1}}\in\mathbb{R}^{L{\times}E} , where E is the dimension of embedding vectors representing for words. In other words, the positional encoding is used to transform discrete words into numerical vectors with E dimension. Finally, the word feature maps are fed into the Decoder component. II-B CNN-based Backbone In this paper we evaluate various CNN-based Backbone of VGG16, MobileNet-V2, Resnet152, Inception, and EfficientNetB2 architectures. To construct these CNN-based Backbone, these pre-trained models of VGG16, MobileNet-V2, Resnet152, Inception, and EfficientNetB2, which were trained on ImageNet dataset [ 15 ] , are leveraged. Then, dense layers of these pre-trained models are removed to obtain the CNN-base Backbones. The CNN-based backbones are used to transform an input image \mathbf{X}\in\mathbb{R}^{H{\times}W{\times}C} into an image feature map of \mathbf{X_{f}}\in\mathbb{R}^{H_{f}{\times}W_{f}{\times}C_{f}} , where H,W,C are height, width, channel of the original image and H_{f},W_{f},C_{f} are height, width, channel of the image feature map. Then, the H_{f} and W_{f} dimensions are flatten to generate a feature map of \mathbf{E_{1}}\in\mathbb{R}^{F_{f}{\times}C_{f}} before feeding into the Encoder component, where F_{f}=W_{f}.H_{f} . TABLE II: Performance comparison between our proposed models and the state-of-the-art systems on NWPU-Caption dataset Models BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L GLCM [ 5 ] 0.554 0.423 0.335 0.272 0.279 0.504 SAT [ 6 ] 0.734 0.612 0.528 0.469 0.337 0.601 SM-Att [ 7 ] 0.739 0.616 0.534 0.469 0.338 0.595 Struc-Att [ 8 ] 0.744 0.609 0.519 0.456 0.309 0.606 MC-Net [ 9 ] 0.741 0.626 0.544 0.478 0.347 0.611 MLCA-NET [ 1 ] 0.745 0.624 0.541 0.478 0.337 0.601 P-to-H [ 11 ] 0.757 0.629 0.546 0.483 0.319 0.586