Semantic-Guided Natural Language and Visual Fusion for Cross-Modal Interaction Based on Tiny Object Detection

Xian-Hong Huang, Hui-Kai Su, Chi-Chia Sun, Jun-Wei Hsieh · 2025

Read Semantic-Guided Natural Language and Visual Fusion for Cross-Modal Interaction Based on Tiny Object Detection by Xian-Hong Huang, Hui-Kai Su, Chi-Chia Sun, Jun-Wei Hsieh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05474.txt.

Introduction

Xian-Hong Huang and co-authors join language and vision tools to spot tiny objects named in text, for cross-modal interaction. Small objects are the hard case for detection, since they hold few pixels and blur into clutter.

Their method pairs a BERT language model with a fusion pyramid network for vision, with backbone designs that sharpen feature extraction. Word forms are cleaned by lemmatization, and fine-tuning aligns text cues with image traits. The aim is a system where a user can name a small target in plain words and have it found with care.

Catalog record

CREATORXian-Hong Huang, Hui-Kai Su, Chi-Chia Sun, Jun-Wei Hsieh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05474

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic-Guided Natural Language and Visual Fusion for Cross-Modal Interaction Based on Tiny Object Detection Thanks: Xian-Hong Huang and Hui-Kai Su are with Department of Electrical Engineering, National Formosa University, Taiwan; Chi-Chia Sun is with Department of Electrical Engineering, National Taipei University, Taiwan; Jun-Wei Hsieh is with College of Artificial Intelligence and Green Energy, National Yang Ming Chiao Tung University, Taiwan; Corresponding Author is Xian-Hong Huang ( E-mail: [email protected] ) Xian-Hong Huang, Hui-Kai Su, Chi-Chia Sun, Jun-Wei Hsieh Abstract This paper introduces a cutting-edge approach to cross-modal interaction for tiny object detection by combining semantic-guided natural language processing with advanced visual recognition backbones. The proposed method integrates the BERT language model with the CNN-based Parallel Residual Bi-Fusion Feature Pyramid Network (PRB-FPN-Net), incorporating innovative backbone architectures such as ELAN, MSP, and CSP to optimize feature extraction and fusion. By employing lemmatization and fine-tuning techniques, the system aligns semantic cues from textual inputs with visual features, enhancing detection precision for small and complex objects. Experimental validation using the COCO and Objects365 datasets demonstrates that the model achieves superior performance. On the COCO2017 validation set, it attains a 52.6% average precision (AP), outperforming YOLO-World significantly while maintaining half the parameter consumption of Transformer-based models like GLIP. Several test on different of backbones such ELAN, MSP, and CSP further enable efficient handling of multi-scale objects, ensuring scalability and robustness in resource-constrained environments. This study underscores the potential of integrating natural language understanding with advanced backbone architectures, setting new benchmarks in object detection accuracy, efficiency, and adaptability to real-world challenges. I Introduction Object detection has been the main downstream of the image processing task. Significant advancements in object detection have been accomplished through the development of deep neural networks, including RCNN [ 10 , 27 ] , FPN [ 19 ] , SSD [ 22 ] , and YOLO [ 1 , 34 , 36 , 33 ] . Despite their success, most of this method focuses on the single modality image that has limited information such as a fixed vocabulary, e.g., 80 categories in the COCO [ 20 ] dataset. Single modality image information with fixed categories lead to the limited the ability and applicability in the open real-world problems. Recent works [ 9 , 11 , 29 ] have explored the prevalent vision-language models [ 13 , 25 ] to address open vocabulary detection through distilling vocabulary knowledge from language encoders, e.g., BERT [ 16 ] . However, beside of the open vocabulary, one of the critical challenges in computer vision is accurate recognition and localization of small objects within images. Small object detection is inherently difficult due to the limited pixel information and the influence of surrounding noise [ 2 , 21 ] . Existing object detection models often struggle with small objects, leading to suboptimal performance in applications such as surveillance, autonomous driving, and medical imaging [ 24 ] . Recent studies have explored cross-modal interactions to enhance small object detection by leveraging semantic information from natural language descriptions [ 37 , 31 ] . The integration of NLP and CV allows models to utilize contextual cues from textual data to improve visual perception tasks [ 17 ] . Transformer-based architectures, such as BERT for NLP [ 16 ] and Vision Transformers for CV [ 8 ] , have shown remarkable success in capturing long-range dependencies and contextual relationships within data, making them suitable for cross-modal applications [ 30 , 23 ] . In this work, we propose a novel approach that combines the BERT language model with the Parallel Residual Bi-Fusion Feature Pyramid Network (PRB-FPN-Net) for enhanced small object detection through semantic-guided cross-modal interaction. Our method leverages the strengths of BERT in understanding natural language to extract semantic cues, which are then integrated with visual features extracted by PRB-FPN-Net, a CNN-based model optimized for small object detection [ 3 ] . By fine-tuning both models and applying lemmatization techniques using WordNetLemmatizer [ 12 ] , we transform extracted words into their root forms to enhance the model’s comprehension of natural language information. Our contributions can be summarized as follows: • Semantic-Guided Fusion: We introduce a cross-modal fusion strategy that effectively combines semantic information from text descriptions with visual features, improving the detection of small objects that are often overlooked by traditional models. • Enhanced Text Encoding: By implementing advanced tokenization and lemmatization techniques, we optimize the text encoding process, enabling more accurate alignment between textual and visual modalities. • Improved Performance: Experimental results on benchmark datasets, including COCO [ 20 ] and Objects365 [ 28 ] , demonstrate that our model outperforms state-of-the-art methods in object detection accuracy and efficiency, particularly in resource-constrained scenarios. The remainder of this paper is organized as follows. Section 2 reviews related work in cross-modal interaction and small object detection. Section 3 details our proposed method, including the integration of BERT and PRB-FPN-Net models. Section 4 presents experimental results and comparisons with existing methods. Finally, Section 5 concludes the paper and discusses future research directions. II Related Work II-A Conventional Object Detection Traditional object detection approaches generally focus on predefined categories, as illustrated by datasets like COCO [ 20 ] and Object365 [ 28 ] , which contain fixed sets of 80 and 365 categories, respectively. This rigid categorization limits flexibility, as models trained on such datasets may struggle to generalize to unseen objects. Detection approaches are typically divided into region-based and pixel-based methods. Among region-based approaches, Faster R-CNN [ 27 ] introduced a two-stage framework in which a Region Proposal Network (RPN) first generates candidate regions, followed by RoI classification and regression, significantly improving detection accuracy over earlier models like R-CNN [ 10 ] . Single-stage models like Single Shot Detector (SSD) [ 22 ] , You Only Look Once (YOLO) [ 26 ] , and Fully Convolutional One-Stage Object Detector (FCOS) [ 32 ] have since been developed to improve inference speed, enabling real-time object detection. SSD and YOLO rely on predefined anchor boxes and grid-based predictions, while FCOS adopts an anchor-free approach, allowing for more flexible object localization without reliance on anchors. The YOLO series has seen significant architectural improvements. For instance, cross-stage partial networks (CSPNet) [ 35 ] were introduced to reduce computational redundancy, partitioning feature maps to balance feature reuse and efficiency. This concept was extended by mixed-stage partial (MSP) networks [ 4 ] , which combine inter-stage and intra-stage connections to enhance feature propagation. CSPNet and MSP have been incorporated into YOLO versions like YOLOv4 [ 1 ] and YOLOv5 [ 14 ] , resulting in models that are both accurate and computationally efficient. More recently, YOLOv7 [ 34 ] and YOLOv10 [ 33 ] introduced innovations in feature aggregation and a non-maximum suppression (NMS)-free approach, allowing for more accurate detection without traditional NMS, enhancing both precision and inference speed. These modifications support YOLO’s use in demanding applications, such as autonomous driving and surveillance, where both accuracy and efficiency are crucial. II-B Visual-Language Model In recent years, there has been a growing trend towards developing vision-and-language models that tackle visual recognition problems by training vision models with free-form language supervision. This approach enables models to better understand and interpret images within a broader, open-vocabulary context. Notable examples include CLIP [

Read the complete text