An Enhanced Large Language Model For Cross Modal Query Understanding System Using DL-KeyBERT Based CAZSSCL-MPGPT
Shreya Singh · 2025
Read An Enhanced Large Language Model For Cross Modal Query Understanding System Using DL-KeyBERT Based CAZSSCL-MPGPT by Shreya Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17000.txt.
Introduction
Shreya Singh presents an enhanced language model system for answering questions that span images and text, in a 2025 paper. Cross-modal systems must take in a picture and a written query together. A known weakness is the echo chamber effect, where repeated visual patterns drag the model toward stale answers and hurt accuracy.
The pipeline first cleans the inputs, cuts objects out of the image with a detection network, and builds skeletons and a knowledge graph from them. Features are drawn from the graph, skeletons, and objects, and an optimization routine keeps the best ones. Key terms from the text are embedded by a keyword model. A custom transformer then blends both streams to produce the answer. The author reports accuracy above ninety-nine percent on one standard image set and above ninety-eight on a question set.
Catalog record
| CREATOR | Shreya Singh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-17000
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Enhanced Large Language Model For Cross Modal Query Understanding System Using DL-KeyBERT Based CAZSSCL-MPGPT Shreya Singh Abstract Large Language Models (LLMs) are advanced deep-learning models designed to understand and generate human language. They work together with models that process data like images, enabling cross-modal understanding. However, none of the existing works concentrated on avoiding the echo chamber effect in the image to enhance the accuracy of the system. Thus, the proposed system considered this limitation and developed an enhanced LLM-based framework for cross-modal query understanding using DL-KeyBERT-based CAZSSCL-MPGPT. Firstly, the collected dataset consists of images and texts. Initially, the images are pre-processed. These preprocessed images then undergo object segmentation using Easom-You Only Look Once (E-YOLO). Then, the object skeletons are generated, and a knowledge graph is constructed using a Conditional Random Knowledge Graph (CRKG). After that, features are extracted from the knowledge graph, generated skeletons, and segmented objects. The optimal features are then selected using the Fossa Optimization Algorithm (FOA). Meanwhile, the text undergoes word embedding using DL-KeyBERT. Finally, the cross-modal query understanding system utilizes CAZSSCL-MPGPT to generate accurate and contextually relevant image descriptions as text. The proposed CAZSSCL-MPGPT achieved an accuracy of 99.14187362% in the COCO dataset 2017 and 98.43224393% in the vqav2-val dataset. keywords Cross Attention Layer and Zero-Shot Semantic Consistency Learning-based Mixup Phish Generative Pre-trained Transformer (CAZSSCL-MPGPT), Damerau-Levenshtein-based KeyBidirectional Encoder for Word Representations (DL-KeyBERT), Large Language Model, Cross-Modal Query Understand, Image Captioning, Echo chamber effect, Visual Question Answering (VQA). 1 Introduction LLMs have significantly impacted Artificial Intelligence (AI) in recent years, enabling machines to understand and generate human-like text (Wang et al., 2024). These models are widely applied across various domains, including Natural Language Processing (NLP), and powering conversational agents like chatbots (Fan et al., 2024). Prominent examples include the Generative Pre-trained Transformer (GPT) family, such as ChatGPT and GPT-4, which are well-known for their strong performance and wide range of applications (Messina et al., 2021). Multimodal Large Language Models (MLLMs) extend the capabilities of LLMs by integrating both textual and visual data for tasks, such as cross-modal retrieval and VQA (Chen et al., 2024) (Chen et al., 2021). One key area of the MLLMs application is cross-modal queries, which involve retrieving information across different modalities, such as text and images (Liu et al., 2023) (Xu et al., 2024). For instance, in cross-modal retrieval, queries and results are represented as vectors in a common space. Also, similarity functions are used to identify relevant matches (Kalyan, 2024). Another prominent application of this approach is VQA, where an AI system answers questions about images (Lu et al., 2023). Other tasks like image captioning and image retrieval also fall under Visio-linguistic tasks that integrate visual and textual information (Zhang et al., 2021) (Chen et al., 2021a). These advancements are essential in applications, such as search engines, enabling accurate retrieval of relevant data across modalities (Liu et al., 2024). Recent advancements in VQA and other cross-modal tasks include methods like Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM, and Graph Convolutional Networks (GCNs) (Salaberria et al., 2023) (Ye et al., 2024). Despite these advancements, challenges like difficulty in capturing long-term dependencies and scalability issues occur when processing complex data (Tingting et al., 2023). Also, none of the existing works concentrated on avoiding the echo chamber effect that arose from the images. To mitigate this issue, a DL-KeyBERT-based CAZSSCL-MPGPT framework is proposed, enhancing performance in cross-modal tasks. 1.1 Problem Statement The existing research methodologies for cross-modal query understanding systems faced several limitations, which are outlined below, • Existing Multi-Modal Query Understanding systems, such as VQA and Image Captioning, did not address the echo chamber effect in image processing. This effect arose from over-reliance on frequent patterns in the training data, which impacted accuracy. • The existing (Cheng et al., 2021) failed to capture the intricate semantic relationships and dependencies between modalities. As a result, the existing work compromised the contextual significance of each data type, limiting the improvement of the LLM. • In the existing work (Zhang et al., 2024), the dynamic alignment between multiple modalities was not analyzed and unseen data was not classified correctly, limiting the improvement of the LLM. • The prevailing (Li et al., 2024) struggled with captioning highly detailed or crowded images, which often included numerous objects, scenes, or backgrounds. It faced difficulty in highlighting the most relevant features of such images. • Most of the Multi-Modal Query Processing Systems used unprocessed images, leading to misleading content due to excessive noise. 1.2 Objectives The objectives of the proposed work for understanding cross-modal queries are listed below. • To address the echo chamber effect in images and improve accuracy, the proposed solution involves constructing a knowledge graph using CRKG. This approach mitigates the model’s overreliance on frequent patterns present in the training data. • To improve LLM by capturing intricate semantic relationships and dependencies between modalities, DL-KeyBERT is utilized. • The analysis of dynamic alignment between multiple modalities and the accurate classification of unseen data is achieved by introducing the cross-attention layer and zero-shot semantic consistency learning (CAZSSCL) to enhance the LLM. • An E-YOLO-based object detection technique is used to better analyze complex image structures, enhancing LLM performance. • The issue of misleading content caused by unprocessed images is solved by processing the image using an MF and PG-CLAHE to reduce noise and enhance image quality for improved LLM accuracy. The remaining structure of the paper is organized as follows: Section 2 discusses the related works; Section 3 presents the proposed methodology; Section 4 reveals the results and provides a discussion; and Section 5 concludes the paper with future work. 2 Literature Survey (Cheng et al., 2021) deployed a deep semantic alignment network in Remote Sensing (RS) for cross-modal image-text retrieval. The Semantic Alignment Module used attention and gate mechanisms to optimize features, enhancing image-text retrieval. The model achieved improved performance in image-text matching. However, it struggled to capture intricate semantic relationships and dependencies between modalities, which compromised the contextual significance of each data type. (Zhang et al., 2024) developed EarthGPT, a universal MLLM for multi-sensor image understanding in the RS domain. EarthGPT integrated a visual-enhanced perception mechanism to refine both coarse and fine visual information. It used cross-modal comprehension and unified instruction tuning, improving RS visual interpretation across sensor data. However, the model failed to classify unseen images that it had never encountered during the training phase. It also struggled to effectively analyze the dynamic alignment between multiple modalities, such as text and image. (Li et al., 2024) presented an interactive perception network for LLMs. The framework extracted global and fine-grained image features using the Contrastive Language–Image Pre-training (CLIP). The Request-based Visual Information-seeking module enabled dynamic interaction, allowing LLMs to generate effective responses. Thus, the model allowed LLMs to incorporate the desired visual information for various human queries. Nevertheless, the model struggled with captioning images by highlighting the most relevant or interesting features due to the presence of multiple objects, scenes, or backgrounds presented in the crowded or detailed images. (Lim et al., 2024) introduced a framework for Unification, Retrieval, and Generation in multimodal question answering using pre-trained language models. This framework utilized a Large Language and Vision Assistant (LLaVA) to generate detailed image descriptions.