ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations ∗ Work done during the internship at Kuaishou Technology. † Corresponding author. ♠ Project Leader.

2025

Read ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations ∗ Work done during the internship at Kuaishou Technology. † Corresponding author. ♠ Project Leader. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23121.txt.

Introduction

Chatbots that see images often lose the thread over long conversations, forgetting what was said and shown turns ago. Yiming Lei and seven co-authors introduce ContextQFormer, a module that gives such models a memory block for carrying context forward in multi-turn, multi-modal dialogue. To train and test it, they also built TMDialog, a dataset of longer multi-turn conversations mixing text and images.

Against three baseline systems on their dataset, the new module raised the available rate, a measure of usable responses, by two to four points. The authors plan to release the dataset openly to support further work. The paper appears in 2025, from a team partly based at Kuaishou Technology.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23121

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations ∗ Work done during the internship at Kuaishou Technology. † Corresponding author. ♠ Project Leader. Abstract Multi-modal large language models have demonstrated remarkable zero-shot abilities and powerful image-understanding capabilities. However, the existing open-source multi-modal models suffer from the weak capability of multi-turn interaction, especially for long contexts. To address the issue, we first introduce a context modeling module, termed ContextQFormer, which utilizes a memory block to enhance the presentation of contextual information. Furthermore, to facilitate further research, we carefully build a new multi- t urn multi- m odal dialog ue dataset (TMDialog) for pre-training, instruction-tuning, and evaluation, which will be open-sourced lately. Compared with other multi-modal dialogue datasets, TMDialog contains longer conversations, which supports the research of multi-turn multi-modal dialogue. In addition, ContextQFormer is compared with three baselines on TMDialog and experimental results illustrate that ContextQFormer achieves an improvement of 2%-4% in available rate over baselines. Keywords: Multimodality, Large Language Model, Multi-Turn Conversation ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations Yiming Lei 1,2∗ Zhizheng Yang 3∗ 1 1 footnotemark: 1 Zeming Liu 1† Haitao Leng 4♠ Shaoguo Liu 4 Tingting Gao 4 Qingjie Liu 1,2† 2 2 footnotemark: 2 Yunhong Wang 1 1 Beihang University, China 2 Hangzhou Innovation Institute, Beihang University, China 3 Nanjing University, China 4 Kuaishou Technology, China [email protected] , [email protected] , [email protected] Abstract content 1. Introduction L arge l anguage m odels (LLMs) lie at the heart of natural language processing and artificial intelligence OpenAI 2023 ; Xu et al. 2023 ; Tu et al. 2023 ; Singhal et al. 2023 thanks to their astonishing capabilities in natural language generation. To further expand the capabilities of large language models, multi-modal models are developed to incorporate various types of input beyond text. For example, GPT-4 OpenAI 2023 , MiniGPT-4 Zhu et al. 2023 , and BLIP-2 Li et al. 2023 demonstrate astonishing zero-shot capabilities and powerful image understanding and generation abilities. Figure 1: A comparison of open-sourced methods and the proposed ContextQFormer. VisualGLM cannot recognize multiple images, and mPLUG-owl shows serious hallucinations. In spite of the promising performance, existing open-sourced multi-modal LLMs are constrained by the maximum input length and the computational complexity of self-attention Wang et al. 2020 . Although some models Press et al. 2021 ; Tay et al. 2020 are capable of processing long inputs, they may still struggle to capture crucial contextual information in exceptionally lengthy texts. As demonstrated in Figure 2 , previous multi-modal LLMs may miss out on essential context from preceding text because of the accumulation of historical noise, which refers to irrelevant or outdated information that can hinder comprehension. To alleviate the issue, we propose ContextQFormer, which enables a better understanding of long context. In ContextQFormer, a memory block is introduced to store core multi-modal information and the context embedding is fused with the memory block to obtain a better context embedding. Specifically, images and text are represented with the representation of the special token [CLS] by VIT Dehghani et al. 2023 and Roberta Liu et al. 2019 , respectively. Then, these representations are stored as memory and fused with context embedding to obtain a better context embedding, thus alleviating the issue. However, there is no publicly available long context dataset for train and evaluation of multi-modal LLMs. The existing multi-modal datasets have a limited long conversation. Consequently, the absence of long multi-modal datasets hinders further research. To address the issue, we carefully build a new multi- t urn multi- m odal dialog ue datasets (TMDialog) for pre-training, instruction-tuning, and evaluation, respectively. For the data generation, demonstration examples, image descriptions, and instructions are concatenated as the input to GPT-4. To reduce hallucinations, GPT-3.5 is utilized to recheck the output. Statistics show that dialogues in TMDialog are much longer, with more relevant questions and answers, than other datasets. To facilitate model comparison, we conduct bench-marking experiments on the dataset. Specifically, experiments are conducted on mPLUG-owl Ye et al. 2023 , visualGLM Du et al. 2022 , LoRA-only Hu et al. 2021 , and ContextQFormer. Five evaluation metrics are utilized in this work, including rationality, information, hallucination, safety, and available rate. Experimental results on the dataset show that ContextQFormer achieves state-of-the-art performance and demonstrates a strong capability of long context understanding. Our contributions can be highlighted in three parts: • We propose TMDialog, a large-scale multi-modal multi-turn dialogue dataset, where each sample contains a long context with multi-modal. The datasets contain sub-data for pretraining, instruction-tuning, and evaluation. • We build benchmarking baselines on the dataset and propose a novel long context understanding method ContextQFormer to better understand long context effectively. • Experimental results show the effectiveness of ContextQFormer and demonstrate the importance of long context. 2. Related Work 2.1. Multi-Modal Large Language Models While large language models(LLMs) have achieved significant success in natural language processing, other modalities such as vision and audio in combination with LLMs are still in the exploratory stage. Existing fusion methods for multi-modal LLMs can generally be categorized into three different paradigms: system collaboration, end-to-end trained model, and alignment of other modalities with LLMs. LLMs, such as ChatGPT, possess a strong ability to understand instructions and have been proven to be powerful tools in enhancing multi-modal understanding through collaboration with other expert systems. System collaboration, such as HuggingGPT Shen et al. 2023 and MM-REACT Yang et al. 2023 , demonstrates how ChatGPT functions as an agent and selects an appropriate expert system to perform various different tasks. However, this method is constrained by the capabilities of expert systems, lacks flexibility, and is susceptible to the influence of prompts. Alternatively, there are end-to-end methods designed for training multi-modal models directly. For example, Flamingo Alayrac et al. 2022 freezes the pre-trained vision encoder and large language model and fuses vision and language with cross-attention. It is rumored that GPT-4 is also trained in this way. However, this method requires amounts of data and resources which may not be feasible. Recent research has focused on aligning different modalities with LLMs. For example, MiniGPT-4 Zhu et al. 2023 uses a fully connected layer to align the visual encoder and LLM, enabling impressive visual understanding capabilities. BLIP-2 Li et al. 2023 design Q-Former to align the visual features from the visual encoder and LLM. mPLUG-owl Ye et al. 2023 is a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module specialized modeling for multiple rounds of conversations. However, these methods do not specifically address multi-turn dialogue and may lack the ability to handle long conversations. 2.2. Multi-Turn Language Models Multi-turn conversation modeling has always been an urgent challenge in natural language processing. While models with fewer parameters may not process the same degree of instruction comprehension as large language models, there have been numerous solutions available before the emergence of large language models. Broadly speaking, there are two main approaches to addressing long conversations: the long-range attention mechanism and the memory mechanism. Recently, significant progress has been made in developing efficient long-range attention mechanisms. For example, Transformer-XL Dai et al. 2019 incorporates a segment-level recurrence mechanism and a novel positional encoding scheme that enables

Read the complete text