Towards General Continuous Memory for Vision-Language Models

Wenyi Wu, Zixuan Song, Kun Zhou, Yifei Shao, Zhiting Hu, Biwei HuangUniversity of California, San Diego. [email protected] · 2025

Read Towards General Continuous Memory for Vision-Language Models by Wenyi Wu, Zixuan Song, Kun Zhou, Yifei Shao, Zhiting Hu, Biwei HuangUniversity of California, San Diego. [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17670.txt.

Introduction

Wenyi Wu and five co-authors give vision-language models a compact continuous memory, in a paper from 2025. Such models still struggle with hard reasoning that needs broad world knowledge in several languages and modes. Storing that knowledge as long token strings bloats context and can hurt results.

Their key idea lets the model itself encode knowledge into a few dense vectors. A light tuning method needs only a tiny share of model weights and a small self-made training set. The memory packs varied knowledge into just eight vectors, and the main model stays frozen, so the module plugs in as needed. Gains held across eight reasoning benchmarks.

Catalog record

CREATORWenyi Wu, Zixuan Song, Kun Zhou, Yifei Shao, Zhiting Hu, Biwei HuangUniversity of California, San Diego. [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17670

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards General Continuous Memory for Vision-Language Models Wenyi Wu, Zixuan Song, Kun Zhou, Yifei Shao, Zhiting Hu, Biwei HuangUniversity of California, San Diego. [email protected] Abstract Language models (LMs) and their extension, vision-language models (VLMs), have achieved remarkable performance across various tasks. However, they still struggle with complex reasoning tasks that require multimodal or multilingual real world knowledge. To support such capabilities, an external memory system that can efficiently provide relevant multimodal information is essential. Existing approaches generally concatenate image and text tokens into a long sequence as memory, which, however, may drastically increase context length and even degrade performance. In contrast, we propose using continuous memory-a compact set of dense embeddings-to more effectively and efficiently represent multimodal and multilingual knowledge. Our key insight is that a VLM can serve as its own continuous memory encoder. We empirically show that this design improves performance on complex multimodal reasoning tasks. Building on this, we introduce a data-efficient and parameter-efficient method to fine-tune the VLM into a memory encoder, requiring only 1.2% of the model’s parameters and a small corpus of 15.6K self-synthesized samples. Our approach CoMEM utilizes VLM’s original capabilities to encode arbitrary multimodal and multilingual knowledge into just 8 continuous embeddings. Since the inference-time VLM remains frozen, our memory module is plug-and-play and can be flexibly integrated as needed. Extensive experiments across eight multimodal reasoning benchmarks demonstrate the effectiveness of our approach. Code and data is publicly released here https://github.com/WenyiWU0111/CoMEM/tree/main . 1 Introduction Through large-scale training, language models (LMs) ( brown2020language ; touvron2023llama ) have demonstrated remarkable performance across diverse real-world tasks. LMs even surpass human capabilities in language reasoning tasks ( dasgupta2022language ) such as mathematical problem solving ( ahn2024large ) , commonsense reasoning ( wei2022chain ) , and code synthesis ( jiang2024survey ) . However, when confronted with complex reasoning tasks that demand multimodal or multilingual world knowledge, both LMs and their vision-language model (VLM) extensions continue to face significant challenges ( wen2025token ) , primarily due to insufficient world knowledge representation. Inspired by how humans offload facts, plans, and ideas to external repositories like notebooks or databases for on-demand access, it is promising to develop a general external memory 1 1 1 In this paper, external memory denotes any detachable module or function that supplies the knowledge without changing LM parameters, in contrast to internal memory that embeds knowledge by modifying parameters. that contains useful world knowledge for augmenting VLMs ( lewis2020rag ; wang2023longmem ) . Early approaches directly concatenate the collected useful information into a long token sequence, and feeds it into VLMs ( lewis2020rag ; guu2020realm ) e.g., retrieval-augmented generation (RAG) methods. However, multimodal representations demand significantly more input tokens ( e.g., 8 to 11427 tokens per image in Qwen2.5-VL ( bai2025qwen2.5-vl ) ). Thus, simple concatenation would greatly increase the input length, making it difficult for the memory content to be used ( yu2024visrag ) (see the degradation in performance shown in Table. 1 after using RAG). To solve the token overload issue, token pruning methods have been proposed to remove unimportant in-context tokens ( chen2024image ; zhang2024sparsevlm ) . However, token pruning generally leads to incomplete contextual contents, which impedes the VLM’s ability to accurately understand and utilize the compressed information ( wen2025token ) . Compared to discrete tokens, continuous embeddings naturally have stronger representation capability for complex data ( elhage2021mathematical ; olsson2022incontext ; hao2024training ) . This advantage makes them particularly promising for memory encoder architectures designed to condense multimodal information into continuous representations. However, training such encoders faces two key challenges: (1) achieving generalizable compression ability across diverse multimodal inputs, and (2) maintaining semantic alignment with the VLM ( jaegle2021perceiver ) . While large-scale training can improve performance, it greatly increases the training cost and becomes heavily sensitive to the training data distribution. For example, when dominated by simple cases or a single domain, the encoder tends to overfit and degrade generalization performance. ( arpit2017closer ) ( grangier-iter-2022-trade ) . In this paper, we focus on efficiently training a general continuous memory encoder to effectively supply multimodal knowledge for VLMs. To avoid costly training for semantic alignment, it is essential to minimize the representation gap between the memory encoder and downstream VLMs before training. Therefore, a natural way is to use the VLM itself as the memory encoder. Our empirical study confirms that the VLM can serve as a memory encoder for itself without any additional training. Benefiting from the stacked self-attention mechanism, its generated continuous embeddings in each layer have already aggregated rich semantic information ( vaswani2017attention ) ( clark2019bert ) . As shown in Fig. 1 , a simple rule-based embedding selection strategy for constructing the memory can greatly boost the performance of VLMs in complex multimodal multilingual reasoning tasks. Based on our empirical findings, we propose a data-efficient and parameter-efficient training recipe to further improve the compressor rate and adaptation performance of the VLM-based general continuous memory encoder. Concretely, we only need to fine-tune the low-rank adaptation matrices (LoRA) ( hu2021lora ) in the VLM-based memory encoder, and a lightweight Q-Former ( li2023blip2 ) for further compressing the VLM representations into only eight embeddings, 1.2% parameters in total. In terms of data, we only need the VLM itself to synthesize 15.6k samples for training. This efficient training strategy enables our continuous memory to reuse the original ability of the VLM, to effectively encode multimodal and multilingual knowledge. Since we do not need to train the inference VLM, our memory is plug-and-play and can be flexibly integrated with the VLM when necessary. To demonstrate the effectiveness of our approach, we apply our method to state-of-the-art VLMs, and evaluate the performance across eight visual reasoning benchmarks. For six English visual reasoning benchmarks, our method achieves an average improvement of +8.0% on Qwen2-Instruct-VL and +7.7% on Qwen2.5-Instruct-VL. On two multilingual multimodal benchmarks, our approach further improves performance by +5.1% and +4.3% on Qwen2-Instruct-VL and Qwen2.5-Instruct-VL, respectively. Furthermore, our adaptation study results also indicate the transferability of our VLM-based memory encoder to improve LMs in visual reasoning tasks. The long context understanding study also exhibits the stable and superior performance of our method. 2 Empirical Analysis with VLM as Memory Encoder In this section, we conduct an empirical study to examine: (1) whether VLM can serve as a continuous memory encoder to compress multimodal information into compatible embeddings. (2) whether a few embeddings from the VLM can preserve key information to improve multimodal reasoning tasks. 2.1 Analysis Setup For the empirical study, we conduct experiments on two state-of-the-art VLMs, i.e., Qwen2-VL-7B and Qwen2.5-VL-7B, and test the performance on three multimodal reasoning benchmarks. Evaluation Settings. To compare the effectiveness of different memory and context compression methods, we select three benchmarks: InfoSeek ( chen2023can ) , OK-VQA ( marino2019ok ) , and A-OKVQA ( schwenk2022okvqa ) . These benchmarks contain complex visual questions that require both accurate visual entity identification and multi-step reasoning to derive the correct answer. Following existing work ( caffagni2024wikillava ) ( qi2024rora ) , for each question, we utilize CLIP-based retriever ( radford2021learning ) to collect relevant top-10 multimodal knowledge items from a Wikipedia-based source dataset WiT ( srinivasan2021wit ) to construct the input data for the memory. Table 1: Comparison

Read the complete text