Global Context Compression with Interleaved Vision-Text Transformation

Dian Jiao * † , Jiaxin Duan * , Shuai Zhao , Jiabing Leng , Yiran Zhang , Feng Huang ‡ China Electronics Cloud Technology Co., Ltd. {jiaodian, duanjiaxin, zhaoshuai, lengjiabing, zhangyiran, huangfeng01}@cestc.cn · 2026

Read Global Context Compression with Interleaved Vision-Text Transformation by Dian Jiao * † , Jiaxin Duan * , Shuai Zhao , Jiabing Leng , Yiran Zhang , Feng Huang ‡ China Electronics Cloud Technology Co., Ltd. {jiaodian, duanjiaxin, zhaoshuai, lengjiabing, zhangyiran, huangfeng01}@cestc.cn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10378.txt.

Introduction

Dian Jiao and co authors pack long text into images to save tokens. New sight and text models can read text from a page image with low loss. That opens a trick. Render text as an image, and the model needs far fewer tokens to take it in.

Fewer tokens cut the cost of attention, which grows fast with length. The team mix sight and text steps in one stream. They call it global context compression. Long files can then fit in one run. The aim is long reads at a cost labs can bear.

Catalog record

CREATORDian Jiao * † , Jiaxin Duan * , Shuai Zhao , Jiabing Leng , Yiran Zhang , Feng Huang ‡ China Electronics Cloud Technology Co., Ltd. {jiaodian, duanjiaxin, zhaoshuai, lengjiabing, zhangyiran, huangfeng01}@cestc.cn
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10378

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Global Context Compression with Interleaved Vision-Text Transformation Dian Jiao * † , Jiaxin Duan * , Shuai Zhao , Jiabing Leng , Yiran Zhang , Feng Huang ‡ China Electronics Cloud Technology Co., Ltd. {jiaodian, duanjiaxin, zhaoshuai, lengjiabing, zhangyiran, huangfeng01}@cestc.cn Abstract Recent achievements of vision-language models in end-to-end OCR point to a new avenue for low-loss compression of textual information. This motivates earlier works that render the Transformer’s input into images for prefilling, which effectively reduces the number of tokens through visual encoding, thereby alleviating the quadratically increased Attention computations. However, this partial compression fails to save computational or memory costs at token-by-token inference. In this paper, we investigate global context compression, which saves tokens at both prefilling and inference stages. Consequently, we propose VIST2, a novel Transformer that interleaves input text chunks alongside their visual encoding, while depending exclusively on visual tokens in the pre-context to predict the next text token distribution. Around this idea, we render text chunks into sketch images and train VIST2 in multiple stages, starting from curriculum-scheduled pretraining for optical language modeling, followed by modal-interleaved instruction tuning. We conduct extensive experiments using VIST2 families scaled from 0.6B to 8B to explore the training recipe and hyperparameters. With a 4 × \times compression ratio, the resulting models demonstrate significant superiority over baselines on long writing tasks, achieving, on average, a 3 × \times speedup in first-token generation, 77% reduction in memory usage, and 74% reduction in FLOPS. Our codes and datasets will be public to support further studies. 1 1 footnotetext: Equal contributions. 2 2 footnotetext: Project leader. 3 3 footnotetext: Corresponding author. 1 Introduction Large language models (LLMs) with Transformer architecture face significant challenges in context scaling because the complexity of self-attention increases quadratically ( \mathcal{O}(n^{2}) ) with sequence length. This yields an urgent need for context compression that can reduce computing costs without sacrificing model performance. Existing approaches for context compression are sparse attention and hierarchical encoding . Following informatics theory, sparse attention 12 ; 3 ; 17 drops out detected tokens with marginal information to reduce the Attention operators. In contrast, hierarchical encoding 4 ; 16 (in Figure 1 ) splits a long text into ordered chunks, where each is compressed into densely informative representations. By preserving the complete context, hierarchical encoding effectively prevents information loss, thereby earning widespread interest. Recently, the emerging vision-language models (VLMs), such as dots.ocr 14 , MinerU-VLM 23 , and DeepSeek-OCR 25 demonstrate remarkable performance in optical character recognition (OCR). The key behind their success - optical compression 13 opened a new door for a more promising hierarchical encoding. For example, Glyph 4 renders millions of tokens into images and achieved a 4 \times lossless compression based on a powerful visual encoder. Figure 1: The illustration of context compression of the Transformer. Right arrow indicates transforming a text chunk to its latent representation. Despite these achievements, the mentioned approaches equip LLMs with partial rather than global context compression. Taking the human-AI conversation in Figure 1 as an example, human queries are compressed in prefilling, while the AI responses are generated token-by-token without any compression. In the case of long-text generation, the model performs intensive computations during training and requires significant memory to store KV-Caches during inference, resulting in considerable costs. To address this problem, we conduct a pioneering exploration for global context compression, aiming to save tokens at both prefilling and inference stages. Specifically, we propose VIST2, a novel Transformer that interleaves text chunks and their optical encoding in input, and predicts the next text token conditioned on visual tokens in pre-context. This interleaved visual-text transformation, named Optical Language Modeling (OLM), effectively bridges the gap between partial and global compressions but is often overlooked in existing works. In experiments, VIST2 is implemented using Qwen3 and SigLip2 connected by a linear projection layer. We pretrain this model through several stages, starting with image captioning and OCR to warm up the visual modules, followed by the OLM to adjust the backbone LLM. We then fine-tune it on conversations with long queries and responses, regularized by a modal-interleaved chat template. Extensive results show that our VIST2, with a compression ratio of 4:1, achieves a 3 \times speedup in the first-token delay while also reducing memory usage and FLOPS during inference by over 75% each. 2 Background and Problem Definition This work focuses on text-to-text generation - the general task form of modern linguistic intelligence, which facilitates human-machine interaction through conversations. Given a long input text \mathbf{X}=[x_{1},x_{2},...,x_{L}] with the target \mathbf{Y}=[y_{1},y_{2},...,y_{M}] , the standard Decoder-only LLMs, e.g., GPT, Llama, Qwen, etc, aim to model the conditional probabilities via next-token prediction: P(\mathbf{Y}|\mathbf{X})\to\sum_{i}^{M}\mathcal{P}_{\theta}(y_{i}|\mathbf{X},y_{<i}) (1) In the settings where L and M exceed the effective context window of standard transformers, the computational complexity increases quadratically, resulting in intensive costs that hinder the training of large language models (LLMs). Partial Context Compression (PCC) methods address this problem by partitioning a long input \mathbf{X} into n continuous chunks \{\mathcal{C}_{i}\}_{i=1}^{n} , where each chunk \mathcal{C}_{i}=[x^{i}_{1},...,x^{i}_{k}] contains k tokens. Additionally, a text renderer \mathcal{R}(\cdot) is employed to render each chunk \mathcal{C}_{i} into a grayscale optical image \mathcal{V}_{i}\in\mathbb{R}^{H\times W\times 3} . Consequently, they convert the causal language modeling objective in Eq. 1 into visual-language modeling: P(\mathbf{Y}|\mathbf{X})\to\sum_{i}^{M}\mathcal{P}_{\theta}(y_{i}|\{\mathbf{v}_{k}\}_{k=1}^{L},y_{<i}) (2) where \mathbf{v}_{k}=VisualEncoder(\mathcal{V}_{i}) are visual tokens derived from the chunk \mathcal{C}_{i} . Although PCC extends the context window of Transformers by up to 10 times wider, the visual compression of text tokens works only for prefilling without the support of inference. As a result, PCC enables long-text understanding (LTU), while still facing challenges in long-text generation (LTG), such as storytelling, novel writing, and complicated multi-step reasoning. To address this limitation, we propose global context compression , which enables both LTU and LTG through interleaved text-vision compression and vision-text modeling. 3 Method: VIST2 VIST2 is an efficient large language model (LLM) architecture that achieves global context compression by performing iterative visual-text transformations. The core idea behind VIST2 is to utilize the spatial redundancy of rendered text to compress information into a dense visual latent space. Figure 2: The illustration of pre-training. 3.1 Model Architecture As illustrated in Figure 1, VIST2 features a sandwich architecture, a popular design in VLMs, comprising a visual encoder (VE) and an LLM backbone connected by a modal aligner. Visual Encoder \mathcal{E}(\cdot) processes images rendered from structure-free texts into token embeddings. For this reason, we utilize a pretrained Vision Transformer with the patches sized 16 \times 16 (ViT-L/16), which yields m=\lfloor\frac{H}{16}\rfloor\times\lfloor\frac{W}{16}\rfloor visual tokens per image: \tilde{\mathbf{v}}_{i}=\mathcal{E}(\mathcal{\hat{V}}_{i})\in\mathbb{R}^{m\times d_{v}} Modal Aligner \mathcal{M}(\cdot) aligns the outputs of VE with the LLM’s embedding space, which is achieved through a multilayer perceptron: \hat{\mathbf{v}}_{i}=\tanh(\tilde{\mathbf{v}}_{i}W_{m}+b_{m})\in\mathbb{R}^{m\times d_{lm}} where W_{m}\in\mathbb{R}^{d_{v}\times d_{lm}} and b_{m}\in\mathbb{R}^{d_{lm}} are trainable parameters. Sparse Attention Mechanism . The final input to the LLM is an interleaved sequence: \mathbf{H}=[\hat{\mathbf{c}}_{1},\hat{\mathbf{v}}_{1},\hat{\mathbf{c}}_{2},\hat{\mathbf{v}}_{2},...,\hat{\mathbf{v}}_{n-1},\hat{\mathbf{c}}_{n}] where \hat{\mathbf{c}}_{i} are token embeddings of \hat{\mathcal{C}}_{i} . To achieve GCC, we implement a sparse causal attention that constrains token visibility and saves computations. The attention mask is illustrated in Figure 2 (lower), where each token’s visibility follows: \text{Mask}(q,k)=\begin{cases}1&\text{if }k\in\{\mathbf{c}_{i},\tilde{\mathbf{v}}_{<i}\}\text{ for }q\in\tilde{\mathbf{v}}_{i}\\ \text{Causal}&\text{otherwise}\end{cases} This ensures that visual tokens act as contextual memory accessible to subsequent chunks, but the context itself can not pass to future textual content. Additionally, the position of an embedded token converts to: \text{Pos}(j)=\sum|\hat{\mathbf{v}}_{<i}|+j where |\hat{\mathbf{v}}_{i}|

Read the complete text