Boosting Reasoning in Large Multimodal Models via Activation Replay

Yun Xing Xiaobin Hu Qingdong He Jiangning Zhang, Shuicheng Yan Shijian Lu Yu-Gang Jiang · 2025

Read Boosting Reasoning in Large Multimodal Models via Activation Replay by Yun Xing Xiaobin Hu Qingdong He Jiangning Zhang, Shuicheng Yan Shijian Lu Yu-Gang Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19972.txt.

Introduction

Yun Xing and six co-authors propose Activation Replay, a training-free way to sharpen multimodal reasoning. Reinforcement learning with verifiable rewards improves reasoning in large multimodal models, but why it works is poorly understood. Using a logit lens view, the authors track how input activations shift after such training across several models.

They find that low-entropy activations shift unexpectedly while high-entropy ones move less, and controlled tests link this pattern to reasoning itself. Activation Replay works at test time on visual tokens, replaying low-entropy activations from the base model to steer its tuned counterpart. It improves reasoning in math, visual agent, and video tasks, widens answer coverage, and beats alternative replay choices in their comparisons.

Catalog record

CREATORYun Xing Xiaobin Hu Qingdong He Jiangning Zhang, Shuicheng Yan Shijian Lu Yu-Gang Jiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19972

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Boosting Reasoning in Large Multimodal Models via Activation Replay Yun Xing Xiaobin Hu Qingdong He Jiangning Zhang, Shuicheng Yan Shijian Lu Yu-Gang Jiang Abstract Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach to incentivizing reasoning capability in Large Multimodal Models (LMMs), while the underlying mechanisms behind this post-training paradigm are poorly understood. We begin by exploring how input activations are affected by RLVR through the perspective of logit lens. Our systematic investigations across multiple post-trained LMMs suggest that RLVR shifts low-entropy activations unexpectedly, while high-entropy ones are less affected. We further demonstrate that such phenomena are associated with LMM reasoning by controlled experiments, suggesting a potentially beneficial role of modulating low-entropy activations. To this end, we propose Activation Replay , a novel simple yet effective training-free approach that boosts multimodal reasoning of post-trained LMMs without requiring expensive policy optimization. Our design involves manipulation of visual tokens at test time, replaying low-entropy activations from the input context of base LMMs to regulating the RLVR counterparts. Activation Replay triggers better reasoning across diverse scenarios, including mathematics, o3-like visual agents, and video reasoning. We further show that Activation Replay boosts Pass@K and mitigates narrower reasoning coverage of RLVR. Our design is compared against alternative choices, such as replaying high-entropy activations instead of low-entropy ones, or direct cross-model intervention instead of manipulating input tokens, demonstrating the superiority of our implementation. Codes will be made publicly available. https://christian42mmreason.github.io/replay $\dagger$ $\dagger$ footnotetext: Corresponding author 1 Introduction Large Multimodal Models (LMMs) 22 ; 11 ; 4 ; 66 ; 14 ; 1 have been playing crucial roles in understanding multimodal interactions from users, across various domains such as images, videos 10 , or 3-D inputs 65 . Recently, Reinforcement Learning with Verifiable Rewards (RLVR) 33 has been verified as an effective approach that incentivizes reasoning capability in LMMs 28 ; 44 , characterized by long chain-of-thought verifications before arriving at the final answers. This approach opens new frontiers for existing LMMs to tackle challenging multimodal scenarios, such as mathematics 45 ; 31 ; 51 , agentic 63 ; 36 ; 18 or video reasoning 8 ; 13 ; 9 . Figure 1 : Lidar Plot for Performance Gains. RLVR LMMs include VL-Rethinker, DeepEyes and Video-R1 44 ; 63 ; 13 . Our method boosts multimodal reasoning across diverse tasks consistently in training-free manner. Sufficient efforts have been made in effective post-training recipes 28 ; 44 ; 7 across a wide spectrum of input domains. Early studies focus on enabling reasoning of LMMs via tailored chain-of-thought verification data and recipes with simple rule-based rewards 28 . More recent studies explore further on explicitly incentivizing reasoning patterns such as reflection 44 ; 43 , or optimizing training efficiency by addressing vanishing gradients 35 or supervising minority fork tokens 47 . These approaches have been solid explorations for incentivizing reasoning of LMMs with dedicated training. In spite of broad explorations of RLVR recipes, it is still unclear on how the inner workings of LMMs are generally affected by such post-training, especially how multimodal contexts are modified unexpectedly. We investigate how input activations are shifted through the perspective of logit lens 29 ; 30 . The input activations of LMMs fundamentally determine how post-trained LMMs perform, especially for reasoning scenarios where decoding over input contexts involves long decoding steps. We first explore on how input activations are affected by RLVR, starting with observing the shifts of input activations. We discover evidences that top predictions through logit lens are shifted after RLVR (Figure 2 ). Furthermore, we explore the effect of RLVR on Kullback-Leibler divergence, the metric that quantitatively measures token-level distributional shifts, and ablate the analysis by base LMM entropy percentiles 47 . To this end, we find high-entropy input activations are less affected by RLVR, as compared to low-entropy input activations (Figure 3 ). We perform studies from two perspectives to investigate the distinctive role of low- and high-entropy input activations in multimodal reasoning. On the one hand, for the perturbation study , we manually disrupt the inputs with random noises to synthesize variations on activations of post-trained LMMs, so that KL divergence shifts over the same inputs vary. The goal is to observe how the perplexity of sampled correct responses are influenced. Qualitative cases suggest that smaller KL divergence shifts on low-entropy activations leads to a lower perplexity of correct reasoning paths (Figure 4 ), while incorrect ones are vise versa ( Appendix ). On the other, we perform a straightforward intervention study to directly interrupt the forward pass of RLVR post-trained LMMs with low-entropy activations from their base LMMs, while keep the high-entropy ones. Despite the representations of RLVR and base are largely different, we find that such strategy is beneficial in some cases. Both studies lead to that regularization on RLVR low-entropy activations benefits multimodal reasoning. On top of the abovementioned analysis, we propose Activation Replay, a simple yet effective training-free solution that boosts multimodal reasoning of post-trained LMMs, without requiring expensive policy optimization. By modulation of visual tokens at test time, Activation Replay enforces RLVR low-entropy activations to mimick the distributions from paired base LMMs at test time. We validate our approach across diverse LMMs post-trained by RLVR and diverse reasoning scenarios, including mathmatics 23 ; 31 ; 60 ; 45 ; 51 , multi-turn agents that perform visual search in high-resolution images 49 , and video reasoners that think across frames 13 , where our approach showcases consistent performance gains across these setups (Figure 1 ). We further demonstrate that our design boosts Pass@K 5 , mitigating narrower reasoning coverage of RLVR 57 . We summarize our contributions as follows, Figure 2 : Qualitative Case on Logit Lens . The math visual input is from 45 . Top-2 or Top-3 predictions (words) of input activations shift from base to RLVR couterpart. • We carry out systematic studies about how RLVR affects LMM input activations. We observe that low-entropy activations are affected more than high-entropy ones, and we reveal their associations with LMM reasoning behaviors through the perturbation and intervention studies. • We propose Activation Replay, a novel training-free approach that involves manipulating visual tokens at test time, specifically, replaying low-entropy activations from the input context of base LMMs to regulate the RLVR counterparts. • We test our approach over multiple multimodal reasoning scenarios, demonstrating that Activation Replay is effective across mathematical, o3-like agentic, and video reasoning. We additionally show that our design boosts Pass@K, mitigating narrower reasoning coverage of RLVR. 2 Motivation In this section, we discuss how LMM input activations are shifted by RLVR unexpectedly, from the perspective of logit lens 29 ; 30 , and how these shifts are associated with LMM reasoning. We first briefly introduce how logit lens is defined and then provide qualitative and quantitative comparisons (Figure 2 and 3 ) on the shifts after RLVR. For exploring how reasoning is affected, we design perturbation and intervention studies, for which details are included in this section, followed by conclusion of our findings. 2.1 Background Notation . Consider an LMM \mathcal{F} , which typically comprises a visual encoder \mathcal{V} , a multimodal connector \mathcal{F}_{c} , and an LLM \mathcal{M} with L layers. The input to an

Read the complete text