HyperTokens: Controlling Token Dynamics for Continual Video–Language Understanding

Toan Nguyen, Yang Liu, Celso De Melo, Flora D. Salim · 2026

Read HyperTokens: Controlling Token Dynamics for Continual Video–Language Understanding by Toan Nguyen, Yang Liu, Celso De Melo, Flora D. Salim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06662.txt.

Introduction

Nguyen and team address continual VideoQA, where tuning causes task clash and storing task-specific prompts grows heavy. They bring DynaTokens, a token generator making fine-tuning tokens on demand for prompt updates through generation weights. Meta-learning-inspired rules look ahead to avoid sharp update directions while tying it to prior-task ways, picking flatter cross-task minima. Slope-free routing from ready token and visual codes reduces router drift. Across such VideoQA benchmarks, It beats baselines on accuracy and forgetting and enables cross-modal transfer.

Catalog record

CREATORToan Nguyen, Yang Liu, Celso De Melo, Flora D. Salim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55386
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.06662

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HyperTokens: Controlling Token Dynamics for Continual Video–Language Understanding Toan Nguyen, Yang Liu, Celso De Melo, Flora D. Salim Abstract Continual VideoQA with multimodal LLMs is hindered by interference between tasks and the prohibitive cost of storing task-specific prompts. We introduce HyperTokens, a transformer-based token generator that produces fine-tuning tokens on demand, giving explicit control over prompt updates while keeping memory fixed. To suppress forgetting, we propose meta-inspired regularisers that look ahead to avoid task-specific sharp directions and anchor the evolving generator to prior tasks. We further connect our objective to sharpness-aware optimisation, providing insight into why it encourages flatter cross-task minima and improves retention. Beyond regularisation, HyperTokens exploits lightweight auxiliary multimodal supervision through shared generation weights; guided by a causal perspective, we design feasible objectives and surrogate mutual-information losses to regularise anti-causal cross-modal directions. Across two standard continual VideoQA benchmarks, HyperTokens achieves higher average accuracy with substantially lower forgetting. Finally, we introduce a challenging cross-modal ImageQA → \rightarrow VideoQA protocol and show that HyperTokens enables robust continual transfer in this setting. W[2] #2 1 Introduction Modern multimodal LLMs are increasingly deployed on continuous multimodal streams of video, audio, and language from dynamic environments (e.g., wearables, cameras, and sensors), driven by their remarkable ability to learn and reason over multimodal structure, as evidenced by strong performance on VideoQA ( 42 ; 29 ; 37 ) . In such settings, the conventional train-then-deploy paradigm for large-scale pretrained models ( 24 ; 46 ; 59 ; 28 ) hinges on stationarity, and can degrade when confronted with evolving tasks, distribution shifts, and previously unseen concepts. Naïvely fine-tuning on new tasks often causes catastrophic forgetting ( 30 ; 6 ) , which overwrites prior knowledge, and the problem compounds as tasks accumulate; moreover, updating all parameters of a large pretrained model is computationally expensive and quickly becomes impractical. A practical alternative is parameter-efficient adaptation (PEA) ( 6 ; 1 ; 39 ) , which updates only a small set of added parameters on top of a frozen backbone to accommodate incoming data while largely preserving the model’s base knowledge. However, in continual learning for challenging multimodal video tasks such as VideoQA, task distributions can differ sharply (e.g., indoor versus outdoor videos or distinct question types), so prompt updates may interfere and still incur substantial forgetting without careful parameter control ( 44 ) . Recent methods ( 39 ; 41 ) sidestep this by storing task-specific adapter prompts and stacking them over time, but the approach scales poorly as the number of tasks grows. Other approaches ( 44 ; 6 ; 51 ) share a prompt parameterisation across tasks, but updates can conflict on the shared components and trigger interference under a small prompt budget. Alternatives such as LoRA or prefix tuning ( 1 ; 54 ) are efficient, yet they remain challenging to (i) scale nicely to many tasks, (ii) preserve fine-grained, task-specific control without cross-task interference, and (iii) extend reliably to multimodal VideoQA. Figure 1 : HyperTokens overview. ( Left ) Continual adaptation with HyperTokens for VideoQA and cross-modal transfer VisualQA \rightarrow VideoQA. A fixed-size generator synthesises task-specific fine-tuning tokens. ( Middle ) Task-code learning via a multimodal contrastive objective with a prototype bank. ( Right ) Transformer-based token-generator architecture. To address these challenges, we draw inspiration from hypernetworks ( 16 ) and propose HyperTokens , a universal token generator that produces adapter tokens on demand . Given a compact multimodal task code, HyperTokens synthesises task-specific prompts while keeping the size of generator fixed, so emphmemory grows only minimally with the number of tasks. Figure 1 illustrates the design. A key challenge in hypernetwork-based prompt control is to learn informative task codes and to update the token generator without forgetting. To learn task codes, we introduce a contrastive embedding loss that leverages both video and question features to capture task-specific structure. To preserve the generator, we use a transformer-based token generator trained with a meta-inspired objective that (i) looks ahead to avoid task-specific overfitting directions and (ii) looks back to match an anchored behaviour given a small set of past task codes. We theoretically connect this objective to sharpness-aware minimisation (SAM) ( 13 ) , showing that it favours flatter cross-task minima and improves robustness under continual adaptation. Beyond regularisation, HyperTokens benefits from lightweight auxiliary multimodal supervision effectively enabled by shared generation weights. From a causal view ( 36 ; 31 ) of VideoQA, we prioritise feasible objectives that follow at least one causal direction (e.g., predicting questions from videos and answers), since the reverse direction (predicting videos from questions and answers) is anti-causal and prone to hallucinated visual evidence. To still exploit this signal, we introduce surrogate mutual-information losses ( 35 ) at both the token and global video levels to regularise cross-modal alignment. Empirically, HyperTokens achieves strong accuracy with substantially reduced forgetting on standard continual VideoQA benchmarks. Further, we introduce a new and highly challenging ImageQA \rightarrow VideoQA setting, where the model continually shift from static image understanding to temporal video reasoning. This transfer induces both a modality mismatch and a learning-nature mismatch (single-frame recognition and temporal reasoning), under which recent SOTA PEA baselines degrade significantly; in contrast, HyperTokens remains robust and surpasses the strongest prior method, underscoring the effectiveness of multimodal token-hypernetwork generation. 2 Related Work Video Question Answering (VideoQA) aims to strengthen video–language understanding by requiring models to answer questions grounded in video inputs. Existing approaches either develop specialised video vision–language models ( 11 ; 2 ; 55 ) or adapt large pretrained LLMs to better capture temporal visual context ( 53 ; 57 ; 58 ) . However, none of these methods is designed for continual learning from a stream of VideoQA datasets, where the input distribution can shift substantially. Continual Learning for VideoQA considers a streaming setting where a model is trained sequentially on multiple VideoQA datasets; the key challenge is mitigating catastrophic forgetting. Replay-based methods alleviate forgetting by storing a small buffer of past samples to replay them while learning new tasks ( 5 ; 32 ) , but storing videos is often prohibitive. Regularisation and architectural approaches have been mainly studied for multimodal classification ( 19 ; 18 ; 27 ) and do not readily transfer to next-token LLM-style VideoQA. Recent work therefore emphasises parameter-efficient adaptation, updating only a small set of added parameters atop a frozen backbone to accommodate new tasks while preserving prior knowledge. Parameter-Efficient Adaptation (PEA) ( 58 ; 23 ) enable continual adaptation in LLMs/VLMs by training lightweight task-specific modules while freezing the backbone, thereby preserving base knowledge. L2P ( 51 ) maintains a prompt pool and appends retrieved prompts to the input embeddings via prompt tuning ( 22 ) for task-agnostic PEA, but the pool’s fixed capacity induces interference and forgetting. DualPrompt ( 50 ) mitigates this by introducing prompts—task-invariant (G-Prompt) and task-specific (E-Prompt), with CODA-Prompt ( 43 ) extends this idea by attention-prompting modules. LAE ( 14 ) proposes a unified PEA framework where an adaptation-speed calibration module guides an online PEA module while consolidating knowledge into an offline PEA module, and combines the two at inference. HiDe-Prompt ( 49 ) adopts a hierarchical prompting design and uses an ensemble at inference. ProgPrompt ( 39 ) learns task-specific prompts

Read the complete text