Tiny-Engram: Trigger-Indexed Concept Tables for Generative Vision

Runyuan Cai, Yiming Wang, Yu Lin, Xiaodong Zeng · 2026

Read Tiny-Engram: Trigger-Indexed Concept Tables for Generative Vision by Runyuan Cai, Yiming Wang, Yu Lin, Xiaodong Zeng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20309.txt.

Introduction

Runyuan Cai and three co-workers set out Tiny-Engram, a trigger-indexed table for frozen image and video makers. Adapters or weight tweaks teach new ideas, with scant say over when they return. Stored slots, keyed to n-gram phrases, steer text encoder states just where the phrase sits. Elsewhere the base model runs unchanged. A rare cue phrase can call up a target face while the rest of the prompt shapes the scene. In video it also works, though a face may drift on new prompts.

Catalog record

CREATORRunyuan Cai, Yiming Wang, Yu Lin, Xiaodong Zeng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35579
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20309

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tiny-Engram: Trigger-Indexed Concept Tables for Generative Vision Runyuan Cai, Yiming Wang, Yu Lin, Xiaodong Zeng Abstract Current personalization methods for generative vision models typically encode new concepts through continuous adapters or weight updates, yet provide limited control over whether and when a concept should be retrieved. In this work, we introduce Tiny-Engram, a compact trigger-indexed concept table that gives visual memories an explicit lexical address and activation boundary inside frozen image and video generators. Tiny-Engram parameterizes each concept as a small set of memory entries indexed by registered n-gram matches, which modulate text-encoder hidden states only within the matched trigger region. Outside this lexical support, the conditioning pathway is identical to that of the frozen base model. Across both single-encoder latent diffusion and multi-encoder diffusion-transformer backbones, this formulation binds a rare trigger phrase to a target identity while preserving compositional control from the surrounding prompt. We further evaluate the same table-based memory in a text-conditioned video generation setting, where the trigger path reliably alters the generated subject but fine-grained identity persistence across held-out video prompts remains limited. Taken together, these results suggest that small, explicitly addressed concept tables are a practical route to modular visual personalization, with strongest evidence in image generation. For video diffusion, the remaining gap points to a broader requirement: temporally stable identity likely depends on tighter coupling between text-side memory and the evolving visual state, motivating future work on memory injection beyond the text-conditioning interface. Keywords Tiny-Engram \cdot TinyEngram \cdot Concept Tables \cdot Generative Vision \cdot Personalization 1 Introduction Model adaptation is often constrained by two requirements: the update should be small enough to train and store cheaply, and it should not unnecessarily disturb behavior already present in the pretrained model. Parameter-efficient fine-tuning (PEFT) methods address the first requirement by using adapters, continuous prompts, low-rank updates, or activation scaling [ 9 , 16 , 14 , 10 , 18 ] . Retrieval-augmented and memory-augmented language models address a related problem by separating part of model behavior from dense parametric storage [ 7 , 15 , 12 , 1 ] . Neither direction, by itself, guarantees that an added capability has a discrete activation condition inside the model. This activation boundary matters for modular adaptation. A continuously active adapter can affect prompts outside the intended domain, and independently trained concept modules can interact when merged or composed. For many personalization and domain-specialization settings, it is useful to ask for stronger locality: the learned update should be applied only when a registered textual condition is present, while all unmatched inputs should follow the original model path. DeepSeek’s Engram work, commonly discussed as a key memory primitive in the DeepSeek-V4 research line, gives one answer to this question. It proposes conditional memory through scalable N -gram lookup, so that matched token spans retrieve learned memory entries inside the model [ 3 ] . Tiny-Engram operationalizes this principle as a compact adaptation module: a registered n -gram in the prompt indexes a learned concept table, and the retrieved vector modifies hidden states only at the matched span. Thus the learned memory is tied to explicit lexical support rather than applied as a globally active adapter. We first validate TinyEngram in the language-model setting, where Engram was originally proposed. Using Qwen3 as the frozen backbone [ 29 ] , we insert Engram modules into selected transformer layers and train only the added memory parameters. In biomedical adaptation, the Engram branch learns domain-specific behavior while preserving broad MMLU performance [ 8 ] . In a high-intensity function-call adaptation setting, a tuned Engram checkpoint reaches comparable or lower validation loss than a LoRA baseline while retaining stronger TruthfulQA performance [ 17 ] . These results establish the premise used in the rest of the paper: trigger-indexed memory can be optimized as a PEFT module, and its localized activation can improve resistance to catastrophic forgetting under the tested adaptation pressure. This premise motivates the extension to generative vision, where personalization requires binding a textual trigger to a visual concept without changing unrelated prompts. Existing text-to-image personalization methods bind new concepts through learned tokens, subject-specific fine-tuning, attention updates, or image-prompt adapters [ 5 , 25 , 13 , 26 , 30 ] . In latent diffusion and diffusion-transformer models, text encoders provide the conditioning states that guide image synthesis [ 19 , 20 , 23 , 4 ] . This makes the text-conditioning stream a suitable interface for trigger-indexed memory: TinyEngram can modify the prompt representation while leaving the image backbone frozen. In SD1.5 [ 23 ] , it injects Engram vectors through the single CLIP text encoder [ 19 ] . In SD3.5 [ 4 ] , it extends the same mechanism to the CLIP-L, OpenCLIP-G, and T5 encoder stack [ 19 , 11 , 20 ] . This multi-encoder setting introduces a practical scale issue: the injected memory must have comparable effect across text streams with different hidden-state magnitudes. We address this with a relative-norm injection rule. Across the image experiments, a rare textual trigger is bound to a target visual identity while the U-Net or DiT-style denoising backbone remains frozen [ 24 , 21 , 4 ] , and no-trigger controls preserve the base generation path. We also include a preliminary video extension. Video diffusion models add temporal and motion constraints to the image-generation problem, making identity personalization more demanding than in single-frame synthesis [ 28 , 6 , 2 , 27 ] . On Wan2.2 TI2V-5B, text-side Engram injection clearly changes generation, but does not yet match the identity stability observed in SD3.5 images. We use this result as a boundary case rather than as the main evidence for the method. This report makes four contributions: a trigger-indexed Engram formulation for parameter-efficient adaptation; a text-encoder memory-injection method for image personalization in frozen generative backbones; a relative-norm injection rule for calibrated text-stream updates; and an initial video study that clarifies both the promise and current limitation of text-side memory beyond images. 2 Method 2.1 Overview Tiny-Engram is a trigger-indexed PEFT module. At the most general level, it adds trainable memory residuals to selected hidden states of a frozen backbone, and activates those residuals only when the input contains registered or hashed n -gram evidence. The base model parameters remain fixed; only the Engram memory parameters and lightweight gates or scales are optimized. This gives the adaptation both small storage cost and an explicit activation boundary. The visual method studied in this paper is a specialization of this formulation for text-conditioned image and video generators. Let G denote an image or video diffusion model with one or more text encoders \{E_{m}\}_{m=1}^{M} . The generator backbone, variational autoencoder, sampler, and original text-encoder weights are kept fixed. For each wrapped text encoder, Tiny-Engram adds only two trainable objects: a table of concept vectors and a per-channel injection-scale vector. A prompt that does not contain the registered trigger follows the frozen model path; a prompt that contains the trigger receives a residual update only on the matched trigger tokens. We instantiate the visual experiments as a controlled study of binding one source appearance to a new fictional trigger name. The source visual concept is the protagonist appearance from Hideo Kojima’s Death Stranding , while the prompt-side

Read the complete text