Latent Space Communication via K-V Cache Alignment
Lucio M. Dery, Zohar Yahav, Henry Prior, Qixuan Feng, Jiajun Shen, Arthur Szlam · 2026
Read Latent Space Communication via K-V Cache Alignment by Lucio M. Dery, Zohar Yahav, Henry Prior, Qixuan Feng, Jiajun Shen, Arthur Szlam (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06123.txt.
Introduction
Models could collaborate better if they shared more than text. Lucio M. Dery and 5 co-authors propose a shared latent space in a 2026 paper, so models can exchange internal states directly. Text is a narrow channel for cooperation between language models.
Each model gets adapters that translate its key-value cache, its stored memory of the conversation, into and out of the shared space. The pretrained models themselves stay unchanged. Experiments with Gemma-2 models showed the channel enables smooth inter-model communication and even improves single-model performance. Learned skills such as soft prompts transferred directly between different models through the shared space. The authors frame it as a step toward models fluidly sharing knowledge and capabilities.
Catalog record
| CREATOR | Lucio M. Dery, Zohar Yahav, Henry Prior, Qixuan Feng, Jiajun Shen, Arthur Szlam |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06123
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Space Communication via K-V Cache Alignment Lucio M. Dery, Zohar Yahav, Henry Prior, Qixuan Feng, Jiajun Shen, Arthur Szlam Abstract Solving increasingly complex problems with large language models (LLMs) necessitates a move beyond individual models and towards multi-model systems that can effectively collaborate. While text has traditionally served as the medium for inter-model communication, a richer and more efficient exchange is possible if models can access each other’s internal states directly. In this paper, we propose learning a shared representation space that aligns the k-v caches of multiple models, creating a high-bandwidth channel for collaboration without altering the underlying pre-trained parameters. We do so by augmenting each model with adapters to translate its state into and out of this shared space. Via a suite of experiments with Gemma-2 models, we demonstrate that this approach not only enables seamless inter-model communication but also improves individual model performance. We also show that the shared space allows for the direct transfer of learned skills, such as soft prompts, between different models. Our work represents a significant step towards a future where models can fluidly share knowledge and capabilities. keywords model collaboration, k-v cache, module portability, LLM 1 Introduction As the abilities of frontier large language models (LLMs) continue to expand, our community is rapidly producing many models with a diverse range of both generalist and domain-specific capabilities ( 5 ; 7 ; 8 ) . This growing pool of models presents both an opportunity and a challenge: how can we effectively orchestrate these individual models to solve problems that are beyond the scope of any single one? Such collaboration between models can take two primary forms. The first is multi-model systems, such as LLM agents ( 39 ; 29 ) , model cascades ( 43 ; 21 ) , model frankensteins ( 3 ) and mixture-of-expert models ( 11 ) . The second is skill reuse ( 36 ) , where modules or knowledge from one model – like soft prompts ( 24 ) or prefix k-v caches ( 25 ) – are shared and leveraged by others. Towards the north-star of seamless inter-model collaboration ( 20 ) , we provide evidence showing the feasibility of allowing models to read and write to each other’s key-value (k-v) cache latent space. A transformer’s ( 38 ) k-v cache is a rich, natural representation of its internal state. A sufficiently high dimensional cache vector space could allow for much higher communication bandwidth between models than token-level text exchange ( 15 ; 32 ) . By accessing each other’s latent space, models could directly read each others’ state and extract relevant knowledge, follow (soft) reasoning chains, and reuse partial computations or skills, leading to a more efficient collaboration on challenging tasks. Figure 1 : Overview of an example instantiation of our framework. [Upper] Each model (here Russian and Spanish LMs) is augmented with adapters for translating a k-v cache block into and out of the shared latent space. E.g Es-Out translates a k-v cache out of the Es model’s latent space and into the shared space. [Lower] We can perform mixed language modelling (for context switching settings), by translating generated prefix caches from a source model and continuing to decode from the target model. Unfortunately, even given the same input text, we cannot expect the latent spaces of two models trained under different conditions to be interchangeable ( 2 ) . This is because any given token passing through a model creates conditional dependencies on the k-v space of the layer(s) before it, causing the discrepancies in latent space between any pair of models with different parameters to compound with depth ( 33 ; 13 ) . Taking inspiration from the machine translation literature ( 35 ; 30 ) , given a set of models with possibly disjoint latent spaces, we propose to learn a global, shared k-v cache representation space that any model in the pool can read from and write to. Specifically, we augment each model with two adapters: one for translating its k-v cache block into the shared representation space, and another for translating a given shared space cache block into its model-specific latent space (Figure 1 ). As designed, this is an implicit (shared) latent space and presents a scalable communication channel with parameter overhead that only grows linearly with the number of models in the pool. Through a battery of experiments on Gemma-2 ( 37 ) open models, we demonstrate that we can teach a set of language models trained under a range of different conditions to read and write to each other’s k-v cache latent space. To achieve this, we provide a scalable architecture and efficient algorithm for learning a global latent space without modifying the individual models’ pre-trained parameters. We share the following findings: 1. Given a set of language models that differ along axes such as size, training data distribution and random initialization, we are able to learn a global shared k-v cache latent representation space. 2. Given sufficient data for learning the global latent space, we can improve the individual models’ language modelling performance by passing prefix k-v caches through the shared space before modelling the suffix text. 3. A global shared latent space enables learned modules to be portable between models. Specifically, we conduct experiments where soft-prompts, learned on a particular task for one model, can be translated via our global latent space, and used directly by another model to perform the same task. This allows modules learned by individual models to become a shared resource of the whole pool. 2 Methodology We enhance a pool of models with the ability to read and write to each other’s k-v cache representation spaces. By read , we mean a model can translate from another’s k-v cache space into its own, and write implies a model translating its k-v cache into another model’s representation space. We do so via a mechanism where individual model latent spaces are mediated by a learned globally shared, but implicit, latent space. In this section, we will detail our framework with architecture specifics and a learning algorithm. 2.1 Setup Consider a pool of N models indexed as \{m_{1},…,m_{i},…,m_{N}\} . We would like to learn a shared latent space, \mathbf{\Sigma} , amongst these models parameterized as follows: for each model, we learn a map \mathbf{T}^{[m_{i}\rightarrow\mathbf{\Sigma}]} into the shared space and another map \mathbf{T}^{[\mathbf{\Sigma}\rightarrow m_{i}]} out of the space. Thus, assuming that we have the cache for model m_{i} , KV^{i}\in\mathbf{R}^{B\times S\times L_{i}\times D_{i}} , our parameterization instantiates \mathbf{T}^{[m_{i}\rightarrow\mathbf{\Sigma}]}:\mathbf{R}^{B\times S\times L_{i}\times D_{i}}\rightarrow\mathbf{R}^{B\times S\times Q} and \mathbf{T}^{[\mathbf{\Sigma}\rightarrow m_{i}]}:\mathbf{R}^{B\times S\times Q}\rightarrow\mathbf{R}^{B\times S\times L_{i}\times D_{i}} . Where B, L, S, D are batch, layer, sequence and model dimensions respectively. For simplicity, we assume that all the models in the pool share the same vocabulary and so for a piece of text, S is the same for a for all models. Our method does not require this assumption, but it is convenient for conveying a simple setup. Though each model may have a different number of layers L_{i} and model dimension D_{i} , we map all of them to a shared embedding space of fixed size, Q . The shared space therefore has no direct layer-wise demarcation. Instead