Direct Semantic Communication Between Large Language Models via Vector Translation

Fu-Chun, Yang, Jason Eshraghian · 2025

Read Direct Semantic Communication Between Large Language Models via Vector Translation by Fu-Chun, Yang, Jason Eshraghian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03945.txt.

Introduction

Fu-Chun Yang and Jason Eshraghian propose letting large language models talk to each other directly through translated internal vectors instead of plain token messages. Token passing in debate, reflection, and tool-calling setups discards most latent meaning and adds computing overhead, the pair argue.

A dual-encoder translator between two seven-billion-parameter models reached a moderate average alignment in cosine terms. Injecting translated vectors at thirty percent blending steered the target model's output without destabilising its scores. Transfer proved uneven by direction, with general-purpose models giving more transferable representations than instruction-tuned ones. Careful injection kept stability while showing that cross-model latent communication is feasible in principle.

Catalog record

CREATORFu-Chun, Yang, Jason Eshraghian
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-03945

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Direct Semantic Communication Between Large Language Models via Vector Translation Fu-Chun, Yang, Jason Eshraghian Abstract In multi-agent settings, such as debate, reflection, or tool-calling, large language models (LLMs) pass messages as plain tokens, discarding most latent semantics. This constrains information transfer and adds unnecessary computational overhead. We form a latent bridge via vector translations, which use learned mappings that enable direct semantic exchange between representation spaces. A dual-encoder translator trained between Llama-2-7B and Mistral-7B-Instruct attains an average cosine alignment of 0.538. Injecting the translated vectors at 30% blending strength steers the target model’s generation without destabilizing logits. Bidirectional evaluation shows a 2.01 : 1 2.01:1 transfer asymmetry, indicating that general-purpose models yield more transferable representations than instruction-tuned variants. This conservative injection preserves computational stability while demonstrating that cross-model latent communication is feasible, enabling collaborative AI systems that share meaning, rather than tokens. 1 Introduction When two Large Language Models (LLMs) debate an answer, critique each other’s chain of thought, or sequentially refine a shared draft of text, they speak through plain tokens. Every round forces each model to flatten rich geometry into text, operate on that, then rebuild meaning. Ultimately, computational resources are wasted, and limited information bandwidth can erase nuance. Specialised LLMs thus operate in isolation, communication only through text interfaces that constrain information transfer and add overhead. Encoding semantics into tokens and re-decoding them discards much of the latent structure that models use internally, blurring complex relationships in the process. Yet each LLM carries a distinct internal representation space shaped by architecture, training objective, and data. Those spaces differ enough that raw vectors are not interchangeable, prompting the question: Can semantic information encoded in one model’s vector space be translated so another model can use them directly? We demonstrate this is possible by learning bidirectional vector translations that create a latent bridge between models. Injecting these translated vectors directly into a target model’s pipeline lets the pair share meaning without serialising to tokens, enabling chains, ensembles, and parallel collaborations to run at latent speed, and bypass text-based limitations. Key Contributions : 1. We introduce a dual-encoder framework that learns robust semantic mappings with minimal training overhead. 2. We develop a conservative injection scheme that preserves computational stability while enabling semantic transfer. 3. We provide evidence of consistent semantic transfer patterns across multiple domains with clear performance boundaries. 2 Related Work Cross - modal semantic alignment. Contrastive Language–Image Pre - training (CLIP) demonstrated that heterogeneous encoders can be trained to share a common representation space through a contrastive objective, enabling zero - shot transfer across more than 30 vision benchmarks [ 7 ] . Our translator borrows this idea of geometric alignment but applies it to two text - only LLMs with no shared parameters or vocabulary. Inter - model knowledge transfer. Classical knowledge - distillation compresses a teacher network into a smaller student by matching softened output distributions [ 2 ] . Earlier work on bilingual word - embedding projection showed that simple linear maps can bridge independently trained vector spaces [ 6 ] . Both approaches, however, operate on output tokens rather than hidden states. Latent - state manipulation for controllable generation. Plug - and - Play Language Models (PPLM) steer generation by performing gradient - based updates on hidden activations at inference time [ 1 ] . CHRT learns explicit transformations that modify hidden states to impose multiple attributes without retraining the base model [ 4 ] . More recently, function vectors were extracted and re - injected across prompts to trigger entire tasks inside an LLM, revealing compact causal directions in latent space [ 8 ] . These works are intra - model; our translator instead learns cross - model mappings. Cross-lingual and cross-family alignment. [ 5 ] aligned internal sentence representations across languages to improve multilingual in - context learning, showing that contrastive alignment inside one model boosts generalisation. 3 Methodology 3.1 Problem Formalization LLaMA-2-7B Source 4096D Dual-Encoder Translator 512D Hidden Mistral-7B Target 4096D Semantic Input Enhanced Output Injection 30% Blend Translate Inject General-Purpose Instruction-Tuned 2.01:1 Asymmetry Figure 1: Cross-model vector translation architecture enabling direct semantic communication between LLaMA-2-7B and Mistral-7B through dual-encoder translation with conservative injection mechanism (30% blending) and bidirectional capabilities showing 2.01:1 performance asymmetry. We formalize cross-model vector translation as learning bidirectional mapping functions between semantic representation spaces of two distinct LLMs. Given source model M_{1} with representation space \mathbb{R}^{d_{1}} and target model M_{2} with representation space \mathbb{R}^{d_{2}} , we learn translation functions: \displaystyle f:\mathbb{R}^{d_{1}} \displaystyle\rightarrow\mathbb{R}^{d_{2}}\quad\text{(forward translation)} (1) \displaystyle g:\mathbb{R}^{d_{2}} \displaystyle\rightarrow\mathbb{R}^{d_{1}}\quad\text{(reverse translation)} (2) For semantic content S with vectors v_{1}^{S} and v_{2}^{S} from M_{1} and M_{2} respectively, translation functions satisfy: f(v_{1}^{S})\approx v_{2}^{S} and g(v_{2}^{S})\approx v_{1}^{S} with respect to semantic similarity measures. 3.2 Architecture Design Our dual-encoder architecture consists of: (1) Semantic Feature Extractor reducing vectors from 4096D to 512D intermediate representation, (2) Cross-Domain Alignment Module implementing multi-head attention (embed_dim=512, num_heads=8), and (3) Target Space Generator expanding aligned representation to target dimensionality. 3.3 Training Strategy We employ composite loss: \mathcal{L}=\mathcal{L}_{\text{trans}}+0.5\mathcal{L}_{\text{cycle}}+0.3\mathcal{L}_{\text{contrast}}+0.2\mathcal{L}_{\text{dist}} combining direct translation loss (MSE between translated and target vectors), cycle consistency loss, contrastive loss using InfoNCE, and distribution preservation loss. 3.4 Vector Injection Mechanism Vector injection represents the core innovation enabling direct semantic communication between models with different architectures. The injection mechanism operates on the target model’s internal processing pipeline without requiring parameter modifications or architectural changes. This approach allows real-time semantic guidance while preserving the target model’s fundamental computational capabilities. The injection process targets the final three transformer layers (layers -3, -2, and -1) where high-level semantic processing occurs and external information can be most effectively integrated. These layers represent the optimal balance between semantic abstraction and generation specificity, ensuring that injected semantic content influences high-level reasoning without disrupting low-level linguistic processing. Translated vectors are injected using conservative blending: h^{\prime}_{i}=(1-\alpha)\cdot h_{i}+\alpha\cdot v_{\text{translated}} where \alpha=0.3 represents the injection strength empirically determined to balance semantic transfer effectiveness with computational stability. The conservative 30% blending ratio ensures that the model’s natural processing pathways remain dominant while allowing meaningful semantic influence from the translated vectors. Spatial targeting within the injection mechanism focuses on the final token positions within each sequence, where the model typically consolidates semantic information for generation decisions. Rather than uniformly modifying all hidden states, the injection process selectively targets the final three token positions, preserving natural processing of earlier context while introducing semantic guidance at critical decision points for content generation. The temporal dynamics of injection ensure that semantic information is introduced at appropriate points in the generation process. The mechanism activates only during initial stages of generation, allowing the model to incorporate semantic guidance while maintaining natural autoregressive generation patterns for subsequent tokens. This approach prevents systematic disruption of the generation process while enabling effective semantic transfer from translated vectors. 4 Results and Experimental Analysis 4.1 Experimental Configuration We employ Llama2-7B [ 9 ] and Mistral-7B [ 3 ] , providing identical 4096D hidden dimensionality while preserving architectural diversity between general-purpose and instruction-tuned paradigms. The experimental infrastructure utilized 4× NVIDIA RTX A6000 GPUs with bfloat16 precision throughout the pipeline for memory efficiency, with LLaMA deployed on GPU 0 and Mistral on GPU 1 for parallel processing. Training employed 50 epochs with AdamW optimizer at learning rate 1e-3. We designed five

Read the complete text