ES-Merging: Biological MLLM Merging via Embedding Space Signals
Wonbin Lee, Dongki Kim, Sung Ju Hwang · 2026
Read ES-Merging: Biological MLLM Merging via Embedding Space Signals by Wonbin Lee, Dongki Kim, Sung Ju Hwang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14405.txt.
Introduction
Lee, Kim, and Hwang merge single-modality biology models into one multimodal model with ES-Merging. Standard merging picks blend weights from parameter-space rules that ignore what each model knows. Their method reads signals from the embedding space instead. Coarse signals set layer-wise blend weights, and fine signals set element-wise ones. The two combine for a fuller estimate. Tests show better cross-modal reasoning and better retention of single-modal knowledge than prior merging methods.
Catalog record
| CREATOR | Wonbin Lee, Dongki Kim, Sung Ju Hwang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63055 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14405
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ES-Merging: Biological MLLM Merging via Embedding Space Signals Wonbin Lee, Dongki Kim, Sung Ju Hwang Abstract Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose a representation-aware merging framework that estimates merging coefficients from embedding space signals. We first design a probe input that consists of different modality tokens and forward it through each specialized MLLM to obtain layer-wise embedding responses that reflect modality-specific representation changes. We then estimate complementary merging coefficients at two granularities from the embedding space: layer-wise coefficients from coarse-grained signals and element-wise coefficients from fine-grained signals, which are jointly combined for robust coefficient estimation. Experiments on interactive effect prediction benchmarks show that our method outperforms existing merging methods and even surpasses task-specific fine-tuned models, establishing that embedding space signals provide a principled and effective foundation for cross-modal MLLM merging. † † footnotetext: * denotes Equal Contribution 1 Introduction Multimodal Large Language Models (MLLMs) have been emerging as crucial foundational models for scientific discovery, extending their perception to diverse biological modalities across molecules ( 34 ; 24 ) , proteins ( 1 ; 14 ) , and cells ( 12 ; 36 ) . Despite their impressive progress within each modality, many scientific problems of interest are cross-modal, requiring to understand the interactive effects such as protein-ligand interactions or drug effectiveness to cell types. However, existing biological MLLMs are specialized to a single modality, resulting in limited intersectional knowledge and unreliable cross-modal reasoning. Building a unified model by jointly training on different modalities is a straightforward approach to upskill cross-modal understanding. However, it is impractical and time-consuming since constructing curated cross-modal instruction datasets in the scientific domain typically requires intensive labor and highly specific expertise to elaborate the underlying principles of complex interactions. As an alternative, model merging has gained attention by efficiently combining parameters of multiple specialized models. To retain their task-specific knowledge, existing methods 50 ; 20 ; 9 exploit parameter space signals, such as magnitudes, signs, and directions, heuristically assigning merging coefficients. However, parameter space heuristics are input-agnostic and therefore provide only weak, indirect proxies for modality-specific adaptation. Such input-blindness makes it difficult to isolate meaningful cross-modal interactions, failing to accurately combine these adaptations and severely degrading cross-modal merging. Our main observation is that the input-aware embedding space contains the modality-specific information. As shown in Fig. 1 , when molecular tokens are processed by different MLLMs, the hidden representations form clearly different embedding distributions. In particular, the molecule-specific LLM induces a more distinct distribution due to its modality-specific understanding. Further, we measure the embedding distribution distance between the base LLM and each specialized MLLM under specialized and non-specialized token inputs. Fig. 2 shows that specialized inputs consistently produce larger embedding distribution distance than non-specialized inputs, suggesting that the embedding responses faithfully reflect modality-specific adaptation when the input matches the model’s specialization. This observation motivates our central design principle: rather than relying solely on heuristic parameter space signals, we estimate merging coefficients based on the layer-wise embedding space signals. Figure 1: Molecule token embedding visualization of the last transformer block for the base LLM and each specialized LLM. Figure 2: Layer-wise embedding distribution distance under specialized and non-specialized tokens using sliced Wasserstein distances (SWD) ( 2 ) . Motivated by this observation, we propose an embedding-signal-based MLLM merging ( ES-Merging ), a novel framework that moves the model merging paradigm from parameter space signals to embedding space signals. Our intuition is that the modality specialization of an MLLM can be measured by the differences in the embedding space between the base LLM and the MLLM as shown in Fig. 1 . To this end, we first design a probe input containing multimodal tokens of different modalities. By forwarding this input through each MLLM and the base LLM, we obtain layer-wise embeddings that reflect modality-specific representation changes across layers. Based on these embeddings, we propose to compute merging coefficients at two complementary granularities: layer-wise and element-wise. In the layer-wise manner, we compute a layer-level importance score by identifying layers where the embedding distributional shift grows the most, capturing coarse-grained specialization. In the element-wise manner, we estimate fine-grained importance by identifying parameters that most influence the representation shift. We then combine the coarse layer importance and fine element importance to produce final merging coefficients, which are used to fuse specialized models into a single unified MLLM. This design enables more robust and calibrated coefficient estimation by combining layer-level specialization with parameter-level sensitivity. We validate the effectiveness of ES-Merging on interactive effect prediction tasks over diverse biological modalities by merging three different specialized MLLMs into a single unified model. Experimental results show that ES-Merging outperforms not only other model merging methods but also the task-specific fine-tuned models, empirically confirming our representation-aware merging framework is crucial to obtain the salient modality-specific signals. Further analyses show the effectiveness of our components: combining layer- and element-wise merging coefficients best performs compared to using only one, highlighting the necessity of integrating complementary specialization signals at different granularities. 2 Related Work MLLMs for Scientific Discovery Large language models (LLMs) ( 39 ; 15 ; 32 ; 33 ; 7 ) are increasingly being extended to scientific discovery through multimodal large language models (MLLMs) that incorporate diverse biological modalities, including molecules ( 43 ) , proteins ( 48 ) , and cells ( 8 ) . In the protein domain, prior works have modeled amino-acid sequences ( 49 ; 35 ) or jointly with structures ( 16 ; 1 ; 40 ; 47 ; 42 ; 14 ) . Single cell LLMs learn the scRNA-seq representations ( 11 ; 23 ; 12 ) or further incorporate histology information ( 27 ; 5 ) . On the other hand, molecular LLMs have been built on 1D string representations such as SMILES ( 44 ) and SELFIES ( 26 ) ( 6 ; 10 ) , 2D molecular graphs ( 30 ; 13 ; 3 ; 52 ; 34 ) , 3D structures ( 28 ; 17 ) , and joint 2D-3D representations ( 24 ) . Despite this progress, biological MLLMs are limited to a single modality, hindering their ability to solve cross-modal scientific problems. Model Merging Model merging aims to fuse knowledge from multiple specialist models with minimal additional data or training by combining them directly in parameter space. Existing methods mainly leverage the parameter space signals to guide merging. Magnitude-based methods include Task Arithmetic ( 22 ) , Consensus Merging ( 41 ) , and PCB-Merging ( 9 ) ; sign-based methods include TIES-Merging ( 50 ) and EMR-Merging ( 20 ) ; and LS-Merge ( 38 ) performs merging in a learned latent space over parameters. Beyond such static merging, test-time adaptation methods dynamically adjust coefficients using unlabeled test data, as in AdaMerging ( 51 ) and Twin-Merging ( 31 ) . However, these methods still rely on parameter space signals,