ProtoCol : Late Interaction Retrieval for Protein Homolog Search

Gabrielle Cohn, Rohan Gumaste, Minh Hoang, Vihan Lakshman · 2026

Read ProtoCol : Late Interaction Retrieval for Protein Homolog Search by Gabrielle Cohn, Rohan Gumaste, Minh Hoang, Vihan Lakshman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29158.txt.

Introduction

Gabrielle Cohn and colleagues introduce ProtoCol, a model for remote protein homolog search in the twilight zone, where broad sequence likeness is weak and alignment loses power. Instead of pooling protein model vectors into one vector, ProtoCol keeps proteins as sets of residue vectors and scores candidates with MaxSim late interaction, in the style of ColBERT. Codes stay ready to compute, and proteins are encoded alone. On SCOPe superfamily and Pfam clan benchmarks, ProtoCol beats composition, alignment, pooled, and single-vector baselines, backing late interaction as a retrieval layer.

Catalog record

CREATORGabrielle Cohn, Rohan Gumaste, Minh Hoang, Vihan Lakshman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19134
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29158

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProtoCol : Late Interaction Retrieval for Protein Homolog Search Gabrielle Cohn, Rohan Gumaste, Minh Hoang, Vihan Lakshman Abstract Protein homology search underlies function annotation, structure prediction, and evolutionary analysis, but remains challenging in the “twilight zone,” where global sequence similarity is weak and classical alignment methods lose sensitivity. Protein language models provide context-aware representations that could improve alignment sensitivity in this regime. However, prior protein embedding-based retrieval pipelines often pool these representations into a single vector, potentially obscuring local motifs, domains, or conserved residues that reveal remote homology. We introduce ProtoCol , a model which represents proteins as sets of residue embeddings and uses ColBERT-style late interaction to test whether residue-level comparison improves homolog retrieval. ProtoCol encodes proteins independently, keeps candidate representations pre-computable, and scores candidates with MaxSim over residue embeddings. On SCOPe superfamily and Pfam clan benchmarks, ProtoCol outperforms sequence-composition, alignment-based, pooled PLM, and trained single-vector baselines, supporting late interaction as an effective retrieval layer for remote homology search. † † affiliationnotice: Equal contribution 1 Introduction Homologous proteins descend from a common ancestral sequence and often preserve related functions or structures. Detecting such relationships is central to computational biology, supporting function annotation, structure prediction, and evolutionary analysis. This task is especially difficult for remote homologs, whose sequences may diverge so substantially that direct sequence-level similarity becomes weak. In this “twilight zone,” classical sequence-alignment methods can miss relationships that remain evident through conserved motifs, domains, or structural constraints ( 1 ; 6 ; 25 ) . Protein language models (PLMs) offer a promising sequence-only alternative because they produce contextual residue embeddings that encode structural and evolutionary signal ( 18 ; 19 ) . However, many PLM-based retrieval pipelines pool these embeddings into a single protein vector and compare proteins by cosine similarity ( 11 ) . This is efficient, but can dilute local evidence such as a conserved motif, domain, or small set of structurally constrained residues that may be decisive for remote homology. This pooling bottleneck motivates our central question: does homolog retrieval improve when proteins are represented as sets of residue embeddings and compared through late interaction? We hypothesize that residue-level scoring can preserve local evolutionary evidence while keeping database proteins independently encodable. We test our hypothesis with ProtoCol (“proteins” with “ColBERT”) (see figure A.1 ), a late-interaction retrieval model for protein homology search. ProtoCol adapts the ColBERT retrieval paradigm ( 15 ) to protein sequences by representing each protein as residue-level PLM embeddings and comparing proteins through lightweight residue-level interactions 1 1 1 Our code is available at https://github.com/gabriellecohn/ProtoCol . We evaluate ProtoCol on SCOPe superfamily and Pfam clan retrieval benchmarks using baselines chosen to isolate sequence composition, alignment sensitivity, PLM scale, contrastive fine-tuning, and late interaction. Across both settings, ProtoCol performs best, supporting residue-level late interaction as an effective retrieval layer for remote homology search. 2 Related Work Alignment-based homology search. Classical homology search relies on residue-level sequence comparison. BLAST ( 1 ) performs fast pairwise local alignment, while profile-based methods such as PSI-BLAST and HMMER ( 6 ) improve sensitivity using multiple sequence alignments. MMseqs2 ( 25 ) further improves the speed–sensitivity tradeoff for large-scale search. These methods remain strong baselines, but their reliance on detectable sequence similarity limits sensitivity for highly diverged homologs. Structure-based search addresses some of these limitations by comparing proteins closer to their conserved three-dimensional form. Foldseek ( 26 ) , for example, discretizes protein structures into residue-level alphabets and aligns the resulting strings. Such methods demonstrate the value of local matching beyond raw sequence identity, but require structural information to work. Our work instead asks whether sequence-only PLM representations can support an analogous retrieval primitive. Protein language models for retrieval. Large protein language models such as ProtTrans ( 7 ) and ESM ( 18 ) learn contextual residue embeddings from unaligned sequence databases. These embeddings have been used for retrieval via mean-pooling into a single protein-level vector and ranking candidates by cosine similarity ( 24 ; 11 ) . This bi-encoder-style setup supports efficient nearest-neighbor search, analogous to dense retrieval methods in NLP such as Sentence-BERT ( 21 ) and Dense Passage Retrieval ( 14 ) , but discards the token-level structure of PLM embeddings. Methods such as PLMAlign and pLM-BLAST ( 19 ; 13 ) point to the utility of residue-level PLM embeddings for remote homology detection by using PLM representations in alignment-oriented comparison. Late-interaction retrieval. Late-interaction models occupy a middle ground between efficient global-vector retrieval and expensive pairwise interaction models. ColBERT ( 15 ) represents each text passage as token embeddings and scores a query–document pair by matching each query token to its most similar document token using MaxSim. This preserves fine-grained matching while allowing document representations to be precomputed. Following the introduction of ColBERT, a number of works have further developed this notion of late interaction retrieval ( 23 ; 22 ; 10 ; 9 ; 16 ; 17 ; 2 ; 5 ; 8 ) . All of these prior works focus on traditional information retrieval settings. ProtoCol adapts this idea to proteins: residues replace tokens, candidate proteins replace documents, and MaxSim provides a residue-level similarity score for homolog retrieval. In contrast to alignment-oriented PLM methods, ProtoCol uses late interaction as a learned retrieval mechanism rather than computing an explicit alignment path. 3 Methodology This section defines the components of ProtoCol : a residue-level PLM encoder, MaxSim scoring, and a contrastive objective based on weak homology labels. Together, these choices test whether homology retrieval benefits from preserving residue embeddings through the scoring layer rather than compressing each protein before comparison. Dataset construction, baselines, and evaluation protocol are described in Section 4 . Late-interaction encoder. We instantiate ColBERT-style late interaction over a protein language model. Let x=(x_{1},\dots,x_{T}) be a protein sequence, and let h_{t}\in\mathbb{R}^{H} denote the contextual residue embedding produced at position t by an ESM-2 backbone f_{\theta} ( 18 ) . We attach a linear projection W\in\mathbb{R}^{D\times H} followed by L2 normalization: e_{t}=\frac{Wh_{t}}{\lVert Wh_{t}\rVert_{2}}\in\mathbb{R}^{D}, (1) with D=128 in all experiments. Unless otherwise stated, the backbone is ESM-2 35M ( H=480 , 12 layers). To keep the trainable footprint modest, we freeze the embedding layer and lower transformer blocks and fine-tune only the final three transformer layers, the post-stack LayerNorm, and W . This yields roughly 8.4M trainable parameters out of 33.6M. MaxSim scoring. A protein is represented by its variable-length set of L2-normalized residue embeddings E=\{e_{1},\dots,e_{T}\} . MaxSim operationalizes residue-level retrieval by allowing each query residue to contribute its strongest match anywhere in the candidate protein. Given query embeddings E^{q} and candidate embeddings E^{d} , we score the pair using the asymmetric MaxSim operator of 15 , \mathrm{MaxSim}(E^{q},E^{d})=\sum_{i=1}^{T_{q}}\max_{j\in[T_{d}]}\langle e^{q}_{i},e^{d}_{j}\rangle. (2) Because embeddings are L2-normalized, each inner product is a cosine similarity. Padding positions are masked in both the inner maximum and outer sum. Contrastive training. Training shapes the embedding space so homologs receive high late-interaction scores and in-batch non-homologs receive lower scores. Each training pair consists of an anchor protein a and a positive protein p sampled from the same superfamily for SCOPe ( 3 ) or from the same clan for Pfam ( 20 ) . For a batch of B pairs, we form S\in\mathbb{R}^{B\times B} with S_{ij}=\mathrm{MaxSim}(E^{a_{i}},E^{p_{j}}) , treat off-diagonal entries as in-batch negatives,

Read the complete text