Col Mate : Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval

Ahmed Masry, Megh Thakkar, Patrice Bechard, Sathwik Tejaswi Madhusudhan, Rabiul Awal, Shambhavi Mishra, Akshay Kalkunte Suresh, Srivatsava Daruru, Enamul Hoque, Spandana Gella, Torsten Scholak, Sai Rajeswar · 2025

Read Col Mate : Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval by Ahmed Masry, Megh Thakkar, Patrice Bechard, Sathwik Tejaswi Madhusudhan, Rabiul Awal, Shambhavi Mishra, Akshay Kalkunte Suresh, Srivatsava Daruru, Enamul Hoque, Spandana Gella, Torsten Scholak, Sai Rajeswar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00903.txt.

Description

Ahmed Masry and colleagues present ColMate, a retrieval model for documents that mix text, layout, and images. Retrieval augmented generation needs such documents, yet most retrieval methods were copied from text only search, in how documents are encoded, trained, and scored.

ColMate changes all three parts. It pretrains with an OCR based objective, learns from a self supervised masked contrastive task, and scores matches with late interaction. That compares fine details between query and document rather than single summary vectors. On the ViDoRe V2 benchmark it improves over existing retrieval models by about three and a half percent, and it generalizes better to benchmarks outside its training domain.

Catalog record

CREATORAhmed Masry, Megh Thakkar, Patrice Bechard, Sathwik Tejaswi Madhusudhan, Rabiul Awal, Shambhavi Mishra, Akshay Kalkunte Suresh, Srivatsava Daruru, Enamul Hoque, Spandana Gella, Torsten Scholak, Sai Rajeswar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00903

Browse the full library · About the Tower of Babel