Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems

Stefano Cirillo, Domenico Desiato, Giuseppe Polese, Giandomenico Solimando · 2026

Read Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems by Stefano Cirillo, Domenico Desiato, Giuseppe Polese, Giandomenico Solimando (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23618.txt.

Description

Cirillo and colleagues benchmark Google Embeddings 2 against five open-source rivals for multilingual dense retrieval and retrieval-augmented generation. Google's Vertex-AI-hosted bi-encoder, with 2048-token context and task-type conditioning, ranks first on every task, scoring 0.638 average nDCG on BEIR and 0.282 on an Italian RAG bench. But at 231.6 milliseconds median latency it runs about 14 times slower than the fastest local models. Multilingual-E5-large nearly matches it on Italian at 31 milliseconds, while LaBSE trails badly at 0.188, and all models saturate at 32-token chunks.

Catalog record

CREATORStefano Cirillo, Domenico Desiato, Giuseppe Polese, Giandomenico Solimando
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44319
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23618

Browse the full library · About the Tower of Babel