Analyzing Similarity Metrics for Data Selection for Language Model Pretraining

Dylan Sam, Ayan Chakrabarti, Afshin Rostamizadeh, Srikumar Ramalingam, Gui Citovsky, Sanjiv Kumar · 2025

Read Analyzing Similarity Metrics for Data Selection for Language Model Pretraining by Dylan Sam, Ayan Chakrabarti, Afshin Rostamizadeh, Srikumar Ramalingam, Gui Citovsky, Sanjiv Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02494.txt.

Description

Dylan Sam and five co-authors test whether standard similarity scores are good tools for choosing training data for language models. Their paper came out in 2025. Similarity scores usually come from embedding models, which turn text into numbers that capture meaning.

The team built a three-part test for any such score. They checked whether distances match how well learning carries over between examples. They trained a model on chosen data and measured real task results. Standard embeddings did poorly on these tests. Simple embeddings taken from a model trained on the same data did better, which points to a need for purpose-built tools.

Catalog record

CREATORDylan Sam, Ayan Chakrabarti, Afshin Rostamizadeh, Srikumar Ramalingam, Gui Citovsky, Sanjiv Kumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02494

Browse the full library · About the Tower of Babel