Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behavior

Christopher J. Anders, Henrique Da Silva Gameiro, Nico Daheim, [4pt] Mohammad Emtiyaz Khan, [1em] RIKEN Center for Advanced Intelligence Project, Tokyo, Japan, [1em] {christopher.anders,emtiyaz.khan}@riken.jp · 2026

Read Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behavior by Christopher J. Anders, Henrique Da Silva Gameiro, Nico Daheim, [4pt] Mohammad Emtiyaz Khan, [1em] RIKEN Center for Advanced Intelligence Project, Tokyo, Japan, [1em] {christopher.anders,emtiyaz.khan}@riken.jp (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23591.txt.

Description

Anders, Da Silva Gameiro, Daheim, and Khan compare two ways of tracing a language model's output back to its training data, in a 2026 paper on model interpretability. Data similarity ranks training documents by how closely their text matches the output. Data influence estimates how much each document actually shaped the model's behavior. Similarity is cheap, while influence is costly but believed more accurate. No one had compared them for tracing.

The team ranks training documents under both measures and measures how much the rankings overlap. The rankings agree strongly, but the agreement is lopsided. Documents that similarity ranks at the top receive far more consistent ranks from influence than the reverse. The pattern holds across five models, including OLMo2, Qwen3, Llama, Gemma, and GPT2. The authors exploit this: run cheap similarity first, then spend influence only on refining its top results.

Catalog record

CREATORChristopher J. Anders, Henrique Da Silva Gameiro, Nico Daheim, [4pt] Mohammad Emtiyaz Khan, [1em] RIKEN Center for Advanced Intelligence Project, Tokyo, Japan, [1em] {christopher.anders,emtiyaz.khan}@riken.jp
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.23591

Browse the full library · About the Tower of Babel