Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models

e Campregher [email protected], u Chen [email protected], er Hoffman [email protected], a Heuss [email protected] · 2025

Read Tracing Facts or just Copies? A critical investigation of the Competitions of Mechanisms in Large Language Models by e Campregher [email protected], u Chen [email protected], er Hoffman [email protected], a Heuss [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-11809.txt.

Description

Dante Campregher, Yanxu Chen, Sander Hoffman and Maria Heuss revisit how language models choose between learned facts and facts given in context. Three recent studies used mechanistic tools to study that contest, with unlike claims about attention heads. Their reproduction study tests whether head strength predicts how often the model gives the learned fact.

Heads that favour factual output seem to work by general copy suppression, not by blocking only false context. Boosting them can even block correct facts. Head behaviour also varied by domain, and larger models showed more specialised, category-sensitive patterns. The findings urge care when single head roles are claimed across all kinds of text.

Catalog record

CREATORe Campregher [email protected], u Chen [email protected], er Hoffman [email protected], a Heuss [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-11809

Browse the full library · About the Tower of Babel