Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Models

Yu Fu, Haz Sameen Shahgir, Hui Liu, Xianfeng Tang, Qi He, Yue Dong · 2025

Read Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Models by Yu Fu, Haz Sameen Shahgir, Hui Liu, Xianfeng Tang, Qi He, Yue Dong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08202.txt.

Description

Yu Fu and co-authors show, in a 2025 paper, that a language model's built-in knowledge quietly steers its answers more as the text it reads grows longer. Models built for very long inputs are prized for using what they are given. What they already know, stored in their weights, got less attention.

The team measured how that stored knowledge shapes output as context stretches. The influence grew with length, they found. Skill at recalling built-in knowledge did not rise alongside skill at retrieving from context. Worse, sharper retrieval could crowd out recall. Their simple remedy aims to restore balance. Long-document tools, from law to research, depend on both sources behaving.

Catalog record

CREATORYu Fu, Haz Sameen Shahgir, Hui Liu, Xianfeng Tang, Qi He, Yue Dong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08202

Browse the full library · About the Tower of Babel