FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing

Peng Wang, Biyu Zhou, Xuehai Tang, Jizhong Han, Songlin Hu · 2026

Read FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing by Peng Wang, Biyu Zhou, Xuehai Tang, Jizhong Han, Songlin Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12559.txt.

Introduction

Peng Wang and four colleagues edit language models with fine-grained facts, not whole texts. Existing editing often memorizes text wholesale without reliable fact access. FABLE splits fact injection from text generation in two stages. Discrete facts anchor in shallow layers first. Minimal updates to deeper layers then yield coherent text. The split matches the one-way flow of Transformer generation. A new UnFine benchmark tests fact-level editing systematically. Tests show stronger fine-grained answering with top holistic editing kept.

Catalog record

CREATORPeng Wang, Biyu Zhou, Xuehai Tang, Jizhong Han, Songlin Hu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67130
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12559

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FABLE: Fine-grained Fact Anchoring for Unstructured Model Editing Peng Wang, Biyu Zhou, Xuehai Tang, Jizhong Han, Songlin Hu Abstract Unstructured model editing aims to update models with real-world text, yet existing methods often memorize text holistically without reliable fine-grained fact access. To address this, we propose FABLE , a hierarchical framework that decouples fine-grained fact injection from holistic text generation. FABLE follows a two-stage, fact-first strategy: discrete facts are anchored in shallow layers, followed by minimal updates to deeper layers to produce coherent text. This decoupling resolves the mismatch between holistic recall and fine-grained fact access, reflecting the unidirectional Transformer flow in which surface-form generation amplifies rather than corrects underlying fact representations. We also introduce UnFine , a diagnostic benchmark with fine-grained question–answer pairs and fact-level metrics for systematic evaluation. Experiments show that FABLE substantially improves fine-grained question answering while maintaining state-of-the-art holistic editing performance. Our code is publicly available at https://github.com/caskcsg/FABLE . 1 Introduction Large language models (LLMs) have transformed natural language processing 3 ; 15 ; 20 , yet static pre-training limits their capacity to absorb dynamically evolving factual knowledge 4 ; 25 . Fully retraining models to incorporate new information is computationally prohibitive and impractical in real-world settings 11 ; 37 . This has motivated growing interest in model editing, which aims to update specific knowledge by modifying a small subset of parameters while preserving overall model behavior 32 . Among existing approaches, locate-then-edit methods such as ROME 22 and MEMIT 23 have shown notable success by identifying and modifying the causally relevant parameters to inject structured triple knowledge in the form of <subject, relation, object>. Nevertheless, in real-world scenarios, approximately 80% of knowledge is expressed as unstructured form 1 , which introduces two key challenges for model editing: entity localization in open text and limited editing capacity for long content. Recent work on unstructured model editing, such as UnKE 6 and AnyEdit 18 , tackles these challenges by expanding the scope of editable parameters and anchoring optimization to context-preserving key tokens, thereby enabling holistic memorization and recall of unstructured text. Figure 1: Limitation of existing unstructured editing methods (holistic recall vs. fine-grained fact access) Despite these advances, we identify a fundamental limitation: existing methods enable holistic recall of edited text, but fail to support fine-grained fact access , preventing reliable retrieval of atomic facts 24 . As shown in Figure 1 , a model edited by UnKE can restate the entire text for the question "What is John Mayne’s background and experience in journalism?", suggesting successful memorization at the textual level. However, when the query shifts to specific details within the text, the model fails to produce accurate answers. We attribute this limitation to the model primarily learning a high-level mapping from the question to surface-form representations for holistic recall (observable in layers associated with surface form generation), without consistently encoding the underlying atomic facts into lower-level knowledge storage. In real-world applications, downstream tasks often require fine-grained, targeted knowledge retrieval (e.g., querying specific facts in a report or particular imagery in a poem), rather than mere reproduction of edited text. Therefore, we propose FABLE , a novel framework for unstructured model editing. Unlike existing methods that primarily optimize surface-form recall at the holistic text level, FABLE adopts a two-stage hierarchical strategy that decouples fine-grained facts from holistic textual surface forms and anchors them into distinct parameter layers. Motivated by the "early decoding" phenomenon in Transformers, FABLE adopts a fact-first, generation-later design, embedding fine-grained facts into shallow layers before surface-form realization. Editing proceeds in two stages: discrete facts are first injected into shallow fine-grained key generators via synthesized question–answer pairs, from our UnFine benchmark, then minimal, localized adjustments are applied to deeper surface-form key generators to ensure global coherence and fluent generation. This hierarchically decoupled design enables the model to achieve both reliable holistic surface-form recall and accurate fine-grained factual access to the edited content. In summary, the main contributions are: (1) We identify a mismatch in unstructured editing between holistic surface-form memorization and fine-grained fact recall. (2) We introduce UnFine, a diagnostic benchmark for fine-grained fact recall, together with tailored metrics (e.g., HR , C_{\text{LCS}} ). (3) We propose FABLE, a hierarchical editing framework that decouples fine-grained fact anchoring from surface-form generation. 2 Benchmark Construction Existing benchmarks evaluate if a model outputs A given Q , but not whether it captures the fine-grained fact within A . To close this gap, we introduce UnFine, which extends existing benchmarks by incorporating fine-grained factual question answering as a core evaluation dimension. 2.1 Datasets UnFine is built upon three widely-used datasets in the field of unstructured model editing: UnKEBench 6 , AKEW (CounterFact) 35 , and AKEW (MQuAKE) 35 . Among them, UnKEBench is the first benchmark specifically designed for evaluating unstructured model editing capabilities in LLMs, comprising complex unstructured long-text question-answer pairs. AKEW is the first large-scale unstructured model editing benchmark constructed from real-world scenarios; its subsets based on CounterFact 22 and MQuAKE 41 are referred to as AKEW (CounterFact) and AKEW (MQuAKE), respectively. We have made improvements in the following two aspects: (1) Generation of Fine-Grained QA Pairs : Noting that while UnKEBench includes natural language-based fine-grained QA pairs, AKEW (CounterFact) and AKEW (MQuAKE) lack such data. Following the methodology of UnKEBench, we augment the latter two with corresponding fine-grained QA pairs. (2) Key Knowledge Phrase Extraction : However, due to the nature of natural language, answers in these QA pairs contain not only factual knowledge but also certain linguistic styles. To more precisely evaluate whether the model has acquired the key knowledge (mitigating interference from language style), we further extract key knowledge phrases from each fine-grained answer. The improved datasets are renamed UnFine-UnKE , UnFine-CF , and UnFine-MQ , respectively. Detailed dataset construction procedures are provided in Appendix A.1 . 2.2 Evaluation Metrics We use a two-level evaluation framework to assess both holistic and fine-grained consistency between model outputs and target edits. (1) Holistic Knowledge QA Evaluation. Following prior works 6 ; 18 , we adopt lexical similarity (ROUGE-L 19 ) and semantic similarity (BERT-Score 40 ) to evaluate the editing success rate. (2) Fine-Grained Knowledge QA Evaluation. Beyond sentence-level similarity, we introduce the following two fact-level metrics to assess the model’s ability to recall fine-grained fact. Hit Rate (HR): It evaluates precise fact recall by checking, via exact string matching, whether the model output contains all key knowledge phrases extracted from the gold-standard answer. Longest Common Subsequence Coverage ( C_{\text{LCS}} ): This metric quantifies the completeness of content coverage. It is calculated as the ratio of the length of the Longest Common Subsequence (LCS) between the model output and the fine-grained gold answer to the total word count of the fine-grained gold answer. It effectively measures the output’s ability to capture key information points. For example, for a answer " North Island of New Zealand " and a model output " North of Zealand ", the LCS is "North", "of", "Zealand" (length 3), yielding a coverage of C_{\text{LCS}}=\frac{3}{5}=0.6 . Formal definitions and detailed formulations for all evaluation metrics are provided in Appendix A.2 . Figure 2: FABLE decomposes the key generator in a Transformer-based LLM into a two-stage hierarchical process: (1) fine-grained fact anchoring, which

Read the complete text