Kongzi: A Historical Large Language Model with Fact Enhancement
Jiashu Yang, Ningning Wang, Yian Zhao, Chaoran Feng, Junjia Du, Hao Pang, Zhirui Fang, Xuxin Cheng · 2025
Read Kongzi: A Historical Large Language Model with Fact Enhancement by Jiashu Yang, Ningning Wang, Yian Zhao, Chaoran Feng, Junjia Du, Hao Pang, Zhirui Fang, Xuxin Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09488.txt.
Introduction
Jiashu Yang and co-authors build Kongzi, a language model tuned for history. Their 2025 paper targets a weak spot: models often invent facts during long reasoning chains or under tricky prompts, which poisons historical work. Kongzi counters with training that rewards factual answers.
The recipe adds continued pre-training on historical text, fine-tuning, and a reinforcement stage with a fact-based reward. That reward compares entities and relations in each answer against references through graph similarity. Two sizes, a small and a seven-billion-parameter model, were judged on historical questions and a Chinese history benchmark. The larger model earned the top accuracy ratings from three judge models. Training lifted scores for both sizes, though a general benchmark still favored stronger baselines.
Catalog record
| CREATOR | Jiashu Yang, Ningning Wang, Yian Zhao, Chaoran Feng, Junjia Du, Hao Pang, Zhirui Fang, Xuxin Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09488
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Kongzi: A Historical Large Language Model with Fact Enhancement Jiashu Yang, Ningning Wang, Yian Zhao, Chaoran Feng, Junjia Du, Hao Pang, Zhirui Fang, Xuxin Cheng Abstract The capabilities of the latest large language models (LLMs) have been extended from pure natural language understanding to complex reasoning tasks. However, current reasoning models often exhibit factual inaccuracies in longer reasoning chains, which poses challenges for historical reasoning and limits the potential of LLMs in complex, knowledge-intensive tasks. Historical studies require not only the accurate presentation of factual information but also the ability to establish cross-temporal correlations and derive coherent conclusions from fragmentary and often ambiguous sources. To address these challenges, we propose Kongzi , a large language model specifically designed for historical analysis. Through the integration of curated, high-quality historical data and a novel fact-reinforcement learning strategy, Kongzi demonstrates strong factual alignment and sophisticated reasoning depth. Extensive experiments on tasks such as historical question answering and narrative generation demonstrate that Kongzi outperforms existing models in both factual accuracy and reasoning depth. By effectively addressing the unique challenges inherent in historical texts, Kongzi sets a new standard for the development of accurate and reliable LLMs in professional domains. † † * Equal contribution. ✉ Corresponding authors. 1 Introduction Recent advancements in large language models have achieved significant breakthroughs ranging from text generation to complex reasoning ( Plaat et al. 2024 ; DeepSeek-AI et al. 2025 ) . Although LLMs have achieved strong performance across general domains, their application to historical tasks remains a challenge. Historical text generation demands both factual accuracy and the ability to infer complex relationships from fragmented evidence ( Cao et al. 2024 ) .However, existing models often suffer from hallucinations (generating factually inaccurate content), particularly in long-chain reasoning ( Huang et al. 2025 ; Chen et al. 2024a ) , which limits their reliability in real-world historical analysis. To mitigate these limitations, existing methods to enhance LLMs—such as reinforcement learning from human feedback (RLHF) ( Ouyang et al. 2022 ) and chain-of-thought (CoT) fine-tuning ( Chen et al. 2024b ; Kim et al. 2023 ) . These approaches primarily target general-purpose tasks, focusing on aligning model outputs with human preferences or emulating human-like reasoning processes. However, they exhibit notable limitations when applied to domains with high factual precision requirements. For instance, RLHF often lacks domain-specific factual constraints, resulting in outputs that may be fluent and preference-aligned, but lack logical consistency with verified facts. Similarly, while CoT fine-tuning enhances reasoning ability, it does not inherently prevent the generation of factually inaccurate content. These limitations underscore the need for a domain-adaptive training framework for the historical domain—where factual reliability and deep, context-aware reasoning are essential. Therefore, in this work, we propose Kongzi , a reasoning LLM specifically tailored for historical analysis tasks, capable of better analyzing and answering historical questions while maintaining higher factual accuracy and consistency. We conduct a comprehensive evaluation across multiple dimensions, with results confirming the robustness and effectiveness of our proposed model. To summarize, the main contributions of this paper are as follows: • Continued Pre-training: We curated a diverse corpus of historical texts and conducted continued pretraining to adapt the base model to the historical domain. This step ensures enhanced domain-specific knowledge for smaller models. • CoT Data Generation: We classified initial data into factual statements and reasoning tasks, generating high-quality CoT data to enhance the model’s reasoning ability through two-stage supervised fine-tuning (SFT). This approach enables the model to perform deep contextual reasoning, a critical capability for historical applications. • Fact-Aware RL: We proposed a novel reward function that incorporates factual rewards into the RL framework, ensuring the model generates content that is both accurate and contextually coherent. 2 Related Work 2.1 Reasoning in Large Language Models Reasoning constitutes a fundamental capability of large language models , enabling them to solve complex tasks by decomposing problems into logical steps. Early reasoning approaches primarily relied on prompt engineering and few-shot learning ( Yu et al. 2023 ) , guiding models to generate intermediate reasoning steps through carefully designed prompts. While these methods excelled in handling simple tasks, they often struggled with more complex problems due to their dependence on static prompts and limited context. Recent research has explored dynamic reasoning frameworks, such as self-consistent decoding ( Wang et al. 2023 ) and scratchpad prompting ( Nye et al. 2021 ) , which allow models to iteratively refine their reasoning processes. However, these approaches were primarily designed for general-domain tasks and did not fully address the unique challenges of historical reasoning, such as dealing with ambiguous or incomplete information. For instance, the DeepSeek-R1 model, trained through pure RL, demonstrated significant improvements in reasoning capabilities, even surpassing traditional SFT methods in certain aspects ( DeepSeek-AI et al. 2025 ) . Additionally, the ReasonFlux framework achieved remarkable performance enhancements in complex mathematical reasoning tasks through hierarchical RL and structured template libraries ( Yang et al. 2025 ) . 2.2 Chain-of-Thought Fine-tuning Chain-of-Thought fine-tuning has emerged as a crucial technique for enhancing the reasoning capabilities of LLMs ( Kim et al. 2023 ) . By generating intermediate reasoning steps, CoT enables models to break down complex problems into simpler subtasks, resulting in more accurate and interpretable solutions. For example, zero-shot CoT demonstrated that LLMs can generate reasoning chains even without explicit fine-tuning, while CoT distillation ( Wadhwa et al. 2024 ) showed that high-quality CoT data can significantly boost model performance. Despite these advancements, most CoT methods have primarily focused on general-domain tasks, such as mathematical reasoning and commonsense question answering, without adequately considering the specific needs of historical reasoning. In this study, we extend CoT to the historical domain by generating high-quality reasoning data tailored to historical contexts, enabling deeper and more accurate reasoning. The ReasonFlux framework, through its hierarchical RL and structured template libraries, has demonstrated significant performance improvements in complex reasoning tasks (e.g., mathematics and programming) by decomposing problems into smaller, more manageable subtasks ( Yang et al. 2025 ) . 2.3 Reinforcement Learning for Reasoning RLHF has been widely used to align LLMs with human preferences ( Ouyang et al. 2022 ; Wu et al. 2025 ; Yu et al. 2024 ) , particularly in tasks requiring complex reasoning. By incorporating human feedback into the training process, RLHF enables models to generate more coherent and contextually appropriate responses. However, traditional RLHF frameworks often lack mechanisms to ensure factual consistency or multi-turn reasoning, which are crucial for historical applications. Recent studies ( Rafailov et al. 2024 ; Schulman et al. 2017 ; Zhong et al. 2025 ; Ramesh et al. 2024 ; Tang et al. 2025 ) have explored fact-aware RL where models are rewarded for generating factually accurate content, but these approaches typically focus on single-turn tasks and do not address the challenges of lengthy historical narratives. Our work introduces a novel fact-aware RL framework that evaluates the accuracy of each generated segment, ensuring consistent factual correctness throughout the generation process. For instance, the DeepSeek-R1-Zero model ( DeepSeek-AI et al. 2025 ) , trained solely using RL, demonstrated robust reasoning capabilities by employing accuracy and format rewards (e.g., code compilation results and deterministic system evaluations). 3 Method In this section, we first describe the data gathering