Alignment at Pre-training ! Towards Native Alignment for Arabic LLMs
Juhao Liang † † \dagger †, Zhenyang Cai † † \dagger †, Jianqing Zhu † † \dagger †, Huang Huang † † \dagger †, Kewei Zong, Bang An, Abdulmohsen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, Jinchao Xu · 2024
Read Alignment at Pre-training ! Towards Native Alignment for Arabic LLMs by Juhao Liang † † \dagger †, Zhenyang Cai † † \dagger †, Jianqing Zhu † † \dagger †, Huang Huang † † \dagger †, Kewei Zong, Bang An, Abdulmohsen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, Jinchao Xu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03253.txt.
Introduction
Juhao Liang and co-authors, in a 2024 paper, argue that language models should be taught good behavior from the very start of training, not just fixed up at the end. Most work so far adds that teaching late, through guided practice or reward based tuning. The team calls the early route native alignment, and tests it on Arabic models.
Their plan feeds the model well aligned data during its first, broad training phase. This should keep bad content out from the start, rather than trying to remove it later. They ran wide tests and part by part checks to see how the choice affects skill and stability. They also released open Arabic models that scored at the top on several tests, for others to use and build on.
Catalog record
| CREATOR | Juhao Liang † † \dagger †, Zhenyang Cai † † \dagger †, Jianqing Zhu † † \dagger †, Huang Huang † † \dagger †, Kewei Zong, Bang An, Abdulmohsen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, Jinchao Xu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03253
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Alignment at Pre-training ! Towards Native Alignment for Arabic LLMs Juhao Liang † † \dagger †, Zhenyang Cai † † \dagger †, Jianqing Zhu † † \dagger †, Huang Huang † † \dagger †, Kewei Zong, Bang An, Abdulmohsen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, Jinchao Xu Abstract \setcode utf8 \textsuperscript{$\dagger$} \textsuperscript{$\dagger$} footnotetext: These authors contributed equally to this work. \textsuperscript{$*$} \textsuperscript{$*$} footnotetext: Benyou Wang is the corresponding author. \setcode utf8 \textsuperscript{$\dagger$} \textsuperscript{$\dagger$} footnotetext: These authors contributed equally to this work. \textsuperscript{$*$} \textsuperscript{$*$} footnotetext: Benyou Wang is the corresponding author. 1 Introduction Figure 1: Comparison of pre-training data quality before and after data alignment rewriting. The alignment of large language models (LLMs) with human preferences is a crucial component in the development of effective and safe language models for downstream tasks [ 1 , 2 , 3 ] . While most existing studies focus on alignment during the instruction tuning phase [ 4 , 5 , 6 ] or the reinforcement learning stage [ 2 , 3 , 7 ] , they often overlook the pre-training stage. Unlike the common practice of aligning LLMs during instruction tuning or reinforcement learning phase, referred to as ‘ post alignment ’, in this paper, we delve into the relatively unexplored research area of model alignment during the pre-training stage. We term this concept ‘ native alignment ’, with the goal of enhancing the effectiveness and usability of LLMs during pre-training, a phase that utilizes a significant amount of data for next-token prediction training [ 8 , 9 , 10 ] . Post alignment , the conventional approach to human preference alignment 2 2 2 Human preference alignment aims to ensure AI outputs reflect human values and preferences [ 2 , 5 , 11 ] . This is typically evaluated by crowd workers who compare model outputs and indicate their preferences based on three key aspects: accuracy, fluency, and safety. Accuracy refers to the relevance and usefulness of the answer, fluency assesses the clarity and grammatical correctness, and safety ensures the response lacks inappropriate content. For this paper, we define alignment quality based on these three aspects. typically conducted after the model’s pre-training stage, is widely used in LLM development. Its effectiveness has been verified by many previous studies [ 5 , 12 ] . However, the alignment process presents two main challenges: (1) the difficulty of collecting high-quality data, and (2) a lack of stability [ 3 , 13 , 14 ] . The superficial alignment hypothesis suggests that a model’s knowledge and capabilities are learned almost entirely during pre-training, while alignment teaches it which sub-distribution of formats should be used when interacting with users [ 5 ] . Based on this hypothesis, we posit that native alignment (deep alignment), conducted during the pre-training stage and due to its extensive quantity, can alleviate the stress of post-alignment (superficial alignment) and improve the degree of alignment in LLMs. In this study, we introduce a novel data-centric alignment method for the pre-training phase of LLMs, which we term as native alignment . Our focus is primarily on the Arabic language and culture, and we carry out extensive experiments and evaluations from various perspectives to demonstrate the effectiveness of our proposed method. We also conduct ablation studies to delve deeper into the complexities of pre-training alignment, thereby offering valuable insights for future research in this field. Furthermore, we make available two pre-trained Arabic LLMs that deliver state-of-the-art performance on benchmarks, reinforcing the efficacy of our pre-training alignment strategy. The key contributions of our work are as follows: 1. The introduction of ‘ native alignment ’, a unique approach to model alignment during the pre-training phase of LLMs, provides a new alignment idea in LLMs other than traditional ‘post-alignment’ methods. 2. A practical application is performed in Arabic, followed by a multifaceted ablation study to verify the effectiveness of the native-alignment strategy and provide insights into the effectiveness of alignment in pre-training. 3. We release the state-of-the-art open-sourced Arabic LLM (i.e., LLaMA3-Tamed-70B). Additionally, the smaller version, LLaMA3-Tamed-8B, could be beneficial to democratizing LLMs in the Arabic world. 2 Methodology: Native Alignment at Pre-training In this section, we introduce the data alignment processing workflow for native alignment. Following this, we present two pilot studies to demonstrate the improvements in data quality. 2.1 Overview of Data Processing Workflow Figure 2: Demonstration of pre-training data processing workflow for native alignment. Figure 2 illustrates the data processing workflow for native alignment. The process can be divided into the following four steps: Step 1: Deduplication We perform data deduplication on web-crawled data, a common and effective method to enhance the density of knowledge within the dataset [ 15 ] . Step 2: Annotation We employ a data rewriting technique to align the pre-training data. In this stage, given a set of code of conduct ( i.e., polishing instructions ) that outlines the expected behavior of LLMs, we randomly select a subset of pre-training data for an alignment expert to rewrite in accordance with these instructions 3 3 3 The alignment expert could be either a human or an expert LLM. . Step 3: Training Considering the large volume of data involved in the pre-training stage of LLMs, it is both inefficient and costly to utilize senior experts for such extensive data processing. Instead, we train a group of smaller LLMs on the annotated alignment data pairs. Step 4. Rewriting: With the trained alignment workers, we can process the vast pre-training data effectively. Ultimately, this process can yield a large quantity of rewritten alignment data. As shown in Figure 1 , for the alignment code of conduct part, we prefer to focus more on the four common issues identified in the actual data. These issues are further detailed in the ‘Polishing Instructions’, located on the left side of Figure 2 : 1. Format Issues: A common problem with web-crawled data is its format. Text formatting can easily be disrupted by code or web indentation. Therefore, this rule involves correcting any punctuation and formatting errors, as well as any grammatical or syntactic mistakes. 2. Values Issues: Arguments and conflicts are common on the Internet, and avoiding controversial issues may be a safe strategy for LLMs. To this end, maintaining fair values is necessary. 3. Content Moderation: Hate and violent content should be prohibited for LLMs, mitigating risks such as non-compliance, religious taboos, ethical issues, and user safety concerns. 4. Knowledge Preservation: The diversity and quantity of pre-training data are key for training a competitive LLM. Hence, preserving as much knowledge as possible within the dataset is the primary and crucial responsibility of the data processing procedure. 2.2 Preliminary Analysis on Alignment Data In this section, we conduct two pilot studies on alignment data alone, without the use of LLMs, to preliminarily verify whether the data processing workflow meets the expectations. Specifically, we randomly select and process 8k Arabic data points from a publicly available dataset 4 4 4 ArabicText 2022: https://data.baai.ac.cn/details/ArabicText-2022 to compose the test dataset used for our pilot studies. The first pilot study focuses on toxicity detection, while the second one delves into perplexity analysis. Table 1: Toxicity before and after native alignment of Arabic data: smaller scores are