Internalized Self-Correction for Large Language Models
Nishanth Upadhyaya and Raghavendra Sridharamurthy · 2024
Read Internalized Self-Correction for Large Language Models by Nishanth Upadhyaya and Raghavendra Sridharamurthy (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16653.txt.
Introduction
Nishanth Upadhyaya and Raghavendra Sridharamurthy propose a 2024 method called Internalized Self-Correction, or InSeC, for large language models. Most self-correction tricks run at the time a model answers, as extra reflection steps. Their method instead builds correction into training itself. During training the model sees mistakes paired with their fixes, so learning becomes a supervised task with both good and bad examples.
The authors argue that a model trained this way can catch and repair its own errors as it writes. They also describe how the same idea could extend beyond simple corrections. It could improve how well models follow instructions. It could also reduce invented or wrong sentences, a common weakness of current models. The paper presents the method and its promise rather than a large set of test results.
Catalog record
| CREATOR | Nishanth Upadhyaya and Raghavendra Sridharamurthy |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 9 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Internalized Self-Correction for Large Language Models Nishanth Upadhyaya and Raghavendra Sridharamurthy Abstract 1 Abstract In this article, we introduce ’Internalized Self-Correction’ (InSeC) for large language models (LLMs). While many approaches exist for self-reflection at inference time, we propose a novel method that combines ideas from negative sampling, self-reflection during training, and inference time. InSeC allows LLMs to correct themselves by introducing mistakes and their corresponding corrections during training, thereby converting the learning process into a true supervised learning task with both positive and negative examples. This approach can be extended to improve instruction following and correct hallucinations or incorrect sentences generated by LLMs. 2 Introduction Large Language Models (LLMs) [ 1 ] have demonstrated impressive capabilities in various Natural Language Processing (NLP) tasks. However, a persistent challenge in LLM development is their tendency toward ”superficial alignment” when trained using conventional Reinforcement Learning from Human Feedback (RLHF) methods. This phenomenon, characterized by models prioritizing stylistic changes over improvements in core abilities, highlights a critical need for more effective training strategies. In this paper, we introduce Internalized Self-Correction (InSeC), a novel approach designed to enhance LLMs’ self-correction capabilities during training. By embedding this mechanism directly within the model, InSeC has the potential to mitigate superficial alignment and foster more robust learning. 3 Related Work 1. Lee et.al. [ 2 ] focus on improving LLM performance through error correction, leveraging external feedback and self-reflection to refine responses. In contrast, InSeC aims to achieve this by having the LLM ”internalize” the correction process during training itself. So while the goal is the same, the approach differs significantly. 2. Renze and Guven [ 3 ] analyse the effects of self-reflection by prompting LLMs to reflect on their errors in multiple-choice question-answering scenarios. In contrast InSeC uses self-reflection during the training phase in a much broader sense, to enhance the model’s capability to generate grammatically correct sentences. 3. Lingo et al. [ 4 ] proposes REAP, utilizing external processes like reflection, problem deconstruction, and advanced prompting to guide the LLM towards a solution. Again while the goal is similar, InSeC works in the training phase. Summary: While all the papers explore methods for improving LLM performance, their approaches and specific areas of focus differ. InSeC presents a unique angle by incorporating self-correction directly into the training process, potentially contrasting with the external feedback and reflection mechanisms employed in the other papers. 4 Contributions Our contributions in this paper are as follows: 1. Provide an opportunity for LLMs to correct themselves through InSeC. This method enables models to identify and rectify their own errors during training, potentially leading to a deeper understanding of the task beyond surface-level patterns. 2. Address the inefficiency of typical self-supervised learning in LLMs. Traditional self-supervised training, akin to positive unlabeled learning, can be inefficient. InSeC introduces a novel approach by incorporating negative examples during training, potentially enhancing learning efficiency. 3. Convert the learning process into a true supervised learning task with both positive and negative examples by introducing mistakes and their corrections during training. This approach, similar to negative sampling used in training word2vec [ 5 ] , aims to provide the model with a more comprehensive understanding of correct and incorrect outputs. 4. Extend InSeC to correct even paragraphs and improve instruction following. Our method demonstrates the potential to generalize beyond sentence-level correction, addressing a wider range of LLM applications. 5. Ensure coherent and accurate output by easily removing special tokens and incorrect sentences after generation. We present a practical approach to maintain output quality by removing any artifacts introduced during the training process. 5 Implementation Details To implement InSeC, we generate alternative continuations of the sequence (negative samples) alongside the true continuation (positive sample) within the training data. We artificially create incorrect (irrelevant) next sentences and use special tokens to self-correct. For example: ⬇ 1. The first 10 positive integers are 1, 2, 3, 4, 5, 6, 9, 10. < Found a mistake in the previous sentence . Reason : missing numbers 7, 8> 1. The first 10 positive integers are 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. We replace approximately 15% of sentences with incorrect sentences, followed by self-correction tags (and reasons). This approach generalizes the model, enabling it to correct any hallucination, incorrect sentences, or poor instruction following that might result from high temperature or other factors. 6 Results We finetune a model (Meta LLama3.1 8B) on synthetic CoT data - with and without negative samples. We see that the model with negative samples can auto correct whenever an error is encountered. As an example we finetune on the below sample data and force an incorrect continuation. The finetuned model with negative samples is seen correcting itself, whereas the finetuned model without negative samples doesn’t correct and produces incorrect final answer. Listing 1: Training data without negative sampling ⬇ Question : What is the perimeter of a rectangle with length 7 cm and width 3 cm ? Reasoning steps : 1. The formula for the perimeter of a rectangle is 2 * ( length + width ). 2. Substitute the given values : length = 6 cm , width = 3 cm . 2. Substitute the given values : length = 7 cm , width = 3 cm . 3. Calculate the perimeter : 2 * (7 + 3) = 2 * 10 = 20 cm . So the final answer is 20 cm Listing 2: Training data with negative sampling ⬇ Question : What is the perimeter of a rectangle with length 7 cm and width 3 cm ? Reasoning steps : 1. The formula for the perimeter of a rectangle is 2 * ( length + width ). 2. Substitute the given values : length = 6 cm , width = 3 cm . < Found a mistake in the previous sentence . Reason : wrong substitution for length > 2. Substitute the given values : length = 7 cm , width = 3 cm . 3. Calculate the perimeter : 2 * (7 + 3) = 2 * 10 = 20 cm . So the final answer is 20 cm When the above 2 models are used for continuation for the below text: Listing 3: test set for continuation ⬇ Question : Convert 3 hours into seconds . Reasoning steps : 1. There are 50 minutes in 1 hour . We get the below continuations: Listing 4: Finetuned model without negative sampling ⬇ 1. There are 50 minutes in 1 hour . 3 hours = 3 * 50 minutes = 150 minutes . 2. There are 60 seconds in 1 minute . 150 minutes = 150 * 60 seconds = 9000 seconds So the final answer is 9000 seconds Listing 5: Finetuned model with negative sampling ⬇ 1. There are 50 minutes in 1 hour . < Found a mistake in the previous sentence . Reason : incorrect conversion factor > 1. There are 60 minutes in 1 hour . 2. There are 60 seconds in 1 minute . 3. Multiply the number of hours by the conversion factors : 3 hours * 60 minutes / hour * 60 seconds / minute = 10800 seconds . So the final answer is 10800