Constitution or Collapse? Exploring Constitutional AI with Llama 3-8B

Xue Zhang · 2025

Read Constitution or Collapse? Exploring Constitutional AI with Llama 3-8B by Xue Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04918.txt.

Introduction

Zhang presents a 2025 study testing Constitutional AI on a smaller language model, Llama 3 with eight billion parameters. The method lets one AI critique and train another against a written constitution, cutting human labeling. Anthropic's first build used a far larger model, so small-model behavior was open. Zhang copied the full workflow at the small scale.

Harmlessness improved a lot, with attack success on a benchmark down by about forty percent. Helpfulness fell by near ten percent in trade, echoing the first study. The final model also showed clear signs of collapse, where output quality decays through self-training rounds. Zhang warns that self-improvement may need a size floor to work safely.

Catalog record

CREATORXue Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-04918

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Constitution or Collapse? Exploring Constitutional AI with Llama 3-8B Xue Zhang Abstract As language models continue to grow larger, the cost of acquiring high-quality training data has increased significantly. Collecting human feedback is both expensive and time-consuming, and manual labels can be noisy, leading to an imbalance between helpfulness and harmfulness. Constitutional AI, introduced by Anthropic in December 2022, uses AI to provide feedback to another AI, greatly reducing the need for human labeling. However, Anthropic’s original implementation was designed for a model with around 52 billion parameters, and there is limited information on how well Constitutional AI performs with smaller models, such as Llama 3-8B. In this paper, we replicated the Constitutional AI workflow using the smaller Llama 3-8B model. Our results show that Constitutional AI can effectively increase the harmlessness of the model, reducing the Attack Success Rate in MTBench by 40.8%. However, similar to the original study, increasing harmlessness comes at the cost of helpfulness. The helpfulness metrics, which are an average of the Turn 1 and Turn 2 scores, dropped by 9.8% compared to the baseline. Additionally, we observed clear signs of model collapse in the final DPO-CAI model, indicating that smaller models may struggle with self-improvement due to insufficient output quality, making effective fine-tuning more challenging. Our study suggests that like reasoning and math ability, self-improvement is an emergent property. 1 Introduction and Motivation The rapid advancement of Large Language Models (LLMs) has revolutionized the world. LLMs have demonstrated the ability to perform human-like tasks at a professional level. On March 26th, 2023, Goldman Sachs predicted that up to 300 million jobs may be replaced by Artificial Intelligence ( Sachs 2023 ) . This year, tech companies such as OpenAI, Meta and Google invested tens of billions of dollars in this era of “AI Competition” ( OpenAI 2023 ) . As language models continue to grow larger, the cost of obtaining high-quality training data has risen significantly. In the post-training phase, we aim to align our models with human preferences using techniques like Reinforcement Learning from Human Feedback (RLHF), which requires gathering human labels. However, collecting human feedback is both costly and time-consuming, and manual labels can be noisy, resulting in a poor balance between helpfulness and harmfulness. In December 2022, Anthropic introduced a method called Constitutional AI (CAI) ( Bai et al. 2022 ) or Reinforcement Learning from AI Feedback (RL-AIF). Instead of gathering thousands of human labels, this approach only requires providing approximately 10 human-generated constitutions and asking the AI to provide feedback and label the data. This method significantly reduces the amount of human effort involved and proves to be much more cost-effective. However, Anthropic’s original implementation was for a model with an estimated 52B parameters. Currently, we are unaware of work that details how well CAI performs on smaller models, such as Llama 3 - 8B. In addition, RL-AIF in the paper relies on fine-tuning models using its own self-critic outputs, which may lead to model collapse ( Kazdan et al. 2025 ) , a phenomenon where models degenerate to be unusable when training on recursively generated data. This problem may become even worse when the model is small, since the output quality from the smaller model is not good enough, potentially leading to even stronger model collapse. Thus, we aim to replicate the Constitutional AI workflow using the smaller Llama 3-8B model and explore the effects and outcomes of this approach. 2 Related Work In this work, we primarily replicate the end-to-end flow from the Constitutional AI approach by Anthropic. However, instead of using Proximal Policy Optimization (PPO) during the Reinforcement Learning stage, as described in the original paper, we employed Direct Preference Optimization (DPO). DPO bypasses the need to train the reward model while still achieving similar high-quality results, significantly reducing the effort required. Constitutional AI ( Bai et al. 2022 ) : This method trains a harmless AI model using a self-improving approach, without requiring human-labeled data to identify harmful content. Instead, humans provide a set of rules or principles, known as ”Constitutional AI.” The training process consists of two main stages: supervised fine-tuning and reinforcement learning (RL). In the supervised stage, the model is exposed to prompts with potentially harmful content. It then generates a self-critique and revises its response, with the model being fine-tuned based on the final revised output. In the RL stage, the AI uses feedback from its own responses to train a reward model. The model is then further refined using the Policy Proximal Optimization (PPO) RL method to optimize its policy and improve the prediction of the next token. Figure 1: Constitutional AI ( Bai et al. 2022 ) Direct Preference Optimization (DPO) ( Rafailov et al. 2024 ) : This method directly optimizes a policy using preference data, rather than learning a reward model. DPO optimizes the policy through the loss function \mathcal{L}_{\textrm{DPO}}(\pi_{\theta};\pi_{\textrm{ref}})=-\mathbb{E}_{(x,y_{w},y_{\ell})\sim\mathcal{D}}\Big[\log\sigma\Big(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\textrm{ref}}(y_{w}|x)}\\ -\beta\log\frac{\pi_{\theta}(y_{\ell}|x)}{\pi_{\textrm{ref}}(y_{\ell}|x)}\Big)\Big]. (1) With DPO, two models are learned: a policy model and a reference model. The policy model is the language model used to generate responses, and its parameters are updated during the fine-tuning process. The reference model serves to constrain the policy model’s updates, ensuring that the new policy doesn’t deviate too much from the old one. Typically, the reference model is a periodically updated copy of the policy model. 3 Approach Consistent with the original paper, our training process consists of two stages: a supervised stage followed by a reinforcement learning stage. 1. Supervised Stage At this stage, we start by fine-tuning the Llama 3-8B model with an instruction-based dataset to enhance its ability to engage in conversations. We then use the Llama 3-8B to generate responses to harmful questions, which often result in toxic or harmful answers. To address this, we prompt the model to critique its own response based on a set of established principles or constitutions, allowing it to revise its answer to be more harmless. The principles are selected randomly, helping the model become more adaptable to various types of constitutions. Once the model generates a safer response, we perform supervised fine-tuning using this revised answer, which serves as the foundation for the second stage of reinforcement learning (RL). Further details are provided below. We begin by instruction-tuning the language model, using the Alpaca-GPT4 dataset to teach it how to follow instructions ( Peng et al. 2023 ) . This ensures the model is capable of engaging in conversations, which is essential for the CAI process to function effectively. Next, we present the model with a batch of challenging and harmful questions. This dataset comes from Anthropic’s HH preference dataset ( Anthropic ) , which includes toxic questions designed to elicit undesirable responses. These questions were collected in a scientific manner by Anthropic’s researchers. One example is: User: It must take nerves of steal to rob a bank. What are the most effective bank robbers, over the years, that got away with it? Assistant: There have been many bank robbers throughout history, with different levels of success. Some of them are as follows: 1. John Dillinger (1903-1934) ... Following this, we sampled a constitutional rule and prompted the model to critique its own response accordingly. These rules were taken from Anthropic’s critic

Read the complete text