Aligning to What? Limits to RLHF Based Alignment
Logan Barnhart, Reza Akbarian Bafghi, Stephen Becker, Maziar Raissi · 2025
Read Aligning to What? Limits to RLHF Based Alignment by Logan Barnhart, Reza Akbarian Bafghi, Stephen Becker, Maziar Raissi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09025.txt.
Introduction
Logan Barnhart, Reza Akbarian Bafghi, Stephen Becker, and Maziar Raissi tested the limits of feedback-based alignment for language models in two thousand twenty-five. Reinforcement learning from human feedback is widely used to align models with human values. Whether it removes hidden bias is unclear.
They applied three feedback methods to a mid-size open model and measured overt and covert bias against African Americans. Covert bias was probed with matched-guise tests, a method that swaps style while holding content. Feedback did not reliably fix covert bias, and training done before feedback seemed to lock bias in. They also extended bias tools to multimodal models. The authors call for better data and alignment tools.
Catalog record
| CREATOR | Logan Barnhart, Reza Akbarian Bafghi, Stephen Becker, Maziar Raissi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09025
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aligning to What? Limits to RLHF Based Alignment Logan Barnhart, Reza Akbarian Bafghi, Stephen Becker, Maziar Raissi Abstract Reinforcement Learning from Human Feedback (RLHF) is increasingly used to align large language models (LLMs) with human preferences. However, the effectiveness of RLHF in addressing underlying biases remains unclear. This study investigates the relationship between RLHF and both covert and overt biases in LLMs, particularly focusing on biases against African Americans. We applied various RLHF techniques (DPO, ORPO, and RLOO) to Llama 3 8B and evaluated the covert and overt biases of the resulting models using matched-guise probing and explicit bias testing. We performed additional tests with DPO on different base models and datasets; among several implications, we found that SFT before RLHF calcifies model biases. Additionally, we extend the tools for measuring biases to multi-modal models. Through our experiments we collect evidence that indicates that current alignment techniques are inadequate for nebulous tasks such as mitigating covert biases, highlighting the need for capable datasets, data curating techniques, or alignment tools. 1 1 1 We have made the source code available to the public at: https://github.com/loganbarnhart01/aligning-to-what Figure 1: Diagram illustrating how the text, traits, and prompt formats are utilized to calculate association scores. This is a sample from the matched-meaning setting where the AAE and SAE text are semantically equivalent. Note that each text sample would be formatted and passed through the model individually. 1 Introduction Increasingly, training state-of-the-art large language models (LLMs) includes reinforcement learning from human feedback (RLHF) to align language models to human preferences such as understanding user intent, harmlessness, helpfulness, etc. ( Bai et al., 2022 ; Dubey et al., 2024 ; OpenAI et al., 2024 ; Anthropic, 2024 ) . The process of collecting a meaningful amount of human feedback data requires the labor of many individuals ( Bai et al., 2022 ) who may not agree on the quality of responses with respect to something like harmlessness, raising the question: is RLHF optimizing for the objective we want? Previous work by Hofmann et al. (2024) inspected off-the-shelf language models to evaluate their overt and covert racial biases. Surprisingly, they found that off-the-shelf models trained with RLHF appeared to hold the strongest covert biases ( Hofmann et al., 2024 ) , but our review of the existing literature did not reveal any studies inspecting the relationship between RLHF and model biases. If covert biases represent — or at least act as a proxy for — a truer state of a model’s ‘moral values,’ then RLHF may not be adequately aligning LLMs to human preferences for more nuanced objectives such as harmlessness. Our goal is thus to analyze the relationship between post-training and model biases to conclude whether or not RLHF effectively aligns models to abstract goals such as harmlessness. We focus on examining the covert biases towards African Americans by examining a model’s attitude towards speakers of two different dialects: African American English (AAE) and Standard American English (SAE). Specifically, we train our own LLMs using alignment techniques to reduce harmful behavior, rather than depending on pre-trained models. After training, we use the methods from Hofmann et al. (2024) to detect and monitor both explicit and implicit biases that may still be present in our model. A subset of the post-training and bias evaluations are repeated on Mistral ( Jiang et al., 2023 ) to see both a different baseline for LLM biases and if RLHF influences different models uniquely. We also study the effects of extended post-training and the influence of different datasets on alignment. Although Llama 3 8B and Mistral are the only models to undergo post-training we conduct additional bias evaluations on Llama 3 Instruct, Llama 3.1 and its instruct tuned version, and Llama 3.2 and its instruct tuned version. All instruct versions have undergone extensive post-training ( Dubey et al., 2024 ) beyond that of our experiments, and comparing Llama 3 - 3.2 will allow us to see how the base LLM biases have changed as models become more capable. We also curate a new preference dataset containing only AAE text to see if the abundance of SAE text in pretraining is responsible for the biases, and whether or not further post-training on this dataset will meaningfully reduce bias. Finally, we extend current techniques limited to just language models to multimodal models to gather reduced-variance measurements of the models overt biases; this extension is performed on Llama 3.2 Vision 11B. Measuring overt biases in LLMs depends on explicit racial group names, whose limited availability leads to high variance measurements, but with VLMs explicit racial information is instead encoded into images of people. Our initial experiments seem to indicate that models are conditioned with covert biases after pre-training, and the overall nature of these biases are not influenced in any meaningful ways by RLHF regardless of post-training technique, dataset, or base model. Examining large-scale-post-trained models such as Llama 3-Instruct leads one to believe that with current techniques, to meaningfully alter a model’s biases you need to introduce new ones. We also find that supervised fine-tuning prior to RLHF appears to calcify the model biases and make them more resistant to change. Additionally, and tangential to future research, our preliminary experiments on measuring multi-modal covert biases seems to indicate a model’s overt and covert biases can be polar opposites of one another. In whole, these findings indicate that existing alignment techniques, such as RLHF, are inadequate for addressing complex tasks like reducing harmfulness or mitigating bias. This points to a potential limitation in current alignment strategies when dealing with subtler and more nebulous issues like model biases. Figure 2: Average favorability scores for the top 5 personality traits most associated with AAE/SAE (covert, left) and African-Americans/Caucasians (overt, right). Red dotted lines represent the average favorability scores for African Americans from the Princeton trilogy studies and Bergsieker et al ( Katz and Braly, 1933 ; Gilbert, 1951 ; Karlins et al., 1969 ; Bergsieker et al., 2012 ) . Note that all models have negative favorability for African-Americans in the overt setting. 2 Preliminaries 2.1 Biases in Language Models Measuring a group’s true beliefs towards another group has long been of interest in sociolinguistics; the matched-guise test was developed to measure participants’ differences in attitudes towards two groups ( Lambert et al., 1960 ) . In the test, a participant is provided with two audio recordings of an excerpt spoken in two different accents or “guises” and asked to answer questions about the speakers. The assumption of the test is that because the content of each message is the same, any difference in response toward different voices reflects the participant’s underlying attitudes toward the speaker’s group. Hofmann et al. (2024) extended this technique — labeled matched-guise probing — to probe an LLM’s attitudes towards the speakers of two different dialects (See Figure 1 ). Because we often have access to the probabilities of tokens assigned by a language model, we can take two text corpus’ of two dialects and analyze the probabilites of specific attributes conditioned on the text in each dataset. This allows one to measure which attributes are more associated with one dialect over another. The attributes we are interested in analyzing originate from