Convex Optimization for Alignment and Preference Learning on a Single GPU

Miria Feng, Mert Pilanci · 2026

Read Convex Optimization for Alignment and Preference Learning on a Single GPU by Miria Feng, Mert Pilanci (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23244.txt.

Introduction

Feng and Pilanci propose COALA, a light plan for aligning language models on one GPU. Fine-tuning drove systems like Gemini, but RLHF stays costly. Direct Preference Optimization is simpler but suffers inconsistent ranking accuracy. COALA uses convex tuning, drops the reference model, and cuts training time and VRAM use. Tests across four datasets and six models show good scores using about 17.6 percent of DPO total TFLOPs. COALA shows rising rewards and reaches peak margins faster than DPO. This is the first use of convex tuning.

Catalog record

CREATORMiria Feng, Mert Pilanci
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93055
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23244

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Convex Optimization for Alignment and Preference Learning on a Single GPU Miria Feng, Mert Pilanci Abstract Fine-tuning large language models (LLMs) to align with human preferences has driven the success of systems such as Gemini and ChatGPT. However, approaches like Reinforcement Learning from Human Feedback (RLHF) remain computationally expensive and complex. Direct Preference Optimization (DPO) offers a simpler alternative but has limitations such as inconsistent ranking accuracy, high dependence on GPU resources, and expensive hyperparameter tuning. We propose the C onvex O ptimization for A lignment and Preference L earning A lgorithm (COALA): a novel lightweight strategy with strong theoretical guarantees. By leveraging the convex optimization reformulation of neural networks, COALA eliminates the need for a reference model and obtains significant reduction in both training time and VRAM consumption, thus enabling efficient training on a single GPU. Experiments across four datasets—including a 26 621 26\,621 -sample synthetic Educational Feedback dataset—and six models (including Llama-3.1-8B) demonstrate COALA’s competitive performance and efficiency while utilizing as little as ∼ 17.6 % {\sim}17.6\% of DPO’s total TFLOPs. COALA exhibits stable, monotonically increasing rewards and reaches peak margins in significantly shorter time in comparison to traditional methods such as DPO and ORPO. To the best of our knowledge, this is the first time convex optimization has been effectively applied to preference fine-tuning of LLMs. 1 Introduction Large Language Models (LLMs) have been trained on increasing amounts of data to capture semantic language patterns and scale to solve more complex problems. The main paradigm combines pre-training and fine-tuning LLMs to achieve aligned user-preferable responses, and train personalized Language Model (LM) assistants for widespread applications ( 32 ; 57 ; 1 ) . Reinforcement Learning from Human Feedback (RLHF) ( 71 ; 60 ; 18 ; 74 ) demonstrates impressive results for preference alignment, and utilizes a three-step approach of Supervised Fine-Tuning (SFT), reward model training, and policy optimization. However, its complex multi-stage methodology presents several optimization challenges, requires humans in the loop, and is notably compute resource-intensive. This has driven timely and significant research on resource-efficient preference learning strategies, especially as energy demands outpace infrastructure ( 17 ) . The Direct Preference Optimization (DPO) ( 63 ) algorithm proposes a simpler and more computationally lightweight alternative to aligning LLMs for user-preferred responses. DPO parametrizes the reward function instead of learning an explicit reward model and incorporates this into the Bradley-Terry ranking objective ( 10 ) . Although simpler, this also yields certain drawbacks: requiring a reference model to stabilize training incurs additional memory costs to host two models in order to train a single model, the approximately 10 \times smaller learning rate (versus SFT) ( 39 ) slows optimization, while a mismatch between the reward optimized in training and the log-likelihood optimized during inference can lead to unstable reward margin gains ( 52 ) . To address these issues, recent studies have proposed eliminating the reference model to reduce memory overhead ( 42 ; 52 ) , advanced deeper theoretical understanding on the behavior of RLHF and DPO with in-depth analysis ( 3 ; 27 ; 72 ) , and reduced dependence on the prerequisite SFT stage, which is typically imperative to convergence ( 36 ; 78 ) . However, these techniques also introduce additional brittle hyperparameters, which the authors note are crucial to achieving good performance, or require small learning rates on the order of 1\times 10^{-9} for stable convergence. The large reliance on heuristic-driven methods may also result in random model ranking accuracy even after time-extensive training ( 16 ) . We present COALA, a fast and lightweight framework for effective preference alignment of LLMs on a single GPU. COALA leverages a convex two-layer neural network (cvxNN) on top of a pre-trained model for fast convergence with theoretical guarantees, and thus mitigates the instability of offline alignment via consistent reward margin increments. Unlike existing DPO style objectives, COALA utilizes an Alternating Direction Method of Multipliers (ADMM) ( 8 ) approach to enable scalable, near-hyperparameter-free optimization. We implement COALA in JAX ( 9 ) and efficiently execute Just-In-Time (JIT) compilation to maximize VRAM throughput. As a result, COALA fine-tunes models such as Llama-3.1-8B on a single RTX-4090 GPU ( 56 ) and demonstrates competitive performance on multiple benchmarks, including AlpacaEval2 ( 48 ) , ArenaHard ( 47 ) , and MT-Bench ( 80 ) . Comprehensive experiments span six models \times four datasets \times four methods. Results presented total over 216 training runs , and TFLOPs metrics further validate COALA’s substantial compute and power efficiency. Since auto-run benchmarks are ultimately surrogate for measuring real human preference, we conduct fair double-blind evaluation with 107 real human samples to assess the validity of benchmark scores against actual human feedback, and establish COALA’s practical effectiveness in a real world scenario. Our main contributions are summarized as follows: • We introduce COALA: a more interpretable and theory-backed convex framework that leverages principled optimization to enable effective, single-GPU preference alignment. • We prove COALA’s convergence guarantees. This theoretical analysis supports COALA’s exhibited stable convergence, while mitigating reliance on hyperparameter tuning and broad heuristics. • We present EduFeedback 1 1 1 https://huggingface.co/datasets/miria0/EduFeedback : the open-source dataset comprised of 26\,621 student-tutor conversations. We additionally introduce the “Alternating Population Strategy” , to uniquely derive 65\,606 preference training pairs, eliminating the need for external re-ranking models and establishing a more efficient paradigm for preference dataset synthesis. • Our modular open-source JAX codebase 2 2 2 https://github.com/pilancilab/COALA ensures ease of reproducibility for ongoing research, as well as practical on-premises single GPU use cases. 2 Related Work Preference fine-tuning LLMs can be classified as three distinct strategies. (1) Initial algorithms of zero-shot and few-shot in-context learning ( 12 ) rely on prompt engineering. This method does not require extensive compute, but is unable to tackle complex tasks due to reliability issues. (2) More sophisticated algorithms use reinforcement learning to align model output with user preferences. Successful algorithms in this class (such as RLHF and RLAIF ( 46 ) ) have been able to create conversational LLMs such as Google Gemini ( 32 ) and ChatGPT ( 57 ) . However, despite their impressive performance, the high computational costs of these methods pose a substantial barrier to entry for many applications. (3) DPO-style methods leverage the 10 model to provide simple yet often performant learning algorithms that do not require explicit reward modeling. Yet good results based on these methods consistently rely on extensive hyperparameter grid-search ( 52 ) , and are largely heuristics-driven. As LLMs become more widely adopted, there is also escalating awareness that scaling compute alone is increasingly unsustainable economically, environmentally, and in terms of diminishing returns ( 69 ) . This has motivated recent work to make progress through avenues such as repeated sampling ( 11 ) , creative chain-of-thought reasoning ( 77 ) , and theoretically interpretable techniques ( 70 ) . Single GPU methods are a timely and practical direction for preference fine-tuning, especially as local accelerators become increasingly powerful ( 2 ) . Targeting a single device improves robustness (fewer distributed failure modes), latency (no cross‑node synchronization), and data locality/security (privacy‑sensitive deployment with on-premises options). This trend is reflected in recent systems that deliver machine learning applications in text, vision,

Read the complete text