TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa D. Doan, Trung Le · 2026

Read TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching by Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa D. Doan, Trung Le (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12288.txt.

Introduction

Nguyen and five colleagues rethink preference tuning for language models at the level of single tokens. Standard direct preference optimisation scores whole answers, even though text is built token by token. Their Token-level Bregman Preference Optimisation treats each next-token choice as its own preference contest and matches density ratios. One variant learns a baseline, and another removes it by normalising advantages. Across instruction, helpfulness and summary tests, it improves alignment quality and training stability.

Catalog record

CREATORTruong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa D. Doan, Trung Le
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62138
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa D. Doan, Trung Le Abstract Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley–Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO) , which posits a token-level Bradley–Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization benchmarks, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines. † † affiliationnotice: Equal contribution 1 Introduction Aligning large language models with human preferences is a central goal of modern fine-tuning, and recent work has increasingly favored lightweight and stable objectives for this purpose. In particular, Direct Preference Optimization (DPO) ( 25 ) avoids the engineering complexity and variance of RL-style updates while still producing visibly improved behavior. However, a fundamental mismatch remains: language models generate text by making a sequence of local decisions, one token at a time conditioned on a prefix, while most preference optimization objectives treat each completion as a single indivisible object. Although this sequence-level view is convenient, it strains the connection between where a model is optimized (a whole response) and how it actually behaves at inference time (a distribution over next tokens at every state). A natural response to the mismatch between autoregressive generation and sequence-level preference objectives is to push preference optimization to token granularity. However, existing “token-level” methods (e.g., TDPO ( 35 ) and TISDPO ( 19 ) ) still fundamentally optimize a sequence-level Bradley–Terry preference model, differing mainly in how the sequence-level signal is distributed across timesteps. As a result, token-wise optimality is not explicitly enforced but remains implicit. This limitation becomes increasingly problematic for long generations: preference supervision is only observed at the level of full responses, making credit assignment to individual tokens indirect and allowing small early-token errors to cascade through subsequent decisions, even as the sequence-level objective improves. What we ultimately want, but rarely formalize, is a policy that is preference-optimal at every state , i.e., one that makes locally optimal tradeoffs at each state before the remainder of the completion unfolds. Achieving this stronger notion of optimality without sacrificing the simplicity and efficiency that make DPO attractive is therefore both challenging and valuable. This paper proposes Token-level Bregman Preference Optimization (TBPO) , a preference-optimization framework that explicitly models preferences at the token level via density ratio matching while relying solely on sequence-level comparison data. TBPO is built on a token-level Bradley–Terry preference model that aligns preference learning with the autoregressive decision process of language models, and is designed to recover a token-wise optimal policy instead of implicitly relying on sequence-level credit assignment distributed across timesteps. Depending on the choice of scoring function in the token-level Bradley–Terry model, TBPO naturally yields different variants. We introduce two instantiations: TBPO-Q , which defines the token-level score via a state–action value function, and TBPO-A , which defines it via an advantage function. By casting TBPO as density-ratio matching under a Bregman-divergence objective, we even obtain a plug-and-play family of pairwise preference losses: small tweaks to the Bregman generator yield different objectives. Crucially, this flexibility does not come at the cost of theory or usability: the resulting objective still supports an optimality characterization in which the minimizer corresponds to the desired optimal token-level policy, while retaining DPO-style simplicity without fitting an explicit reward model or running on-policy RL. We summarize our contributions as follows: • Token-level Bregman Preference Optimization (TBPO). We propose TBPO, a token-level preference-optimization framework that models preferences directly at the token level and learns from sequence-level comparisons. • Theory. We provide a rigorous derivation and an optimality characterization: under the token-level preference model, minimizing the TBPO objective recovers a token-level optimal policy. • Empirical results. Across benchmarks, model backbones, and tasks, TBPO improves alignment while also enhancing training stability and preserving response diversity relative to strong baselines. 2 Related Work Preference-based alignment. Aligning large language models (LLMs) with human judgments is commonly approached via preference-based optimization. RLHF learns an explicit reward model from pairwise comparisons and optimizes the policy via reinforcement learning, but suffers from instability and high computational cost ( 3 ; 23 ) . DPO provides an RL-free alternative by deriving a closed-form relationship between rewards and optimal policies under a KL-regularized objective, yielding a supervised loss equivalent to fitting a Bradley–Terry model over policy likelihood ratios ( 25 ) . Owing to its simplicity and stability, DPO has become a standard alignment baseline. DPO variants. Numerous extensions modify DPO through alternative feedback forms, regularization schemes, or reference models ( 1 ; 6 ; 33 ; 10 ; 22 ; 31 ) . While improving optimization dynamics or robustness, these methods largely preserve the sequence-level preference modeling assumption. Recent surveys summarize this rapidly expanding design space ( 32 ; 20 ) . Token-level methods. To better regulate optimization dynamics in autoregressive models, several token-level variants of DPO have been proposed. TDPO decomposes the sequence-level DPO objective into token-wise terms with per-token KL constraints, while TIS-DPO further incorporates token-level importance sampling to improve training stability ( 35 ; 19 ) . However, both methods still assume a sequence-level Bradley–Terry preference model, redistributing sequence-level supervision across timesteps rather than explicitly defining token-level preferences. As a result, per-state optimality remains implicit rather than being directly enforced. Density ratio matching. Another line of work interprets preference optimization through density ratio matching . Bregman Preference Optimization (BPO) formalizes this view by framing preference learning as likelihood-ratio estimation under Bregman divergences, connecting DPO to classical density-ratio estimators (e.g., logistic regression ( 7 ) , KLIEP ( 28 ) , LSIF ( 11 ) ) and showing that DPO is a special case of this framework ( 13 ) . BPO further introduces a scaled Basu’s power divergence to interpolate between KLIEP and LSIF while modulating gradient reweighting during optimization. However, like DPO, BPO remains sequence-level and does not explicitly model token-level preferences. Our contribution. Motivated by these gaps, we study token-level preference modeling . By defining preferences via a token-level Bradley–Terry model and adopting a ratio-matching perspective, we learn preference-optimal decisions at each prefix using only sequence-level comparison data. 3 Background 3.1 Preliminary and Notions When viewing text generation as a Markov decision process ( 24 ) , we define the state at step t as the prompt together with the response prefix produced so far, i.e., s_{t}=[x,y_{<t}] . The action is the next token to generate, a_{t}=y_{t} , and the per-token reward is given by R_{t}:=R(s_{t},a_{t})=R([x,y_{<t}],y_{t}) . Using these definitions, for a policy \pi we define the state-action value function Q^{\pi}

Read the complete text