RANKING-AWARE REINFORCEMENT LEARNING FOR ORDINAL RANKING

Aiming Hao, Chen Zhu, Jiashu Zhu, Jiahong Wu \sthanks Corresponding Author, Xiangxiang Chu · 2026

Read RANKING-AWARE REINFORCEMENT LEARNING FOR ORDINAL RANKING by Aiming Hao, Chen Zhu, Jiashu Zhu, Jiahong Wu \sthanks Corresponding Author, Xiangxiang Chu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20585.txt.

Introduction

Aiming Hao and four colleagues present ranking-aware reinforcement learning for ordinal ranking, where labels come in an ordered scale such as star ratings. Ordinal tasks are hard because the labels depend on each other in order, which conventional methods model poorly. Their framework, RARL, learns those relationships explicitly.

At its core is a unified objective that joins regression and learning-to-rank so each improves the other. Regression pins down the level; ranking enforces the order. Ordinal outputs drive recommendations, grading, and medical staging, so respecting order rather than treating labels as plain classes should raise accuracy across those uses.

Catalog record

CREATORAiming Hao, Chen Zhu, Jiashu Zhu, Jiahong Wu \sthanks Corresponding Author, Xiangxiang Chu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20585

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RANKING-AWARE REINFORCEMENT LEARNING FOR ORDINAL RANKING Aiming Hao, Chen Zhu, Jiashu Zhu, Jiahong Wu \sthanks Corresponding Author, Xiangxiang Chu Abstract Ordinal regression and ranking are challenging due to inherent ordinal dependencies that conventional methods struggle to model. We propose Ranking-Aware Reinforcement Learning (RARL), a novel RL framework that explicitly learns these relationships. At its core, RARL features a unified objective that synergistically integrates regression and Learning-to-Rank (L2R), enabling mutual improvement between the two tasks. This is driven by a ranking-aware verifiable reward that jointly assesses regression precision and ranking accuracy, facilitating direct model updates via policy optimization. To further enhance training, we introduce Response Mutation Operations (RMO), which inject controlled noise to improve exploration and prevent stagnation at saddle points. The effectiveness of RARL is validated through extensive experiments on three distinct benchmarks. † † address: AMAP, Alibaba Group 1 INTRODUCTION Ordinal regression and ranking are fundamental tasks in signal processing and machine learning, where the objective is to predict and sort data based on labels with inherent order. These tasks appear in diverse applications, from medical signal analysis (e.g., severity staging) [ 3 , 18 ] to content quality assessment (e.g., audio/image aesthetics) [ 17 , 10 ] . Unlike standard regression or classification, ordinal tasks demand that a model captures both the continuous nature of values and the hierarchical dependencies among categories. This dual requirement poses a significant challenge: models must balance regression accuracy (predicting exact values) and ranking consistency (preserving relative order). Conventional methods often fail to address this trade-off effectively. Regression models [ 14 , 13 , 5 ] minimize point-wise errors but neglect ordinal dependencies, while classification-based approaches [ 7 , 6 ] lose granularity by discretizing labels. Even Learning-to-Rank (L2R) methods [ 12 , 2 ] , which focus on relative order, struggle to jointly optimize both objectives within a single framework. Crucially, these paradigms lack an explicit mechanism to model the causal relationships between input signals and their ordinal structure. Recent advances have shown Reinforcement Learning (RL) [ 11 , 4 ] to be a powerful tool for optimizing complex, multi-objective rewards, particularly in the context of Large Language Models (LLMs) [ 9 ] . Inspired by this, we propose Ranking-Aware Reinforcement Learning (RARL), a novel RL framework that directly models ordinal dependencies. RARL unifies regression and L2R through a ranking-aware verifiable reward, which allows a policy to be optimized for both regression accuracy and ranking consistency simultaneously. Furthermore, to combat the entropy collapse phenomenon [ 19 ] common in RL, we introduce the Response Mutation Operation (RMO), a mechanism to enhance exploration and escape saddle points. Our contributions are: •A unified RL framework RARL is introduced to jointly optimize regression and ranking via ranking-aware verifiable rewards, ensuring bidirectional alignment between objectives. •RMO is developed to alleviate the entropy collapse phenomenon during GRPO training and reactivate gradient signals to escape the saddle points, thereby motivating further exploration of evolutionary strategies in RL frameworks. Finally, we demonstrate through extensive experiments that RARL achieves significantly better performance on three distinct benchmarks: facial age estimation [ 20 ] , object count ranking [ 15 ] , and aesthetic assessment [ 8 ] , proving its effectiveness and generalizability. 2 BACKGROUND Reinforcement Learning Fine-tuning . Our work is grounded in the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm [ 9 , 16 ] , which refines a policy model \pi_{\theta} by optimizing for a task-specific, verifiable reward function R:\left(\boldsymbol{q},\boldsymbol{o}\right)\mapsto\mathbb{R} instead of a learned reward model. This approach eliminates reward model bias and is ideal for tasks with clear correctness criteria. The objective maximizes the expected reward, regularized by a KL-divergence term to prevent large deviations from a reference policy \pi_{\text{ref}} : \displaystyle\max_{\theta}\ \mathbb{E}_{\boldsymbol{q},\boldsymbol{o}\sim\pi_{\theta}}\left[R\left(\boldsymbol{q},\boldsymbol{o}\right)\right]-\beta\mathbb{E}_{\boldsymbol{q}}\left[\text{KL}\left(\pi_{\theta}\left(\boldsymbol{o}\mid\boldsymbol{q}\right)\parallel\pi_{\text{ref}}\left(\boldsymbol{o}\mid\boldsymbol{q}\right)\right)\right], (1) We specifically employ Group Relative Policy Optimization (GRPO), an advanced RLVR algorithm. For each input \boldsymbol{q} , GRPO samples a group of responses \left\{\boldsymbol{o}_{i}\right\}_{i=1}^{G} and calculates a normalized, intra-group advantage \hat{A}_{i} for each response: \displaystyle\hat{A}_{i}=\tilde{r}_{i}=\frac{r_{i}-\text{mean}\left(\left\{r_{i}\right\}_{i=1}^{G}\right)}{\text{std}\left(\left\{r_{i}\right\}_{i=1}^{G}\right)}. (2) This relative advantage encourages the model to prioritize responses that outperform others within the same generation group. Our proposed framework leverages this mechanism with custom-designed rewards for ordinal ranking. Table 1 : Prompts used in RARL. We have listed the system and instruct prompt separately. System Prompt: The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within < think > < /think > and < answer > < /answer > tags, respectively, i.e., < think > reasoning process here < /think > < answer > answer here < /answer > . Instruct Prompt: Analyze each image to determine the [Rule], and if there are multiple images, sort them in ascending order of [Rule]. Output the results as a JSON list of dictionaries in this format: [{’image_id’: number, ’value’: value}, …], where value is the [Rule] ([Value_Range]). Ensure the list is sorted correctly with strictly followed syntax. Problem Formulation . Ordinal regression is a structured prediction task where labels possess inherent ordinal relationships, such as facial age levels or aesthetic scores. Extending this concept, the ordinal ranking task dynamically sequences images by fusing label hierarchies (e.g., age progression) with textual semantics (e.g., rank from youngest to oldest). These two tasks can be unified under an input-dependent framework: single-image inputs trigger pure ordinal regression, while multi-image batches require simultaneous optimization of regression accuracy and ranking consistency through multi-task learning. Mathematically, given an input text prompt \boldsymbol{q} and an unordered list of N images \boldsymbol{X}=\left[\boldsymbol{x}_{1},\cdots\boldsymbol{x}_{N}\right] , each image \boldsymbol{x}_{i} is assigned an ordinal rank y_{i}\in\left\{r_{1},...,r_{M}\right\} under \boldsymbol{q} , where r_{M}\succ r_{M-1}\succ\cdots\succ r_{1} defines a total order over M distinct ranks. Here, \succ denotes strict precedence (e.g., higher age level or quality). The model \mathcal{M} predicts the ordinal ranks \tilde{\boldsymbol{Y}}=\left\{\tilde{\boldsymbol{y}}_{i}\right\}_{i=1}^{N} and a permutation \pi that sorts \boldsymbol{X} into an ordered sequence \tilde{\boldsymbol{X}}=\left[\boldsymbol{x}_{\pi\left(1\right)},\cdots\boldsymbol{x}_{\pi\left(N\right)}\right] satisfying \boldsymbol{y}_{\pi\left(1\right)}\succ\cdots\succ\boldsymbol{y}_{\pi\left(N\right)} . To model the above tasks, we implement \mathcal{M} as a VLM that generates JSON-formatted outputs (prompt templates in Table 1 ) with two structured fields: • value : Predicted ordinal label; • image_id : Image IDs in ranked order. Then, the RARL framework is proposed to unify ordinal image regression and ranking under a reinforcement learning paradigm, where verifiable reward functions assess both the accuracy of the regression values and the correctness of the ranking permutation, as detailed in the next section. 3 METHODOLOGY 3.1 Ranking-Aware Verifiable Rewards Figure 1 : Flowchart of proposed RARL. For a given image-question pair, our RARL model generates multiple reasoning-based responses. It is then optimized via policy gradient using ranking-aware verifiable rewards. Then, the response mutation operation is employed to reactivate gradient signals and escape saddle points. The structure of the proposed RARL is presented in Fig. 1 . In RARL, instead of a learned reward model, we employ a verifiable reward function to quantify prediction correctness. We design this function with two primary components: a Regression Reward and a Ranking Reward. Regression Reward in Ordinal Regression Tasks . To measure the accuracy of the predicted continuous value \tilde{\boldsymbol{y}}_{i} , we use an L2-inspired reward that is non-zero only if the prediction is within a tolerance \delta of the ground truth \boldsymbol{y}_{i} .

Read the complete text