Explanation Quality Assessment as Ranking with Listwise Rewards Thomas Bailleux, Tanmoy Mukherjee, Emmanuel Lonca, Pierre Marquis, Zied Bouraoui Abstract We reformulate explanation quality assessment as a ranking problem rather than a generation problem. Instead of optimizing models to produce a single “best” explanation token-by-token, we train reward models to discriminate among multiple candidate explanations and learn their relative quality. Concretely, we construct per-instance candidate sets with graded quality levels and train listwise and pairwise ranking models (ListNet, LambdaRank, RankNet) to preserve ordinal structure and avoid score compression typical of pointwise regression or binary preference objectives. We observe three findings: First, ranking losses consistently outperform regression on score separation across all domains tested. Second, the optimal ranking loss depends on data characteristics: listwise objectives excel with well-separated quality tiers, while pairwise methods are more robust to noisy natural annotations. Third, when trained on carefully curated and well-structured data, small encoder models can match models that are orders of magnitude larger, suggesting that data quality matters more than model scale. Finally, when used as rewards in policy optimization, ranking-based scores enable stable convergence in settings where regression-based rewards fail entirely. Code and data are available at: https://github.com/Tankiit/PPO_Learning_to_rank 1 Introduction Although Large Language Models (LLMs) can produce many plausible explanations for a given query, a central challenge lies in determining which explanations are actually better than others. Explanation quality is inherently graded: two explanations may both appear plausible while differing substantially in logical consistency, factual correctness, and relevance to the underlying query. Yet most existing approaches treat explanation evaluation either as binary classification or as a pointwise generation problem, thereby collapsing fine-grained quality distinctions. In this work, we instead formulate explanation quality assessment as a learning-to-rank problem, in which models learn to order multiple candidate explanations by relative quality. A key difficulty is that standard reward objectives do not preserve these graded distinctions well enough for policy optimization. In particular, regression and binary preference objectives tend to compress explanation scores into a narrow range, making high- and low-quality explanations insufficiently distinguishable. This weakens the reward signal used by Proximal Policy Optimization (PPO): when score separation is too small, advantage estimates become dominated by noise and policy updates become ineffective. By contrast, ranking objectives preserve ordinal structure and maintain substantially larger score separation, yielding rewards that better support policy learning. Neural ranking models are well established in information retrieval 13 ; 19 . Our contribution is therefore not ranking per se, but identifying why ranking matters for explanation quality assessment and policy optimization. Specifically, we show that score compression is a key failure mode of regression and preference-based reward modeling; that the most effective ranking objective depends on the structure and noise profile of the data; and that ranking objectives require quality-diverse candidate sets in order to preserve useful score separation. We integrate ranking-based reward models into PPO so that generated explanations are evaluated using relative quality signals rather than compressed point estimates. We validate the approach on 50,000 human-annotated explanations from e-SNLI, whose gold-level annotations were collected with 87% inter-annotator agreement and Fleiss’ \kappa=0.72 as reported by the e-SNLI authors 3 . Our experiments span seven architectures, ranging from 110M to 7B parameters, and four NLI benchmarks. We focus on NLI because it provides rich explanation annotations and controlled validation settings, and we additionally test transfer to commonsense reasoning tasks to assess generalization beyond the primary domain. Figure 2 (Appendix) contrasts the generation-centric and ranking-based pipelines. (i) We identify score compression as a failure mode in explanation reward modeling. Regression and binary preference objectives compress graded quality scores into a narrow range, which weakens PPO learning signal. (ii) We establish an end-to-end link between ranking objectives and policy optimization. We show that better preservation of score separation leads to more informative advantage estimates, faster PPO convergence, and higher explanation quality. (iii) We show that the effectiveness of ranking depends on both the loss and the data. Listwise and pairwise objectives behave differently depending on the structure and noise level of the candidate sets, and ranking alone is insufficient without quality-diverse training data. (iv) We introduce a graded dataset construction paradigm for explanation ranking. We recast NLI benchmarks as ranking tasks by building five-level quality hierarchies with overlapping score ranges, anchored to e-SNLI’s human-authored gold explanations. 2 Related Work Our work connects to research on explanation evaluation, human judgment modeling, and ranking-based reinforcement learning from feedback. Explanation Evaluation. Recent work has advanced the generation and evaluation of explanations through structured and abductive reasoning. DecompRC 11 and Least-to-Most Prompting 18 improve interpretability by decomposing reasoning into sub-steps, while DRFACT 10 and GEAR 8 formalize open-ended and abductive reasoning using criteria such as consistency and diversity. We integrates quality supervision directly into training via ranking objectives, quantifying quality for continuous optimization. Human Judgment and Disagreement. ChaosNLI 12 showed inherent annotation uncertainty. Following findings that ambiguity improves calibration 17 ; 15 , we induce controlled variance through overlapping score ranges, emulating natural disagreement without costly annotation. Policy Optimization with Learned Rewards. Reward learning from preferences ( 5 ; 14 ) aligns models with human intent. We replace discrete ratings with continuous listwise supervision from graded rankings. Ranking and Preference Learning. We adapt ListNet, RankNet, LambdaRank, and ApproxNDCG that directly optimize for correct ordering rather than pointwise errors. Unlike text generation applications 19 , we target explanation evaluation where quality exists on a continuum requiring specialized ranking objectives. 3 Methodology We start from the insight that explanation quality is inherently a ranking problem rather than a regression or classification task. Instead, we rank existing explanations by quality. Motivation. Current explanation datasets (e.g., e-SNLI, ChaosNLI, or \delta -NLI) reveal quality hierarchies, yet models reduce these to binary or regressed values. Reward models trained with MSE compress rich scales into narrow ranges, producing negligible variance for PPO. This discards ranking-style supervision, limiting effective policy learning. From Generation to Ranking. Given an input (p,h,y) , where p denotes the premise, h the hypothesis, and y\in\{\textit{entail},\textit{contradict},\textit{neutral}\} , a model generates an explanation e=(t_{1},\ldots,t_{n}) token by token using \pi_{\theta}(\cdot\,|\,t_{y_{j}}-\log\sigma(f_{\theta}(q,e_{i})-f_{\theta}(q,e_{j})) (2) LambdaRank Gradient : \lambda_{ij}=\frac{|\Delta_{\text{NDCG}}|}{1+\exp(f_{\theta}(q,e_{i})-f_{\theta}(q,e_{j}))} (3) A.2 Model Architecture Details The ranking reward model consists of: 1. BERT-base encoder (110M parameters) 2. Two-layer projection head: 768\rightarrow 384\rightarrow 1 3. Dropout (0.1) between projection layers 4. No activation on final layer (raw scores for ranking) Training Hyperparameters Hyperparameter Value Learning rate 2e-5 Batch size 64 Warmup steps 500 Max epochs 30 Gradient clip 1.0 Weight decay 0.01 Table 8: Training configuration for ranking reward model Reproducibility All experiments use fixed random seeds (42, 123, 7 for multi-seed runs). Encoder models are trained with AdamW (lr = 2{\times}10^{-5} , batch = 16, warmup = 500 steps, weight decay = 0.01) for up to 50 epochs with early stopping (patience = 10) on a single NVIDIA V100 32 GB GPU. Decoder models use lr = 1{\times}10^{-5} , batch = 4, 4-bit NF4 quantisation 7 . PPO training uses GPT-2 medium (355M) with lr = 1.4{\times}10^{-5} , KL penalty \beta\,=\,0.1 , batch = 128, mini-batch = 4 for 3,000 steps. Code and graded datasets are available at https://github.com/Tankiit/PPO_Learning_to_rank , which also contains a step-by-step reproduction guide. Appendix B Graded Dataset Construction Problem: Zero Score Variance in NLI Datasets. Table 9 shows that original NLI datasets group by premise, assigning all hypotheses identical labels. This creates perfect NDCG@5 = 1.0, making them unsuitable for ranking evaluation. Dataset Grouping Variance NDCG@5 Delta-NLI By premise 0.0 1.0000 SNLI By premise 0.0 1.0000 MultiNLI By premise 0.0 1.0000 Table 9: Original NLI datasets have zero score variance per query. Solution: Quality-Graded Generation. For each quality level and NLI label, we design templates that systematically degrade explanation quality. Table 10 shows examples. Quality Entailment Template Contradiction Template Gold The premise “{premise}” directly supports the hypothesis “{hypothesis}”. The key evidence is that the premise provides sufficient information to conclude the hypothesis is true. The premise “{premise}” contradicts the hypothesis “{hypothesis}”. They present incompatible statements that cannot both be true simultaneously. Good The premise entails the hypothesis because they convey compatible information. These statements contradict because they make incompatible claims. Fair The premise supports the hypothesis. The statements conflict. Poor This is a contradiction because the premise and hypothesis are different. [Wrong!] This is neutral because they’re both statements. [Wrong!] Nonsense The quantum mechanics of penguin migration patterns suggest umbrella distribution. Purple elephants dance backwards when triangles sing opera. Table 10: Explanation templates for each quality level. Poor templates intentionally use wrong labels. The critical innovation is overlapping score ranges that create ambiguity: \text{score}(e,q_{\text{level}})\sim\mathcal{U}(\text{min}_{q},\text{max}_{q})+\text{Adjustments}(e) (4) where base ranges are: \displaystyle q_{\text{gold}} \displaystyle\in[0.70,1.00] (5) \displaystyle q_{\text{good}} \displaystyle\in[0.50,0.85]\quad\text{(overlaps gold \& fair)} (6) \displaystyle q_{\text{fair}} \displaystyle\in[0.30,0.70]\quad\text{(overlaps good \& poor)} (7) \displaystyle q_{\text{poor}} \displaystyle\in[0.10,0.50]\quad\text{(overlaps fair \& nonsense)} (8) \displaystyle q_{\text{nonsense}} \displaystyle\in[0.00,0.30]\quad\text{(overlaps poor)} (9) Content-Aware Scoring Heuristics. Scores are adjusted deterministically based on explanation content: Algorithm 1 Heuristic Scoring Function 1: Input: explanation e , premise p , hypothesis h , label y , quality q 2: s\sim\mathcal{U}(q_{min},q_{max}) \triangleright Base score 3: if y is mentioned in e then 4: s\leftarrow s+0.05 \triangleright Label matching 5: end if 6: if reasoning keywords in e then 7: s\leftarrow s+0.03 \triangleright “because”, “therefore”, etc. 8: end if 9: overlap\leftarrow\frac{|words(e)\cap(words(p)\cup words(h))|}{|words(p)\cup words(h)|} 10: if overlap>0.3 then 11: s\leftarrow s+0.02 \triangleright Content relevance 12: end if 13: if q\in\{gold,good\} and |words(e)|<15 then 14: s\leftarrow s-0.10 \triangleright Too short penalty 15: end if 16: if q\neq nonsense and gibberish words in e then 17: s\leftarrow s-0.30 \triangleright Gibberish penalty 18: end if 19: return \max(0,\min(1,s)) \triangleright Manual clip for clarity All scoring uses deterministic random seeds (hash of content) for reproducibility. Table 11 shows statistics for graded datasets. Table 12 compares Non-Overlapping vs Overlapping. Our approach applies to any NLI dataset with (p,h,y) structure: e-SNLI: 50,000 examples graded; Delta-NLI: 700 examples graded; SNLI: 698 examples graded; and MultiNLI: 700 examples graded. Dataset Size Ambiguity Score Range NDCG@5 Graded Delta-NLI 700 78% 0.716 0.9928 Graded SNLI 698 54% 0.709 0.9861 Graded MultiNLI 700 56% 0.714 0.9887 Table 11: Graded dataset statistics. Ambiguity = % examples with quality-order violations (e.g., poor > good). Scoring Method NDCG@3 NDCG@5 Spearman Status Fixed (non-overlap) 1.0000 1.0000 1.0000 Too easy Overlapping (ours) 0.9861 0.9928 0.8740 Realistic Table 12: Ablation: non-overlapping ranges yield perfect metrics (unusable), while overlapping ranges create realistic ranking difficulty. Example: Graded Delta-NLI Instance Premise: A man is playing guitar on stage. Hypothesis: A musician is performing. Label: Entailment Candidates: 1. Gold (score=0.92): The premise “A man is playing guitar on stage” directly supports the hypothesis “A musician is performing”. The key evidence is that playing guitar on stage is a form of musical performance. 2. Good (score=0.71): The premise entails the hypothesis because playing guitar on stage is performing music. 3. Fair (score=0.58): The premise supports the hypothesis. 4. Poor (score=0.32): This is a contradiction because the premise mentions a man while the hypothesis says musician. [Wrong label!] 5. Nonsense (score=0.14): The quantum mechanics of penguin migration patterns suggest umbrella distribution. Note: Poor explanation (0.32) could score higher than Fair (0.58) in other examples due to overlapping ranges, creating ranking ambiguity. Appendix C Complete Experimental Results C.1 Loss Function Comparison Loss NDCG@1 NDCG@3 NDCG@5 MAP MRR Spearman Sep. \rho Ratio MSE 0.682 0.748 0.789 0.712 0.745 0.512 0.089 RankNet 0.745 0.821 0.856 0.789 0.812 0.682 0.341 ApproxNDCG 0.768 0.839 0.874 0.801 0.834 0.721 0.523 ListNet 0.812 0.881 0.912 0.827 0.865 0.798 0.847 Table 13: Final validation performance (epoch 50 or best) across loss functions with DeBERTa-v3-base. ListNet consistently outperforms alternatives across all metrics. Model Params NDCG@1 NDCG@3 NDCG@5 Spearman Speed Sep. \rho (it/s) Ratio BERT-base 110M 0.768 0.841 0.873 0.724 18.2 0.712 RoBERTa-base 125M 0.789 0.859 0.891 0.751 16.5 0.765 DeBERTa-v3 184M 0.812 0.881 0.912 0.798 15.3 0.847 Table 14: Encoder model comparison with ListNet loss. DeBERTa-v3-base achieves best performance with acceptable speed (15.3 iterations/second on V100 GPU). Epoch Train Loss NDCG@1 NDCG@3 NDCG@5 \rho Sep. Ratio 1 0.428 0.634 0.701 0.712 0.542 0.312 5 0.312 0.698 0.759 0.782 0.621 0.487 10 0.245 0.745 0.812 0.834 0.689 0.623 15 0.198 0.778 0.841 0.869 0.734 0.712 20 0.167 0.795 0.862 0.891 0.765 0.778 25 0.149 0.803 0.872 0.903 0.781 0.812 30 0.138 0.809 0.878 0.910 0.792 0.834 32 0.135 0.812 0.881 0.912 0.798 0.847 35 0.134 0.812 0.880 0.912 0.797 0.845 40 0.133 0.811 0.880 0.911 0.796 0.843 45 0.133 0.812 0.881 0.912 0.798 0.846 50 0.132 0.811 0.880 0.911 0.797 0.844 Table 15: Training progression for DeBERTa-v3 + ListNet. Best validation performance (highlighted) achieved at epoch 32. Performance plateaus after epoch 30 with minor fluctuations. Epoch Train Loss NDCG@1 NDCG@3 NDCG@5 Spearman Sep. Ratio 1 0.156 0.512 0.589 0.612 0.298 0.045 5 0.089 0.578 0.641 0.673 0.367 0.056 10 0.067 0.612 0.678 0.712 0.412 0.062 15 0.054 0.634 0.701 0.738 0.445 0.068 20 0.046 0.651 0.718 0.756 0.468 0.073 25 0.041 0.663 0.729 0.768 0.485 0.078 30 0.038 0.672 0.738 0.778 0.497 0.082 35 0.036 0.677 0.743 0.784 0.504 0.085 40 0.035 0.680 0.746 0.787 0.509 0.087 45 0.034 0.681 0.747 0.788 0.511 0.088 48 0.034 0.682 0.748 0.789 0.512 0.089 50 0.034 0.682 0.748 0.789 0.512 0.089 Table 16: MSE baseline training progression. Shows slower convergence (48 epochs to best) and poor score separation throughout training. Sep. Ratio never exceeds 0.089, insufficient for effective PPO. Epoch Train Loss NDCG@1 NDCG@3 NDCG@5 Spearman Sep. Ratio 1 0.389 0.589 0.656 0.678 0.423 0.156 5 0.267 0.651 0.723 0.752 0.534 0.223 10 0.201 0.698 0.768 0.801 0.612 0.267 15 0.165 0.723 0.795 0.828 0.648 0.298 20 0.142 0.735 0.809 0.843 0.667 0.318 25 0.128 0.741 0.816 0.851 0.676 0.329 30 0.119 0.744 0.820 0.855 0.681 0.337 35 0.113 0.745 0.821 0.856 0.682 0.340 40 0.111 0.745 0.821 0.856 0.682 0.341 42 0.110 0.745 0.821 0.856 0.682 0.341 45 0.110 0.745 0.821 0.856 0.682 0.340 50 0.110 0.745 0.821 0.856 0.682 0.341 Table 17: RankNet training progression. Converges at epoch 42 with moderate performance. Sep. Ratio = 0.341 is better than MSE but insufficient for strong PPO signals. Epoch Train Loss NDCG@1 NDCG@3 NDCG@5 Spearman Sep. Ratio 1 0.412 0.612 0.681 0.701 0.478 0.234 5 0.289 0.678 0.745 0.771 0.578 0.334 10 0.218 0.723 0.789 0.818 0.645 0.412 15 0.178 0.748 0.814 0.845 0.684 0.467 20 0.153 0.759 0.827 0.860 0.704 0.498 25 0.138 0.765 0.834 0.868 0.714 0.512 30 0.128 0.768 0.838 0.873 0.719 0.519 35 0.122 0.768 0.839 0.874 0.721 0.522 38 0.120 0.768 0.839 0.874 0.721 0.523 40 0.119 0.768 0.839 0.874 0.721 0.523 45 0.119 0.768 0.839 0.874 0.721 0.522 50 0.119 0.768 0.839 0.874 0.721 0.523 Table 18: ApproxNDCG training progression. Converges at epoch 38 with strong performance, second only to ListNet. Sep. Ratio = 0.523 is better than RankNet but still below ListNet’s 0.847. Loss Epoch to Epoch to Epoch to Final Training NDCG 0.8 NDCG 0.85 Best NDCG@5 Time (hrs) MSE Never Never 48 0.789 6.8 RankNet 18 28 42 0.856 6.1 ApproxNDCG 12 22 38 0.874 5.5 ListNet 10 16 32 0.912 4.7 Table 19: Convergence speed comparison. ListNet reaches NDCG@5 = 0.85 in 16 epochs vs 28 for RankNet (43% faster). Total training time reduced by 31% compared to MSE baseline. Table 13 shows final validation performance across loss functions with DeBERTa-v3-base, confirming ListNet’s consistent advantage across all metrics. Table 14 reports the architecture comparison; DeBERTa-v3-base achieves the best overall performance while maintaining reasonable inference speed. Table 15 shows ListNet training dynamics: best performance is reached at epoch 32, after which metrics plateau with only minor fluctuations. Table 16 shows the MSE baseline converges more slowly, peaking at epoch 48 with poor score separation throughout —its separation ratio remains too low for effective PPO regardless of training duration. Table 17 reports RankNet dynamics; the model converges between MSE and ListNet with moderate ranking quality and improved but not optimal score separation. Table 18 reports ApproxNDCG dynamics; ranking quality and separation both place it second among all objectives, outperforming pairwise comparisons but falling short of the listwise approach. Finally, Table 19 compares convergence speed across objectives: ListNet reaches NDCG@5 = 0.85 in 16 epochs, compared with 28 epochs for RankNet, making it 43% faster, and reduces total training time by 31% relative to the MSE baseline. C.2 Per-Dataset Detailed Results Table 20 reports the results on E-SNLI (50k training examples), while Table 21 presents the corresponding results on Delta-NLI (68k training examples). Table 22 reports WinoWhy (14k training examples) results. Table 23 shows results on MultiNLI (700 validation examples). Table 24 defines the score separation ratio, which measures the quality of reward signals for PPO training: \text{Sep. Ratio}=\frac{\sigma(\text{predicted scores})}{\sigma(\text{true scores})} (10) Score separation should be viewed as a spectrum: any improvement over the MSE baseline (0.089) strengthens the PPO signal relative to regression. However, achieving high absolute separation (above 0.8) also requires graded training data that spans a broad range of explanation qualities; ranking objectives alone are not sufficient without quality-diverse candidates. All experiments were conducted on a single NVIDIA V100 GPU (32GB). Table 25 summarizes the computational requirements of encoder models trained with ListNet loss. Loss NDCG@5 MAP MRR Kendall Spearman \tau \rho MSE 0.801 0.723 0.756 0.412 0.523 RankNet 0.868 0.798 0.834 0.556 0.689 ApproxNDCG 0.887 0.821 0.856 0.589 0.728 ListNet 0.924 0.856 0.891 0.634 0.812 Table 20: E-SNLI validation set performance (10k queries). ListNet shows the strongest performance on this large, diverse dataset. Loss NDCG@5 MAP MRR Kendall Spearman \tau \rho MSE 0.795 0.714 0.748 0.398 0.512 RankNet 0.862 0.786 0.823 0.543 0.674 ApproxNDCG 0.881 0.809 0.845 0.576 0.715 ListNet 0.918 0.841 0.879 0.621 0.795 Table 21: Delta-NLI validation set performance (1,785 queries). Consistent performance across all metrics. Loss NDCG@5 MAP MRR Kendall Spearman \tau \rho MSE 0.768 0.689 0.721 0.367 0.478 RankNet 0.834 0.754 0.789 0.512 0.634 ApproxNDCG 0.851 0.776 0.812 0.541 0.672 ListNet 0.897 0.812 0.854 0.589 0.768 Table 22: WinoWhy validation set performance (573 queries). Smaller dataset shows slightly lower absolute performance but similar relative improvements. Loss NDCG@5 MAP MRR Kendall Spearman \tau \rho MSE 0.756 0.673 0.708 0.345 0.456 RankNet 0.823 0.738 0.774 0.489 0.612 ApproxNDCG 0.841 0.761 0.798 0.521 0.651 ListNet 0.889 0.798 0.835 0.567 0.741 Table 23: MultiNLI validation set performance (700 queries). Smallest dataset shows most variability but consistent ranking of loss functions. Loss Score Std. Score Range Sep. Ratio PPO Viable? MSE 0.043 0.094 0.089 \times No RankNet 0.164 0.428 0.341 ⚫ Weak ApproxNDCG 0.251 0.612 0.523 ⚫ Moderate ListNet 0.407 0.889 0.847 \checkmark Yes Table 24: Score separation analysis across loss functions. ListNet maintains 0.407 std. dev. in predicted scores compared to 0.481 in true scores (Sep. Ratio = 0.847). MSE compresses to 0.043 std. dev. (Sep. Ratio = 0.089), destroying PPO learning signal. Model Batch Speed Memory Time/Epoch Total Time (it/s) (GB) (min) (50 epochs) BERT-base 32 18.2 8.4 3.2 2.7 hrs RoBERTa-base 32 16.5 9.1 3.5 2.9 hrs DeBERTa-v3 32 15.3 10.7 3.8 3.2 hrs Table 25: Computational requirements for encoder models with ListNet loss. DeBERTa-v3 requires 18% more time than BERT-base but delivers 4.5% better NDCG@5 (0.912 vs 0.873). Appendix D Human Validation: Scoring and Agreement Protocol. This section details how the graded gold scores are produced, how each reward model’s predicted ordering is compared against them, and where the remaining model errors come from. How the scores are assigned. For each NLI instance (p,h,y) we construct five candidate explanations, one per quality tier (gold, good, fair, poor, nonsense). Gold-tier candidates are drawn directly from the human-authored e-SNLI explanations; the remaining four tiers are produced by the deterministic label-aware templates in Table 10 . Every candidate then receives a score in [0,1] via the content-aware heuristic in Algorithm 1 : a base score is sampled from the tier’s range (Eq. ( 5 )–( 9 )) and adjusted by label matching ( +0.05 ), reasoning keywords ( +0.03 ), premise–hypothesis content overlap ( +0.02 ), and penalties for too-short gold/good candidates ( -0.10 ) or gibberish in non-nonsense tiers ( -0.30 ). All random draws use a deterministic seed derived from a hash of the content, so scores are reproducible. The full list of training hyperparameters kept fixed across all experiments is given in Table A.2 (App. A.2 ). Worked example. Applying the procedure to the instance in Table 2 (premise: “A man is playing guitar on stage” ; hypothesis: “A musician is performing” ) yields: • Gold: “Playing guitar on stage is a form of musical performance.” — e-SNLI annotation, base score drawn from the gold range, label match and keyword adjustments applied. • Good: label-aware entailment template, score from the good range with content-overlap adjustment. • Fair: minimal template ( “The premise supports the hypothesis” ), short-penalty may apply. • Poor: wrong-label template ( “Contradiction: man \neq musician” ), drawn from the poor range. • Nonsense: irrelevant filler, drawn from the nonsense range with no content adjustment. These scores induce a strict total ordering \text{gold}>\text{good}>\text{fair}>\text{poor}>\text{nonsense} , which defines \binom{5}{2}=10 ordered candidate pairs per instance. How agreement is measured. Each reward model produces a predicted score for every candidate, which induces a predicted ordering on the same ten pairs. Pairwise agreement for a given query is the fraction of pairs on which the predicted and gold orderings agree; the reported per-model agreement rates average this quantity over 200 held-out e-SNLI queries — 200 queries \times 10 pairs = 2,000 pairwise tests per model. This procedure yields ListNet 87%, RankNet 76%, MSE 62% . The differences are also visible at the distributional level: Figure 1 shows predicted versus true quality score distributions across 10k validation examples, where ListNet preserves the full distributional shape while MSE collapses scores into a narrow range. Where the remaining errors come from. A manual analysis of 100 misranked examples attributes the remaining disagreements to four sources: overlapping quality ranges (45%, expected ambiguity introduced by our graded data construction), subtle reasoning differences (32%), domain-specific knowledge gaps (15%), and annotation errors (8%). The dominant category is ambiguity introduced by our own graded data design rather than model failure, which explains why even ListNet’s 87% agreement is close to, rather than at, the ceiling of the gold annotations themselves. Reliability of the anchor. The gold tier inherits its reliability from the e-SNLI annotation protocol, for which the e-SNLI authors report 87% inter-annotator agreement and Fleiss’ \kappa=0.72 . Model–gold agreement is therefore measured against an underlying human-reliability level, rather than against a newly-collected annotator study. Appendix E Future work Future work should explore integrating ranking objectives more directly into policy optimization, investigating scalable hybrid losses, and validating synthetic quality scores against human judgments across domains. Extending the ranking paradigm to multi-aspect quality evaluation and step-wise reasoning signals could enable richer, more interpretable feedback mechanisms. Finally, improving the efficiency of ranking models through distillation, quantization, or architectural advances will be essential for real-world deployment. Overall, our results highlight that explanation quality is inherently graded rather than binary. Ranking-based objectives offer a principled way to model this continuum, bridging information retrieval and NLP, and laying the groundwork for more robust, nuanced, and effective explanation evaluation and training.