Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers
Saron Samuel, Benjamin Van Durme, Eugene Yang · 2026
Read Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers by Saron Samuel, Benjamin Van Durme, Eugene Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10332.txt.
Introduction
Saron Samuel, Benjamin Van Durme, and Eugene Yang ask whether reasoning rerankers treat groups more fairly than plain rerankers in search. Using the TREC 2022 Fair Ranking Track data, they test six models across settings and demographic attributes. Reasoning neither helped nor hurt fairness: their attention-weighted fairness metric stayed flat while relevance varied a lot. Gaps tied to geography showed up for every model, whatever its design. The authors suggest future reasoning models could do better if built to notice fairness attributes directly.
Catalog record
| CREATOR | Saron Samuel, Benjamin Van Durme, Eugene Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33158 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10332
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers Saron Samuel, Benjamin Van Durme, Eugene Yang Abstract While reasoning rerankers, such as Rank1, have demonstrated strong abilities in improving ranking relevance, it is unclear how they perform on other retrieval qualities such as fairness. We conduct the first systematic comparison of fairness between reasoning and non-reasoning rerankers. Using the TREC 2022 Fair Ranking Track dataset, we evaluate six reranking models across multiple retrieval settings and demographic attributes. Our findings demonstrate reasoning neither improve nor harm fairness compared to non-reasoning approaches. Our fairness metric, Attention-Weighted Rank Fairness (AWRF) remained stable (0.33-0.35) across all models, even as relevance varies substantially (nDCG 0.247-1.000). Demographic breakdown analysis revealed fairness gaps for geographic attributes regardless of model architecture. These results indicate that future work in specializing reasoning models to be aware of fairness attributes could lead to improvements, as current implementations preserve the fairness characteristics of their input ranking. 1 Introduction Search systems play a critical role in shaping how we access information since their rankings determine which perspectives gain visibility. While surfacing relevant information usually yields better utility in tasks [ 7 ] , ensuring a comprehensive coverage of different opinions and sources is critical when information systems are used for decision making [ 20 ] .Recently, as reasoning large language models (LLMs) have shown strong effectiveness in various tasks through test-time reasoning [ 2 ] , rerankers based on these reasoning models can reason about a ranking, generating thoughts and justifications before producing a final ranking. These reasoning rerankers, such as Rank1 [ 28 ] , Qwen3-Reranker [ 31 ] , and ReasonRank [ 13 ] , have demonstrated strong improvements in relevance metrics across several benchmarks, such as BEIR [ 26 ] , NeuCLIR [ 10 ] and BRIGHT [ 25 ] . While reasoning rerankers may produce more relevant rankings, how they impact fairness in a ranking remains underexplored. Fairness in ranking can be operationalized as an equitable distribution of exposure across groups defined by sensitive attributes (such as gender, age, occupation, geography) while still maintaining retrieval effectiveness [ 5 ] . For example, a search system might overexpose content about men when women are equally relevant. On one hand, reasoning could help by encouraging the model to deliberate more deeply about context. On the other hand, the reasoning process itself is shaped by pretraining data and internal biases [ 11 ] . If those biases are reflected in the generated justifications, reasoning may instead amplify unfairness by producing confidently skewed rationales. Consider a search query “basic overview of sailing and types of sailboats.” Searching over the Fair Ranking 2022 track document collection, Table 1 shows, all three rankings consist of relevant documents. BM25 [ 21 ] retrieval surfaces mainly USA-based articles (7 of 10), while an oracle ranking constructed from relevance annotations achieves better geographic diversity with articles from Sweden, Finland, Israel, UK, Poland, and New Zealand. Rank1 [ 28 ] , a pointwise reasoning reranker, reranks on top of BM25, also outputs a diverse ranked list with different document origins. Table 1: Qualitative comparison of subject geography between a BM25 initial retrieval, then reranked by Rank1, and the ground truth Oracle rankings for the query “basic overview of sailing and types of sailboats” using the Fair Ranking 2022 document collection. These were marked by hand annotation (country level), and closely followed the original annotations (continent level). Sections with – mean that the subject geography could not be identified. BM25 Rank1 Oracle Ranking Title Origin Title Origin Title Origin Outline of sailing – Boating – Seafarer 37 USA Ultimate 20 USA Outline of sailing – Stefan Krook Sweden Moore 30 USA Sailboat – Jali Mäkilä Finland Catalina 275 Sport USA Puffer (dinghy) USA Yoav Omer Israel Ranger 22 USA LaserPerformance UK/USA Humber Keel UK Catalina 16.5 USA Sandeq Indonesia Helsen 22 USA CS 44 Canada MC Scow USA Brandworkers Int’l USA Nonsuch 36 Canada Kaep Palau David Barnes New Zealand Hunter 45 DS USA Pearson 26 USA Joanna Burzyńska Poland Hunter 33.5 USA Inclusion Catamaran Portugal Newport 214 USA To investigate this, we conduct the first systematic comparison of fairness between reasoning and non-reasoning rerankers. We want to answer the following research questions: RQ1 How does query formulation impact fairness and relevance in both reasoning and non-reasoning rerankers? RQ2 Do reasoning rerankers exhibit different fairness characteristics compared to non-reasoning rerankers? RQ3 How does fairness vary across different demographic attributes for reasoning vs non-reasoning rerankers? This work examines exposure-based group fairness, where documents from different demographic groups are expected to receive visibility proportional to their representation among relevant documents and to real-world population statistics. We assess fairness using Attention Weighted Rank Fairness (AWRF), which quantifies how closely the position-weighted exposure distribution across protected groups matches a target fair distribution. Using the TREC 2022 Fair Ranking Track dataset [ 5 ] , we evaluate models across multiple sensitive attributes: age, alphabetical (topic-based), gender, languages, occupation, popularity, source geography, and subject geography. We experiment with two query settings, original keyword queries, and rewritten queries generated via gpt-4o-mini [ 17 ] . Our contributions are as follows: • We conduct the first systematic comparison of fairness between reasoning and non-reasoning rerankers for information retrieval. • We demonstrate that reasoning capabilities, as currently implemented, neither improve nor harm fairness compared to non-reasoning approaches. • We show that query formulation substantially impacts both relevance and fairness, with natural language queries improving effectiveness for all models. • We find demographic differences in fairness, particularly subject geography attribute consistently showing 10-15% lower fairness than other attributes across all models and retrieval settings. 2 Background 2.1 Reasoning vs. Non-Reasoning Rerankers Rerankers refine an initial list of candidate documents (from BM25 [ 21 ] or dense retrieval like Qwen3 [ 31 ] ) by re-evaluating their relevance to a given query. Traditional rerankers learn implicit notions of relevance from labeled pairs, while recent approaches integrate explicit reasoning to simulate step-by-step judgment at query time. We distinguish between reasoning and non-reasoning rerankers. Non-reasoning rerankers predict a scalar relevance score or an ordered list of document id without generating explicit reasoning chains [ 16 , 18 ] . Reasoning rerankers, such as Rank1 [ 28 ] , generate intermediate reasoning steps before assigning a relevance score. These models aim to emulate human-like thinking trained with distillation from larger reasoning models such as DeepSeek R1 [ 28 , 29 ] or explicit preference optimization [ 32 ] . This difference makes reasoning rerankers a good test case for fairness. 2.2 Fairness in Ranking Fairness in ranking has been a subject long studied, from resume ranking for jobs [ 6 ] to recommendation systems [ 1 ] . The challenge of balancing relevance with fairness has been approached from multiple perspectives, including individual fairness [ 4 ] , group fairness [ 30 ] , and exposure-based fairness [ 24 , 3 ] . Early work on fair ranking focused on ensuring that qualified candidates receive appropriate representation in ranked lists. Zehlike et al. [ 30 ] introduced FA*IR, an algorithm that ensures minimum representation of protected groups. Singh and Joachims [ 24 ] proposed fairness of exposure metrics, arguing that