PJB: A Reasoning-Aware Benchmark for Person-Job Retrieval
Guangzhi Wang, Xiaohui Yang, Kai Li, Jiawen He, Kai Yang, Ruixuan Zhang · 2026
Read PJB: A Reasoning-Aware Benchmark for Person-Job Retrieval by Guangzhi Wang, Xiaohui Yang, Kai Li, Jiawen He, Kai Yang, Ruixuan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17386.txt.
Introduction
Wang and co-authors build PJB, a benchmark for matching people to jobs. Full job posts serve as queries and full resumes as documents. Labels mark domain families and reasoning types, so users see where systems fail and why. The data span six industries and nearly 200,000 resumes. Dense retrieval varied far more across domains than across model upgrades, they find. Reranking helped steadily, while query rewriting hurt.
Catalog record
| CREATOR | Guangzhi Wang, Xiaohui Yang, Kai Li, Jiawen He, Kai Yang, Ruixuan Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48838 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17386
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PJB: A Reasoning-Aware Benchmark for Person-Job Retrieval Guangzhi Wang, Xiaohui Yang, Kai Li, Jiawen He, Kai Yang, Ruixuan Zhang Abstract As retrieval models converge on generic benchmarks, the pressing question is no longer “who scores higher” but rather “where do systems fail, and why?” Person-job matching is a domain that urgently demands such diagnostic capability—it requires systems not only to verify explicit constraints but also to perform skill-transfer inference and job-competency reasoning, yet existing benchmarks provide no systematic diagnostic support for this task. We introduce PJB (Person-Job Benchmark), a reasoning-aware retrieval evaluation dataset that uses complete job descriptions as queries and complete resumes as documents, defines relevance through job-competency judgment, is grounded in real-world recruitment data spanning six industry domains and nearly 200,000 resumes, and upgrades evaluation from “who scores higher” to “where do systems differ, and why” through domain-family and reasoning-type diagnostic labels. Diagnostic experiments using dense retrieval reveal that performance heterogeneity across industry domains far exceeds the gains from module upgrades for the same model, indicating that aggregate scores alone can severely mislead optimization decisions. At the module level, reranking yields stable improvements while query understanding not only fails to help but actually degrades overall performance when combined with reranking—the two modules face fundamentally different improvement bottlenecks. The value of PJB lies not in yet another leaderboard of average scores, but in providing recruitment retrieval systems with a capability map that pinpoints where to invest. 1 Introduction The competitive focus of AI is shifting from pretraining scale and general capability demonstrations toward utility verification and system deployment for real-world tasks [ 16 ] . For evaluation, this shift means that benchmarks must go beyond answering “how strong is the model on average” and further address “whether the model is truly useful in complex, high-stakes, inference-demanding real-world scenarios.” The development of retrieval benchmarks clearly reflects this evolution: BEIR [ 11 ] and MTEB [ 5 ] expanded evaluation from single datasets to cross-domain, cross-task zero-shot generalization; BRIGHT [ 9 ] further focused on reasoning-intensive retrieval, probing whether systems can handle complex queries that cannot be resolved through lexical matching or shallow semantic similarity alone. However, these benchmarks remain largely confined to general or semi-general retrieval scenarios and have yet to systematically address recruitment—a real-world task characterized by long documents, multiple constraints, and job-competency-driven relevance judgments. Person-job matching is precisely such a compound retrieval problem: on the surface, it retrieves suitable candidates from job descriptions, but in reality it simultaneously involves two types of reasoning demands. Parallel reasoning requires systems to independently verify explicit constraints—location, education, years of experience, salary range, and job keywords—and synthesize the results, a class of multi-constraint retrieval problems that has received increasing attention in recent compound retrieval research [ 4 ] . Serial reasoning, by contrast, requires systems to perform multi-hop semantic abstraction around job competency—for example, mapping job responsibilities to implicit skill requirements, interpreting cross-industry experience as transferable capabilities, and aggregating candidate evidence across multiple resume sections—bearing structural similarity to the cross-document evidence chain reasoning studied in multi-hop retrieval [ 10 ] . Because both types of reasoning coexist, person-job matching can be reduced to neither structured filtering nor standard text similarity ranking; nor can a single aggregate metric adequately characterize system utility on real business queries. To evaluate such systems, a benchmark must go beyond overall ranking quality and explain where systems fail—across which domain families, reasoning types, and low-gain queries. To this end, we propose PJB (Person-Job Benchmark), a reasoning-aware benchmark that formalizes the matching of complete job descriptions and complete resumes as a reproducible, diagnostic offline retrieval evaluation dataset, designated PJB v1.0. PJB v1.0 is constructed from real recruitment data and, like mainstream retrieval benchmarks, adopts a fixed query set, fixed document corpus, and fixed relevance judgments in an offline evaluation setting. Through domain-family and reasoning-type diagnostic labels, the benchmark goes beyond reporting aggregate scores to localize system capability structures. Specifically, the contributions of this paper are as follows: 1. We formalize person-job matching as a job-competency-driven retrieval task and construct an evaluation dataset comprising nearly 300 queries, nearly 200,000 resumes, and over 2,000 positive relevance judgments, along with domain-family and reasoning-type label systems that support diagnostic analysis. 2. Using dense retrieval as the entry point, we conduct unified evaluation across different model versions and module combinations, revealing pronounced domain heterogeneity and reasoning heterogeneity in person-job retrieval, with the reranking module being the most stable source of improvement. 3. On the data construction and usage boundary, we address the bias [ 14 , 12 ] and privacy risks [ 15 ] specific to recruitment scenarios through de-identification and compliance measures, described in the method section. These results demonstrate that PJB can serve as a more business-relevant unified baseline for comparing, diagnosing, and subsequently optimizing recruitment retrieval systems. 2 Related Work From the perspective of retrieval evaluation evolution, PJB inherits a lineage from fixed-collection scoring toward cross-domain and diagnostic benchmarks. Cranfield established the offline evaluation paradigm of fixed document collections, fixed query sets, and fixed relevance judgments [ 2 ] , and TREC extended this paradigm to large-scale pooling-based evaluation [ 13 ] . Subsequent work addressed the incomplete judgment problem inherent in pooling by proposing more robust methods such as bpref and sampling-based estimation, aiming to reduce the bias from treating unjudged documents as irrelevant [ 1 , 17 ] . With the advent of representation learning and universal embedding models, benchmark focus shifted further from aggregate scores on a single collection toward cross-dataset, cross-task, and zero-shot generalization, with BEIR and MTEB representing this heterogeneous evaluation turn [ 11 , 5 ] ; BRIGHT then pushed this trajectory toward reasoning-intensive retrieval, emphasizing that benchmarks should distinguish not only “whether something can be retrieved” but also “whether the system can handle queries requiring inference and evidence organization” [ 9 ] . Concurrently, LLM-as-a-Judge has begun to alleviate the cost bottleneck of large-scale relevance annotation, though it introduces its own calibration and bias risks [ 8 ] . Overall, retrieval evaluation has increasingly prioritized cross-domain generalization and complex reasoning capabilities, yet rarely anchors the evaluation target in real-world person-job matching—a long-document, multi-constraint, competency-judgment scenario. From the application perspective, person-job matching research itself has evolved from structured feature alignment to semantic matching, and further to knowledge-enhanced and LLM-assisted systems. Early systems relied predominantly on explicit fields—education, years of experience, skill keywords, and job titles—for rule-based matching or ranking, which handled enumerable constraints effectively but struggled with cross-industry experience transfer, implicit skill mapping, and long-text responsibility understanding. As the need for modeling job descriptions and resume text grew, PJFNN [ 18 ] and APJFNN [ 7 ] formulated person-job matching as joint representation learning and interaction-aware matching between job text and resume text, enabling models to learn finer-grained competency alignment than manual features. Subsequent work further incorporated skill–occupation graph context and LLM distillation, attempting to bridge the implicit semantic gap between job requirements and candidate experience, embedding person-job matching capabilities into broader HR NLP pipelines [ 6 ] . However, this field remains largely organized around paired scoring or ranking optimization on proprietary data, with evaluation protocols, annotation standards, and error analysis dimensions varying across tasks and organizations, making it difficult to compare method improvements