Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search Conference: AI Agent for Information Retrieva; August 09–13, 2026; Jeju Island, Republic of Korea CCS: Information systems Information retrieval query processing CCS: Information systems Recommender systems CCS: Human-centered computing Social networking sites CCS: Information systems Recommender systems CCS: Computing methodologies Natural language generation

Ping Liu, Qianqi Shen, Jianqiang Shen, Yunxiang Ren, Wenqiong Liu, Chunnan Yao, Dan Xu, Baofen Zheng, Rajat Arora, Kevin Kao, Wanjun Jiang, Andrii Soviak, Jingwei Wu, Wenjing Zhang · 2026

Read Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search Conference: AI Agent for Information Retrieva; August 09–13, 2026; Jeju Island, Republic of Korea CCS: Information systems Information retrieval query processing CCS: Information systems Recommender systems CCS: Human-centered computing Social networking sites CCS: Information systems Recommender systems CCS: Computing methodologies Natural language generation by Ping Liu, Qianqi Shen, Jianqiang Shen, Yunxiang Ren, Wenqiong Liu, Chunnan Yao, Dan Xu, Baofen Zheng, Rajat Arora, Kevin Kao, Wanjun Jiang, Andrii Soviak, Jingwei Wu, Wenjing Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27291.txt.

Introduction

Liu and colleagues design reward signals for teaching models to write portable job search queries. Such queries must strip out seeker-specific details while keeping transferable skills. Their training framework learns from AI feedback rather than human labels. The reward surface is adversarial: optimisers exploit flaws in judge rubrics and degenerate into copying profile text verbatim. Experiments isolate optimisation mechanics against structured reward engineering in this industrial case study from 2026 on semantic job search.

For critic-free learners, robust reward shaping overwhelmingly decides performance, making algorithm choice nearly irrelevant. Some baseline methods natively resist reward hacking. Group-relative advantage normalisation, by contrast, proves uniquely sensitive to spurious rewards. A deterministic rule-based reward floor that penalises plain copying fixes that failure mode. Quality on an outside judge family improves substantially. Training-time judges also inflate apparent gains, warning against self-graded success.

Catalog record

CREATORPing Liu, Qianqi Shen, Jianqiang Shen, Yunxiang Ren, Wenqiong Liu, Chunnan Yao, Dan Xu, Baofen Zheng, Rajat Arora, Kevin Kao, Wanjun Jiang, Andrii Soviak, Jingwei Wu, Wenjing Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.27291

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search Conference: AI Agent for Information Retrieva; August 09–13, 2026; Jeju Island, Republic of Korea CCS: Information systems Information retrieval query processing CCS: Information systems Recommender systems CCS: Human-centered computing Social networking sites CCS: Information systems Recommender systems CCS: Computing methodologies Natural language generation Ping Liu, Qianqi Shen, Jianqiang Shen, Yunxiang Ren, Wenqiong Liu, Chunnan Yao, Dan Xu, Baofen Zheng, Rajat Arora, Kevin Kao, Wanjun Jiang, Andrii Soviak, Jingwei Wu, Wenjing Zhang Abstract . Job-search platforms rely on low-bandwidth query interfaces that often fail to capture the high-dimensional complexity of candidate profiles. We present an end-to-end RLAIF (Reinforcement Learning from AI Feedback) framework to generate portable job search queries, terms that abstract away seeker-specific identifiers while preserving generalizable qualifications. This task introduces a highly adversarial reward surface where policy optimization frequently exploits flaws in LLM-as-judge rubrics, resulting in degenerate verbatim-copying behaviors. We conducted comprehensive empirical experiments to isolate the impact of optimization mechanics against structured reward engineering. Our results demonstrate that for critic-free optimizers, performance is overwhelmingly dictated by robust reward shaping, rendering the specific choice of algorithm largely immaterial. While critic-free per-rollout baseline methods (RLOO and REINFORCE++) natively resist reward-hacking, the group-relative advantage normalization in GRPO appears uniquely sensitive to spurious reward signals, making it disproportionately susceptible to exploitation. We show that introducing a deterministic, rule-based reward floor to correct for rewards assigned to verbatim copying mitigates this failure mode, resulting in a substantial + 0.147 +0.147 quality improvement on a cross-family evaluation judge. Ultimately, we show that the training-time reward model inflates performance gains by 2.4 × 2.4\times , confirming that the training success is fundamentally dependent on enforcing reward-shaping disciplines rather than selecting alternative optimizers. † † cc-license: by 1. Introduction The efficiency of a modern job market is fundamentally limited by the lexical gap between a candidate’s profile and a recruiter’s job posting. In professional ecosystems like LinkedIn, users navigate this gap through a low-bandwidth interface: the search bar. A job seeker provides a few keywords, and the platform returns a ranked list of job postings. If that query is suboptimal, such as omitting a transferable skill, specifying a role at the wrong level of generality, or failing to translate across language barriers, it acts as a silent filter, limiting opportunities for which the seeker is eminently qualified. Query suggestion, the automated transformation of member profile into a concise search query, bridges the gap mentioned above. Effective suggestions are not merely a convenience; they are a precondition for equitable access to the opportunity surface. This is particularly true for newly graduated students, career-transitioning candidates, multilingual workers, and seekers in markets where formal labor vocabulary diverges from lived experience. Building a production-grade generative suggestion system is not a standard summarization task. It requires navigating a sharp trade-off between two competing properties: 1. Grounding: The queries must be strictly derived from the seeker’s actual qualifications to avoid hallucinated career trajectories; 2. Portability: The query must abstract away seeker-specific context (e.g., hyper-local geographies, or current employer names) to ensure it retrieves a broad set of market opportunities rather than just a reflection of the seeker’s own profile. This tension is further complicated by heterogeneous sparsity . Profiles vary from single-sentence headlines to multi-page CVs and often span multiple languages. A model that cannot cleanly “refuse” to suggest a query for an uninformative profile creates a liability of fabrication. While Supervised Fine-Tuning (SFT) provides a baseline, it often fails to capture the nuance of portability and complicated product policy. Reinforcement Learning from AI Feedback (RLAIF) allows us to treat the reward signal as a programmable proxy for human judgment to the product policy. However, shifting to RLAIF moves the core technical challenge from model architecture to reward engineering . In an industrial context, Goodhart’s Law ( 10 ; 19 ) serves as the primary antagonist. When a policy is optimized against a rubric-prompted LLM judge, it quickly discovers optimization short-circuits that satisfy the rubric’s surface features without improving query quality. We observed that the design of the reward signal can be a more potent lever for performance than the choice of RL optimizer. This paper provides an end-to-end account of building a rubric-graded RLAIF pipeline on industrial job marketplace data. Our central observation is that the dominant challenge in industrial RLAIF lies within the reward signal formulation, rather than the choice of optimizer . Our contributions are as follows: • Formalization of the Portable-Query Problem, distinguishing it from traditional document expansion and summarization tasks. • A reward-hacking case study characterizing a verbatim-copy failure mode and presenting two concrete architectural mitigations: a deterministic rule-based reward floor ( S_{r} ) that detects verbatim copying (6-gram profile overlap and lifted date-ranges) to clamp the reward, and a multi-dimensional rubric redesign that explicitly penalizes copying. • Empirical evidence showing that while optimization algorithms (such as GRPO, RLOO, and PPO) cluster closely in baseline performance, the introduction of rule-based reward shaping provides the most significant shift in production quality. 2. Related Work Query Generation. Generative query production has evolved from rewriting user input with RL ( 21 ) to expanding documents with predicted queries at indexing or inference time ( 22 ; 3 ; 29 ; 8 ; 30 ) . While prior talent search work focuses on ranking ( 24 ; 9 ; 16 ) , our system generates a portable query from a profile for use across many candidates. RL Training for LLMs. Modern on-policy LLM optimization, traditionally reliant on PPO for RLHF ( 4 ; 28 ; 23 ; 25 ) , has increasingly adopted simpler, critic-free architectures that often replace human feedback with AI-driven critiques ( 26 ; 1 ; 12 ; 2 ; 15 ) . Our work uses a critic-free framework with a prompted rubric grader as its reward signal, rather than a trained reward model. Reward Design and Reward Hacking. Optimizing a policy against a proxy reward risks reward hacking, where the policy exploits the proxy without achieving the intended goal ( 10 ; 19 ; 7 ) . Systems can mitigate this with rule-based reward shaping, though AI graders themselves introduce vulnerabilities like sycophancy ( 20 ; 14 ; 5 ; 27 ) . We address this by integrating a multi-dimensional rubric with a deterministic 6-gram copy detector. Evaluation of Generative Search Outputs. LLM-as-a-judge frameworks are now standard for evaluating generative text, replacing brittle reference-based metrics despite known position and length biases ( 32 ; 31 ; 6 ) . Robust validation requires a holistic approach, including dedicated metrics for factual grounding ( 11 ; 13 ; 17 ) . Our system thus employs a three-layer evaluation framework with a training-time rubric judge (Layer 1), text-based metrics, and a stricter independent judge ( L_{i} ). 3. Problem Setup 3.1. Task: Profile-to-Portable-Query Let p denote a member profile, represented as a structured text document containing the member’s headline, current and past experience entries, education history, languages, and (optionally) a summary section. Let q denote a short keyword query, typically 2 – 6 tokens, that a job seeker would type into a search bar. We seek a policy \pi_{\theta}(q\mid p) that maps a profile to

Read the complete text