QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards
Rongzhi Zhang, Rui Feng, Zhihan Zhang, Qingyu Yin, Jingfeng Yang, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, Chao Zhang · 2026
Read QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards by Rongzhi Zhang, Rui Feng, Zhihan Zhang, Qingyu Yin, Jingfeng Yang, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, Chao Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03968.txt.
Introduction
Zhang and co-authors tackle training with rubric rewards beyond checkable answers. Past rubric methods kept the query set fixed while tuning rubrics. They find a block in that split. Open queries yield vague rubrics that give weak guidance. Narrow queries can invent facts no model can verify. Then all replies fail and training gets no signal.
QUBRIC designs queries and rubrics side by side. Key points from a teacher ground the rewrite of open queries. New queries take a scenario form that can be judged. Contrast between teacher and policy replies shapes each rubric. A filter keeps just the pairs that teach something. Gains reached 5.5 points on ArenaHard over supervised training. Transfer gains also showed on legal, moral, and story tasks.
Catalog record
| CREATOR | Rongzhi Zhang, Rui Feng, Zhihan Zhang, Qingyu Yin, Jingfeng Yang, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, Chao Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03968
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards Rongzhi Zhang, Rui Feng, Zhihan Zhang, Qingyu Yin, Jingfeng Yang, Xin Liu, Zixuan Zhang, Priyanka Nigam, Bing Yin, Tuo Zhao, Chao Zhang Abstract Rubric-based RL is a promising route for extending reinforcement learning beyond verifiable rewards, yet existing methods optimize rubrics while treating the query distribution as fixed. We identify a structural bottleneck: rubric quality is constrained by query structure . Open-ended queries yield vague rubrics; naively narrowing them introduces fabricated references that no model can verify, so all responses fail and training receives no reward signal. We present QUBRIC, a framework that co-designs queries and rubrics. Teacher-derived key points ground the rewriting of open-ended queries into scenario-based, evaluable questions. Contrastive rubric generation then turns teacher–policy gaps into query-level criteria, and learnability filtering retains only informative query–rubric pairs for GRPO training. QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline. Trained only on instruction-following data, it further transfers to three held-out benchmarks spanning legal, moral, and narrative reasoning (+6.3 points on average), with improvements concentrated in reasoning-related dimensions. These results provide evidence that co-designing queries and rubrics can make rubric-based RL a practical complement to RLVR beyond strictly verifiable tasks. 1 Introduction Reinforcement learning with verifiable rewards (RLVR) has driven significant capability gains in mathematics and coding ( 20 ; 4 ; 12 ) , where ground-truth answers provide a clean reward signal. Extending RL beyond verifiable domains requires alternative reward signals. Recent work either decomposes rewards into structured rubrics evaluated by LLM judges ( 24 ; 7 ; 27 ; 25 ) or reshapes queries for RL training ( 6 ; 15 ; 11 ) , but these two lines develop in isolation, leaving underexplored the interaction between query structure and evaluability. QUBRIC targets a harder regime where queries are genuinely open-ended and the primary bottleneck is not rubric phrasing but making the query evaluable in the first place. We argue that query structure and rubric design are fundamentally coupled: rubric quality is constrained by query structure . The most reliable rubrics specify exactly what to check, but writing such rubrics requires that the query be specific enough to determine what a good answer should contain. For a broad query like “explain machine learning,” the space of valid responses is too large for any rubric set to enumerate; the alternative is vague criteria that judges cannot apply consistently. In either case, the reward signal becomes noisy. A more serious failure arises under naive narrowing: to make a query look specific, the rewriter often introduces concrete but unverifiable references—citing a guideline, glossary, or document that does not exist—after which the rubrics collapse into testing whether the model refuses to answer rather than whether it reasons correctly, yielding uninformative reward signals (Table 4 ). Ablations confirm both effects (Section 5.2 ). Co-design is therefore not a convenience but a requirement: the query must be narrowed in a way that gives the rubric generator a determinate reasoning target. QUBRIC operationalizes this coupling through a single query-rubric pipeline (Figure 1 ). Key points extracted from multiple teacher reference responses (used to construct evaluation targets, not for distillation) ground the rewriting of open-ended queries into scenario-based, evaluable questions, avoiding the fabricated-reference failure of ungrounded narrowing. Contrastive rubric generation then compares teacher responses against policy responses and extracts the gap into precise query-level criteria, replacing the vague rubrics that arise from under-specified queries. A learnability filter retains only query–rubric pairs where the policy’s pass rate falls within a difficulty corridor, concentrating training on the most informative examples. These per-rubric binary rewards then drive policy optimization via GRPO. We evaluate QUBRIC across instruction following, cross-domain transfer, and shopping helpfulness. On challenging instruction-following benchmarks, QUBRIC achieves a +5.5 point gain on ArenaHard ( 13 ) over the SFT baseline. Trained only on instruction-following data, it further transfers to three held-out benchmarks spanning legal, moral, and narrative reasoning (+6.3 points on average), with improvements concentrated in reasoning-related dimensions. These results suggest that co-designing queries and rubrics enables effective rubric-based RL for improving instruction following and transferable reasoning performance beyond strictly verifiable tasks. Our contributions are as follows: \bullet We identify a structural coupling between query structure and rubric quality: when queries lack sufficient structure, rubrics degenerate into vague or unverifiable criteria that yield uninformative rewards (Section 5.2 ). \bullet We propose QUBRIC, which operationalizes this coupling through key-point-grounded query rewriting, contrastive rubric generation, and learnability filtering (Section 3 ). \bullet We evaluate QUBRIC on instruction-following benchmarks, three held-out reasoning benchmarks, and a proprietary shopping benchmark; matched controls confirm that the gains require jointly constructing evaluable queries and rubrics rather than rubric RL alone or query narrowing alone (Sections 4 – 5 ). 2 Related Work From RLHF to Rubric-Based RL. RLHF ( 17 ; 3 ) trains a monolithic scalar reward that is vulnerable to reward misspecification ( 18 ; 22 ) and offers limited diagnostic visibility. A growing body of work addresses this by decomposing the reward into structured criteria. One family uses fixed or instance-specific rubrics ( 16 ; 24 ; 7 ; 27 ) ; a second makes rubrics or judges adaptive during training ( 10 ; 25 ; 19 ) . These methods vary in whether rubrics are static or adaptive and whether grading knowledge is encoded in the rubric or delegated to the judge, but they generally assume the training prompt itself is fixed. QUBRIC instead modifies the prompt so that precise, content-bearing rubric evaluation becomes possible. Query Design for RL Training. A parallel line of work focuses on constructing better training queries. AutoIF ( 6 ) augments queries with Python-verifiable constraints; Golden Goose ( 15 ) synthesizes fill-in-the-middle tasks as multiple-choice RLVR problems; and RLVRR ( 11 ) extracts ordered reward chains from reference answers. These approaches primarily optimize query construction for executable or reference-based supervision, rather than for downstream rubric gradeability. QUBRIC also rewrites prompts, but for a different purpose: not to manufacture executable verification, but to produce evaluable, self-contained scenarios whose answer space is narrow enough for precise rubric generation, with teacher-derived key points preserving the original task intent throughout the rewrite. RLVR, Verifiability, and the Scope of QUBRIC. Recent work raises concerns that domain-specific RLVR may reinforce superficial heuristics rather than acquiring transferable reasoning ( 1 ; 4 ; 12 ) . QUBRIC targets an intermediate regime between exact verification and unconstrained preference judgments: tasks without a single gold answer, but with decomposable, scenario-grounded criteria that can be checked reliably enough for RL training. In summary, one line of work changes the reward while holding queries fixed; another changes the queries to obtain executable supervision. QUBRIC lies at their intersection, rewriting queries to support precise rubric evaluation in tasks without exact answer keys. 3 Method: QUBRIC QUBRIC is a three-stage pipeline that jointly designs queries and rubrics for RL training (Figure 1 ). We first rewrite open-ended queries into scenario-grounded queries with narrower answer spaces (§ 3.1 ). We then generate query-level rubrics offline by contrasting teacher responses with policy responses (§ 3.2 ). Finally, we combine these query-level rubrics with global rubrics and optimize the policy with GRPO (§ 3.3 ). All query rewriting and