RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses Feiyu Wu Xu Zheng Zhuocheng Wang Yi ming Dai Hui Li Abstract Large language models (LLMs) make reward design in reinforcement learning substantially more scalable, but generated rewards are not automatically reliable training objectives. Existing work has focused primarily on generating, evolving, or selecting reward candidates, while paying less attention to when such candidates can be verified and deployed during policy optimization. We study this deployment-time problem by treating generated rewards as reward hypotheses whose utility depends on the competence of the current policy and the phase of training. We propose RHyVE , a competence-aware verification and phase-aware deployment protocol that compares small sets of reward hypotheses from shared policy checkpoints using short-horizon fork verification. Our experiments show that reward rankings are unreliable at low competence but become informative after task-dependent thresholds. On a sparse manipulation task, phase-aware deployment improves peak and retained performance under a locked protocol. Updated LLM-generated reward-candidate experiments show candidate-family-dependent behavior: generated pools can exhibit phase-dependent winner changes, but no fixed warm-up schedule is universally optimal. Held-out schedule selection, conservative selector baselines, compute-matched controls, and scale controls further show that RHyVE is best understood as a verification-informed deployment protocol rather than a universal scheduler. Dense and all-failure boundary experiments delimit the scope of the method. Together, these results suggest that reward generation and reward deployment should be studied as coupled problems: generated rewards must be verified and deployed under changing policy competence. 1 Introduction Reward design remains a central bottleneck in reinforcement learning: good shaping rewards can accelerate learning, while misspecified rewards can induce proxy optimization, reward hacking, and poor generalization 25 ; 37 ; 9 ; 3 ; 11 ; 10 ; 32 ; 28 . A broad literature studies how to obtain rewards, including inverse RL, preference-based RL, RLHF, and structured reward specification 26 ; 1 ; 42 ; 12 ; 8 ; 18 ; 33 ; 27 ; 4 ; 34 ; 7 ; 35 ; 6 . Yet once a candidate reward is available, an equally practical question remains: when is it reliable enough to drive optimization? This question is especially important for LLM-based reward design. Recent systems synthesize reward code, dense shaping functions, and online reward selectors from language 22 ; 38 ; 20 ; 17 ; 40 ; 21 ; 29 . These methods can generate plausible candidates, but usually treat reward quality as a property of the candidate alone: generate a set of rewards, evaluate or refine them, and deploy the selected one. Our central observation is that this view is incomplete. The apparent quality of a generated reward can depend strongly on the competence of the current policy and on the phase of training in which the reward is used. We therefore argue that generated rewards should be treated as reward hypotheses rather than immediately reliable objectives. A reward that is useful for late-stage optimization may be too sparse, too sharp, or too poorly scaled to bootstrap learning from a weak policy. Conversely, a dense reward that exposes coarse progress signals early in training may later become misaligned with the task objective. This view connects reward deployment to curricula and adaptive objectives 5 ; 13 ; 14 ; 24 ; 19 ; 36 ; 39 ; 41 , but sharpens the deployment question in the reward-design setting: the issue is not only which reward is best, but also when the learner is competent enough to verify that fact. A second deployment challenge arises when the training objective changes. Switching rewards mid-training can create non-stationary targets for the actor and critic, produce reward-scale shocks, and destabilize value estimation. Potential-based shaping and trust-region methods motivate caution, but they do not guarantee that an immature learned critic can safely support an online reward switch 30 ; 2 ; 25 ; 37 ; 9 . Reward deployment therefore requires attention both to verification timing and to switching stability. We therefore propose RHyVE , a competence-aware verification and phase-aware deployment protocol for small sets of reward hypotheses. RHyVE compares candidates from shared policy checkpoints using short-horizon fork verification, records when winner identity becomes reliable, and uses the resulting phase profile to decide whether a single reward suffices or whether a two-stage schedule is warranted. The method is deliberately local in scope: it is not a new reward generator and not a universal adaptive scheduler, but a protocol for deciding when already available candidates are trustworthy enough to compare and deploy. In this sense, RHyVE addresses the missing middle between reward generation and reward commitment. This paper makes the following contributions: • We formulate generated rewards as hypotheses whose reliability depends on policy competence and training phase, separating reward generation from reward verification and deployment. • We introduce shared-checkpoint fork verification and phase-profile construction for small candidate sets, together with deployment rules that choose a single reward, a two-stage schedule, or conservative fallback depending on verification reliability. • We show on the locked FrankaCabinet protocol that phase-aware deployment improves both peak and recomputed terminal performance in a sparse phase-sensitive manipulation regime. • We extend the evidence to LLM-generated reward-candidate families under a reduced 6 \times 3090 protocol, showing candidate-family-dependent deployment behavior rather than universal warm-up optimality. • We provide controls and scope evidence, including held-out schedule selection, conservative online selector baselines, compute-matched direct training, reward-scale controls, dense-oracle boundary behavior, candidate-pool stress tests, and an all-failure extra-task pilot. 2 Related Work Recent work shows that large language models and foundation models can automate substantial parts of reward engineering. Eureka, Text2Reward, Auto MC-Reward, REvolve, and R* synthesize or refine executable reward programs and shaping functions; ORSO is the closest operational prior because it studies online reward selection during training; and pretrained vision-language models can act as zero-shot reward models. These works mainly study how to generate, refine, search over, or reactively select reward candidates. RHyVE studies a complementary deployment-time question: when does a small candidate set become reliable to compare from the current learner state, and should the resulting evidence lead to a single reward, a phase-aware schedule, or conservative fallback? Our conservative selector baselines are included to test reactive alternatives, but we do not claim to reproduce or dominate the full ORSO system. Learned and structured reward methods reduce manual reward engineering by inferring or constraining reward signals from demonstrations, preferences, human feedback, or formal task structure 26 ; 1 ; 42 ; 12 ; 8 ; 18 ; 33 ; 27 ; 4 ; 16 ; 34 ; 7 ; 35 ; 6 . Curriculum, teacher-student, population-based, and meta-gradient methods likewise exploit the fact that useful supervision can change during learning 5 ; 13 ; 14 ; 24 ; 19 ; 36 ; 39 ; 41 . Our focus is narrower: given already available rewards, we ask when the current learner is competent enough for local comparison, when reward utility changes across phases, and when a fixed phase-aware deployment rule is warranted. Reward shaping and safe optimization are directly relevant once the objective changes. Potential-based shaping gives policy-invariance guarantees under suitable assumptions 25 ; 37 ; 9 ; 15 , while trust-region, constrained, and safety-oriented perspectives stress that changing objectives can induce instability, proxy optimization, or reward tampering 30 ; 2 ; 3 ; 11 ; 10 ; 32 ; 28 . RHyVE does not propose a universal switch operator; instead, it separates competence-aware verification from the trade-offs of how a reward change is executed, which is why our experiments treat hard switching, PBRS, and critic reset as conditional mechanisms rather than interchangeable defaults. 3 Problem Setup and Method In RHyVE , we study the deployment-time problem that arises after an upstream reward-design procedure, such as an LLM-based reward generator, has produced a small set of candidate rewards. Our goal is not to generate new reward code, but to decide when candidate rewards are reliable to compare, when they should be deployed, and how objective changes should be introduced without destabilizing training. The central distinction is that candidate rewards are treated as hypotheses whose usefulness may depend on learner competence rather than as immediately trustworthy objectives. The method is intentionally modest in scope. RHyVE is designed for the practically common regime in which a small number of structured candidate rewards is already available and the learner’s competence changes over training. 3.1 Reward Hypotheses We consider an episodic Markov decision process \mathcal{M}=(\mathcal{S},\mathcal{A},P,\rho_{0},\gamma), where \mathcal{S} and \mathcal{A} are the state and action spaces, P is the transition kernel, \rho_{0} is the initial-state distribution, and \gamma\in(0,1) is the discount factor. Instead of assuming a single fixed reward, we are given a small candidate set \mathcal{H}=\{h_{1},\ldots,h_{K}\}, where each hypothesis h_{k} induces a reward function r_{h_{k}}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}. We use the term reward hypothesis deliberately. A generated reward is not assumed to be immediately trustworthy, globally optimal, or useful at every training phase. Rather, it is a candidate training signal whose value may depend on the competence of the current policy. Let z_{t}=(\pi_{t},V_{t},\Omega_{t}) denote a training checkpoint at optimization step t , consisting of the current policy \pi_{t} , critic V_{t} , and optimizer or training state \Omega_{t} . Associated with each checkpoint is a task-dependent competence proxy c_{t}=c(z_{t}), such as success rate, task progress, or another downstream metric. We do not assume a universal competence scale across tasks; competence is used only to determine whether the current learner is in a regime where reward comparisons are informative. Figure 1: Overview of RHyVE . Reward candidates are treated as hypotheses, compared from shared checkpoints using fork verification, and deployed as a single reward, two-stage schedule, or conservative fallback depending on the phase profile. 3.2 Shared-Checkpoint Fork Verification The central operation in RHyVE is shared-checkpoint fork verification. At a checkpoint z_{t} , each reward hypothesis h_{k} is evaluated by cloning the same learner state, continuing training for a short fork horizon L under reward r_{h_{k}} , and evaluating the resulting forked policy with a common downstream metric: J_{t}^{(L)}(h_{k})=\operatorname{Eval}\!\left(\operatorname{Train}(z_{t},r_{h_{k}},L)\right). The local winner at checkpoint t is \hat{h}_{t}=\arg\max_{h_{k}\in\mathcal{H}}J_{t}^{(L)}(h_{k}), and its margin over the runner-up is m_{t}=J_{t}^{(L)}(\hat{h}_{t})-\max_{h_{j}\neq\hat{h}_{t}}J_{t}^{(L)}(h_{j}). Shared checkpoints avoid confounding reward quality with independent initialization, exploration history, and optimizer state. The fork horizon L is intentionally short: RHyVE estimates which reward is locally useful for the current learner rather than fully optimizing every candidate. This keeps verification aligned with the practical question of whether a reward should be deployed from the current policy state. If M checkpoints are probed, K hypotheses are compared, and each fork runs for L update steps, the additional verification cost scales as O(MKL). Thus RHyVE is designed for small candidate sets, sparse checkpoint probes, and short local verification windows. 3.3 Verification-Informed Phase Profiles Fork verification produces a phase profile over training: \mathcal{P}=\left\{(t,c_{t},\hat{h}_{t},m_{t},\{J_{t}^{(L)}(h_{k})\}_{k=1}^{K}):\,t\in\mathcal{T}\right\}, where \mathcal{T}=\{t_{1},\ldots,t_{M}\} is a sparse set of probed checkpoints. This profile records how apparent reward utility changes as the policy becomes more competent and provides a compact diagnostic of whether the candidate family exhibits a stable phase structure. Early checkpoints may be uninformative because a weak policy has not yet reached states where reward differences matter. We therefore treat a checkpoint as verification-informative only when the comparison shows sufficient separation and stability, assessed using m_{t},\qquad\operatorname{Agree}_{t},\qquad\operatorname{Ent}_{t}, where m_{t} is the winner margin, \operatorname{Agree}_{t} is repeated-fork winner agreement, and \operatorname{Ent}_{t} is winner entropy across repeated forks or nearby horizons. In practice, these diagnostics are used to distinguish true phase structure from noisy local rank fluctuations. If winner identity is unstable or margins are negligible, RHyVE abstains from aggressive reward commitment; if a winner transition stabilizes, the profile suggests phase-dependent reward utility. 3.4 Phase-Aware Deployment Given a phase profile \mathcal{P} , RHyVE deploys a single reward when one hypothesis is stable from the first informative checkpoint onward, a two-stage schedule when an early winner is later overtaken by a stable later winner, and a conservative no-switch rule when the profile remains ambiguous. The two-stage case uses r_{t}=\begin{cases}r_{h^{(1)}}&t0.01. For FrankaCabinet , margins are small because success metrics are normalized, and the same criterion is used only as a diagnostic threshold rather than a universal rule. The important qualitative pattern is that early checkpoints are less reliable and later checkpoints become more informative under sufficiently long fork horizons. Table 10: First reliable checkpoint by task and fork horizon. Reliability depends on both policy competence and verification horizon. Task Fork horizon First reliable checkpoint BallBalance 50 30 BallBalance 100 300 BallBalance 200 300 BallBalance 400 30 FrankaCabinet 50 1500 FrankaCabinet 100 1500 FrankaCabinet 200 1000 FrankaCabinet 400 1000 Figure 4: Full fork-verification reliability. Each cell aggregates repeated forks from a shared checkpoint and reports modal winner identity, top-1 reliability, and winner margin. The reliable region appears only after sufficient policy competence and depends on fork horizon. Table 11: Full FrankaCabinet fork-verification grid. The table reports modal winner and top-1 reliability for each checkpoint and fork horizon. Margins and entropy are reported in the released CSV. Checkpoint Fork horizon Modal winner Top-1 reliability Margin mean Informative 500 50 cand_b7d2e641 0.75 0.00017 no 500 100 cand_b7d2e641 0.50 0.00007 no 500 200 cand_b7d2e641 0.50 0.00057 no 500 400 cand_b7d2e641 0.50 0.01552 no 1000 50 cand_bc49b15b 1.00 0.01419 no 1000 100 cand_6ef57213 0.50 0.00755 no 1000 200 cand_6ef57213 1.00 0.09161 yes 1000 400 cand_6ef57213 1.00 0.08832 yes 1500 50 cand_bc49b15b 0.75 0.05101 yes 1500 100 cand_bc49b15b 1.00 0.05602 yes 1500 200 cand_6ef57213 0.50 0.02488 no 1500 400 cand_6ef57213 1.00 0.09381 yes 2000 50 cand_bc49b15b 0.50 0.04170 no 2000 100 cand_bc49b15b 0.50 0.01654 no 2000 200 cand_6ef57213 0.50 0.00329 no 2000 400 cand_6ef57213 1.00 0.04448 no The FrankaCabinet grid supports two observations. First, the earliest checkpoint is not verification-informative despite occasionally producing a modal winner. Second, later checkpoints become informative only under sufficiently reliable fork horizons. This justifies the main text’s competence-aware interpretation: reward ranking is not a static property of the candidate set, but depends on the current learner and the verification budget. Table 12: Full BallBalance fork-verification grid. BallBalance reaches high task competence quickly, and peak metrics saturate, making the task a boundary condition rather than a main performance benchmark. Checkpoint Fork horizon Modal winner Top-1 reliability Margin mean Informative 30 50 cand_bb_speed_bias 0.75 54.10 yes 30 100 cand_bb_speed_bias 0.75 52.59 no 30 200 cand_bb_speed_bias 0.50 39.96 no 30 400 cand_bb_speed_bias 0.75 26.16 yes 100 50 cand_bb_speed_bias 0.50 30.01 no 100 100 cand_bb_speed_bias 0.50 19.16 no 100 200 cand_bb_speed_bias 0.50 26.27 no 100 400 cand_bb_speed_bias 0.50 25.82 no 300 50 cand_bb_dist_only 0.50 17.12 no 300 100 cand_bb_pos_speed 0.75 37.02 yes 300 200 cand_bb_pos_speed 0.75 46.08 yes 300 400 cand_bb_pos_speed 0.50 30.73 no 500 50 cand_bb_dist_only 0.75 22.98 yes 500 100 cand_bb_dist_only 0.75 18.56 yes 500 200 cand_bb_pos_speed 0.50 8.45 no 500 400 cand_bb_pos_speed 0.50 17.03 no The BallBalance grid shows that reward winner identity can still change across checkpoints, but the downstream peak metric is less discriminative because all main deployment methods reach the ceiling. We therefore use BallBalance primarily to show that warm-up is not universally needed when the oracle-like reward is already dense. B.2 FrankaCabinet Locked Main Results This section expands the locked FrankaCabinet comparison from the main text. The locked comparison uses eight seeds: \{42,123,456,789,999,1234,2025,3031\}. All methods use the same task, reward-family protocol, training budget, and evaluation pipeline. Table 13 reports the aggregate peak results. Table 13: Locked FrankaCabinet peak results. wu=50 hard switching achieves the highest mean consec_max . Method n consec_max mean Std. 95% CI Direct oracle 8 0.21498 0.28109 [0.05100, 0.41471] One-shot oracle 8 0.22815 0.27984 [0.06691, 0.42390] Fixed wu=50 hard 8 0.38894 0.22560 [0.23248, 0.53150] Fixed wu=100 hard 8 0.23263 0.19816 [0.11569, 0.37862] wu=50 + PBRS 8 0.31341 0.27950 [0.13502, 0.49579] Table 14: Per-seed FrankaCabinet consec_max values for locked main methods. These values are included to make the seed variance visible. Method 42 123 456 789 999 1234 2025 3031 Direct oracle 0.251 0.026 0.146 0.000 0.003 0.602 0.689 0.002 One-shot oracle 0.251 0.026 0.251 0.000 0.003 0.602 0.689 0.002 Fixed wu=50 hard 0.033 0.505 0.602 0.180 0.655 0.380 0.210 0.546 Fixed wu=100 hard 0.634 0.206 0.205 0.271 0.233 0.306 0.004 0.002 wu=50 + PBRS 0.663 0.013 0.451 0.341 0.389 0.650 0.001 0.000 Tail-metric audit. The main text reports recomputed final and tail metrics rather than older summary fields. Table 15 summarizes the tail audit. These values are computed from full learning curves and are used to distinguish transient peaks from retained performance. Table 15: Tail-metric audit for locked FrankaCabinet runs. Final values are recomputed from the full learning curves and are used to distinguish transient peaks from retained performance. Method n Recomputed final \uparrow Direct oracle 8 0.0223 One-shot oracle 8 0.0216 Fixed wu=50 hard 8 0.1634 Fixed wu=100 hard 8 0.0893 wu=50 + PBRS 8 0.0364 Additional last-5, tail-AUC, and collapse diagnostics are visualized in Figure 6 when available from the audit logs. The tail audit strengthens the main result. The wu=50 hard-switch schedule is not only the strongest peak method; it also improves recomputed terminal and tail metrics over direct and one-shot deployment. Therefore the main result is not merely a transient success spike. Paired tests. Table 16 reports paired tests for the main peak metric. Additional paired tests for final, last-5, tail AUC, and collapse diagnostics are generated from the paper-ready audit files. Table 16: Paired sign-flip tests for FrankaCabinet consec_max . Effect sizes are moderate, but high seed variance prevents strong significance claims on the peak metric. Comparison Mean diff. p d_{z} wu50 vs direct 0.174 0.281 0.408 wu50 vs one-shot 0.161 0.305 0.384 wu50 vs wu100 0.156 0.258 0.426 Figure 5: Per-seed learning curves for locked FrankaCabinet runs. The figure visualizes the high seed variance and shows why we report both peak and tail metrics. Figure 6: Tail retention diagnostics for locked FrankaCabinet runs. The wu=50 schedule improves recomputed terminal and tail behavior relative to direct and one-shot deployment. B.3 BallBalance Boundary Details BallBalance is used as a dense-oracle boundary condition. In the locked comparison, all methods reach the peak ceiling, so consec_max alone is not informative. Table 17: Locked BallBalance peak results. All methods saturate the peak metric, so this table should not be used for ranking methods. Method n consec_max mean Std. Direct oracle (wu=0) 5 499.0 0.0 Fixed wu=50 hard switch 5 499.0 0.0 Fixed wu=100 hard switch 5 499.0 0.0 The available locked boundary evidence shows that all methods reach the peak ceiling of 499. Therefore, this task is used to establish a dense-oracle boundary condition rather than to rank methods by peak performance. Appendix C Mechanism and Ablation Analyses This appendix provides the mechanism and ablation evidence behind the main experimental interpretation. The following subsections study reward-scale controls, analyze switching operators as stability–retention trade-offs, and report selector and trigger ablations, including failure modes of reactive alternatives. C.1 Reward Scale and Phase-Composition Controls A possible alternative explanation for the main result is that direct oracle deployment fails only because of reward-scale mismatch or reward shock. To test this, we compare direct oracle deployment, scale-normalized direct deployment, scale-matched direct deployment, reward interpolation, early-only training, late-only training, and the phase-aware wu=50 hard-switch schedule. Table 18 retains only the updated scale-topup summary aligned with the revised main-text control semantics. We do not keep the older appendix tables that still reported stale scale-matched direct rows from the earlier appendix snapshot. Table 18: Updated reward-scale top-up summary on FrankaCabinet . These rows are appendix_support_topup evidence and replace the older stale appendix scale aggregates. Method n Max \uparrow Std. 95% CI low 95% CI high Status direct_oracle_running_norm 5 0.078 0.170 0.001 0.230 appendix_support_topup direct_oracle_scale_matched_to_early 8 0.235 0.281 0.074 0.432 appendix_support_topup early_dense_only 5 0.013 0.025 0.001 0.036 appendix_support_topup late_reward_only 5 0.013 0.026 0.001 0.036 appendix_support_topup The updated scale-control appendix still supports the same narrow interpretation: reward scale is a real confound, but scale correction alone does not subsume the phase-aware deployment result reported in the main text. Figure 7: Reward-shift diagnostic. Reward-scale and reward-shock effects influence training stability, but they do not fully account for the phase-aware deployment gain. Interpretation. These controls should be read as mechanism evidence rather than as additional main baselines. They rule out a simple explanation in which warm-up only acts as reward-scale normalization. Scale and shock matter, but the failure of early-only and late-only training shows that correct phase composition is also necessary. C.2 Switching Operator and Critic-Maturity Analysis We next analyze how the reward switch is executed. The main paper emphasizes that switching safety is not solved by a universal operator. Here we provide the full switching summary and the operator-by-epoch grid. Table 19 reports the locked FrankaCabinet safe-switching summary. PBRS is not a universal improvement, and critic reset improves terminal retention in some settings but induces much larger critic peaks. Table 19: Locked FrankaCabinet switching summary. Final values use recomputed terminal metrics. No operator uniformly dominates across metrics; the wu=50 hard switch gives the strongest peak and recomputed final performance with low critic instability, whereas critic reset trades retention for high critic instability. Method n Max \uparrow Final \uparrow Critic peak \downarrow |\Delta\bar{r}| Direct oracle 8 0.215 0.022 0.480 – Fixed wu=50 hard 8 0.389 0.163 0.064 0.304 Fixed wu=100 hard 8 0.233 0.089 0.019 4.189 wu=50 + V-old PBRS 8 0.313 0.036 0.077 0.214 Hard switch + critic reset 8 0.258 0.142 3.113 0.927 Operator-by-epoch grid. Table 20 reports the multiseed switch-operator grid. We evaluate hard switching, PBRS with \Phi=V_{\mathrm{old}} , and critic reset at three switch epochs. The best operator depends on both the target metric and the switch timing. This reinforces the stability–retention interpretation: no switch operator dominates across all regimes. Table 20: Switch operator \times switch epoch grid on FrankaCabinet . Values are means over eight seeds. No operator uniformly dominates across timing and metrics. Switch epoch Regime Operator n Final \uparrow Max \uparrow Critic peak \downarrow |\Delta\bar{r}| 50 immature hard switch 8 0.1200 0.1845 0.5430 0.2073 50 immature PBRS 8 0.0892 0.2263 1.2868 0.2115 50 immature critic reset 8 0.0868 0.2959 2.4217 0.5279 500 sweet spot hard switch 8 0.0150 0.0182 0.4797 0.9122 500 sweet spot PBRS 8 0.0597 0.0607 0.4877 0.9091 500 sweet spot critic reset 8 0.1421 0.2582 3.1131 0.9268 1000 later hard switch 8 0.0776 0.4190 1.1653 1.2253 1000 later PBRS 8 0.1265 0.3103 1.2125 1.2513 1000 later critic reset 8 0.1466 0.3284 2.3612 1.2370 Figure 8: Switching operator heatmaps. Operator performance depends on switch timing and metric. PBRS is not a universal improvement, and critic reset can increase retention while producing high critic instability. Interpretation. The switch-operator results should not be interpreted as evidence for a universal safe-switching mechanism. Hard switching is the practical default in the locked setting because it combines strong peak and tail performance with low critic instability. PBRS can be useful in some regimes but is not robustly dominant. Critic reset can improve terminal retention, but its large critic peaks make it a trade-off rather than a default solution. C.3 Selector and Trigger Ablations A natural question is whether fixed warm-up can be replaced by an automatic online selector or competence-triggered switch. To avoid keeping the older small- n selector pilot table alongside the revised main-text control framing, we retain only the updated 6x3090 selector-support comparison here. Table 21 summarizes the updated selector-support comparison. On FrankaCabinet , the fixed wu=50 hard switch remains stronger than the selector variants in the revised support comparison. Table 21: Updated selector support comparison on FrankaCabinet . These rows are selector_support_6x3090 evidence and replace the older small- n selector ablation table. Method n Mean CI low CI high No-switch rate Mean switches conservative_periodic_selector 8 0.0508 0.0143 0.0967 0.50 0.50 direct_oracle_or_direct_best 8 0.2150 0.0586 0.4019 1.00 0.00 fixed_wu50_hard 8 0.3889 0.2362 0.5226 1.00 0.00 moving_average_selector 8 0.1127 0.0680 0.1595 0.125 0.875 naive_last_selector 8 0.0919 0.0521 0.1374 0.125 0.875 one_shot_early 8 0.1252 0.0785 0.1752 1.00 0.00 one_shot_oracle 8 0.0175 0.0068 0.0304 1.00 0.00 Table 22: Table 22: Selector failure taxonomy. The appendix records a fuller failure taxonomy than the representative failure patterns visualized in Figure 9. Failure mode Description Never switch Selector fails to trigger a reward change despite later-phase evidence. Switch without gain Selector switches, but the chosen transition does not improve downstream performance. No reliable checkpoint Verification signal does not satisfy the reliability criterion. Over-conservative trigger Trigger delays switching until the useful phase window has passed. Wrong reward identity Selector commits to a candidate whose early local advantage does not translate to downstream utility. Figure 9: Selector and trigger diagnostics. The figure visualizes the dominant reactive-selector failure patterns: never switching, switching too late, or failing to find a reliable checkpoint. Figure 9 visualizes the dominant selector failure patterns, while Table 22 records the fuller failure taxonomy used in the appendix diagnostics. Blocked BallBalance selector rows. Two BallBalance selector rows remain documented as blocked appendix entries: the moving-average selector and a legacy globally tuned critic-loss threshold selector. They are not part of the locked main evidence and do not affect the boundary conclusion, since BallBalance peak success saturates for all main methods. Interpretation. The selector ablations support the paper’s main positioning. RHyVE is not strongest as an increasingly reactive scheduler. Its value lies in competence-aware verification and stable phase-aware deployment. In the sparse small-candidate setting, a simple fixed schedule can be more reliable than reactive selectors that depend on noisy early verification signals. Appendix D Generalization, Stress Tests, and Scope This appendix reports evidence that delimits the scope of RHyVE . The following subsections cover LLM-generated candidate families, cross-task breadth, candidate-pool scaling, and support-only or negative evidence retained for transparency. D.1 LLM-Generated Candidate Families The main experiments use small structured reward-hypothesis families so that verification and deployment effects can be cleanly analyzed. To test whether the observed phase-dependence is limited to hand-crafted reward families, we also evaluate LLM-generated candidate pools. For each task, candidate rewards are generated using multiple prompt variants, temperatures, and replicates. For analysis, we distinguish a hand-crafted reference family and two updated LLM source families (lightly curated and minimally filtered). The hand-crafted reference family is reported in the locked main comparison and in the held-out detail table, while Table 24 focuses on the updated LLM source families. The purpose of this experiment is not to prove that a fixed warm-up schedule is universally optimal for LLM-generated reward pools. Rather, it tests whether phase-dependent winner changes also arise when candidate rewards are generated by an upstream language model. Table 23: LLM candidate-generation statistics. The table summarizes generated, parsed, valid, and executable candidates. These statistics are used to characterize candidate-source diversity, not to support main performance claims. Task Generated Parsed Executable rate Eval family size Pool size Semantic diversity FrankaCabinet 105 105 0.905 3 10 10.76–34.41 BallBalance 108 108 0.787 3 10 11.02–12.10 Cartpole 108 108 0.972 3 10 6.88–13.47 Table 24 reports the updated reduced 6x3090 LLM source-generalization summary, with deployable rows separated from oracle references by an explicit status field. Table 25 records the corresponding held-out schedule-selection details for the deployable source families. Table 24: Updated reduced 6x3090 LLM source-generalization summary. Deployable rows and oracle references are separated explicitly so that the appendix does not preserve the older pilot-style completed-only summary as if it were current main evidence. Oracle rows are non-deployable references and are not guaranteed to upper-bound deployable rows when seed sets or comparison protocols differ. Candidate source Method n Max / consec_max \uparrow Flip rate Status llm_lightly_curated Direct 8 0.145 0.875 deployable llm_lightly_curated wu=50 hard 8 0.207 0.875 deployable llm_lightly_curated Held-out selected 5 0.298 – deployable llm_lightly_curated Best adaptive (oracle reference) 3 0.580 0.875 oracle llm_minimally_filtered Direct 8 0.211 0.500 deployable llm_minimally_filtered wu=50 hard 8 0.639 0.500 deployable llm_minimally_filtered Held-out selected 5 0.800 – deployable llm_minimally_filtered Best adaptive (oracle reference) 2 0.369 0.500 oracle Table 25: Held-out schedule-selection details for the reduced 6x3090 source families. Non-deployable oracle references are recorded explicitly but kept separate from deployable held-out rows in the main text. Candidate source Dev seeds Test seeds Selected schedule Evaluated method Notes structured_handcrafted_K3 42,123,456 789,999,1234,2025,3031 fixed_wu50_hard fixed_wu50_hard dev_prefers_fixed_wu50 structured_handcrafted_K3 42,123,456 789,999,1234,2025,3031 oracle_upper_bound best_adaptive_upper_bound non-deployable reference llm_lightly_curated 42,123,456 789,999,1234,2025,3031 direct_oracle_or_direct_best direct_oracle_or_direct_best dev_prefers_direct llm_lightly_curated 42,123,456 789,999,1234,2025,3031 oracle_upper_bound best_adaptive_upper_bound non-deployable reference llm_minimally_filtered 42,123,456 789,999,1234,2025,3031 fixed_wu50_hard fixed_wu50_hard dev_prefers_fixed_wu50 llm_minimally_filtered 42,123,456 789,999,1234,2025,3031 oracle_upper_bound best_adaptive_upper_bound non-deployable reference Figure 10: Left: Winner-flip rates across candidate sources. LLM-generated FrankaCabinet reward families exhibit phase-dependent winner changes, while dense or ceiling-prone tasks show little variation. Right: Method comparison across reward-candidate sources. The best deployment rule can vary across generated pools, so this experiment is used as phenomenon evidence rather than a fixed-schedule superiority claim. Takeaway. The source-generalization experiments support the hypothesis that phase-dependent reward utility is not an artifact of a hand-designed reward family. However, they also show why RHyVE should be framed as a verification protocol rather than a fixed universal schedule: different generated pools can favor different deployment rules. D.2 Cross-Task Breadth Table 26 summarizes the cross-task phenomenon scorecard. The goal of this table is not benchmark dominance. Instead, it records whether the main phenomena appear across tasks and where the method’s assumptions break down, including the optional FrankaCubeStack all-failure boundary. Table 26: Cross-task phenomenon scorecard. The strongest practical evidence is on FrankaCabinet ; other tasks are used to establish boundary conditions and phenomenon breadth. Task Competence threshold Winner flip Warm-up helps Evidence Interpretation FrankaCabinet task-dependent observed yes locked_main_existing / reduced_main_6x3090 main sparse phase-sensitive evidence BallBalance not limiting not central not meaningful (ceiling) historical_legacy dense boundary control CartPole weak fragile not reliable historical_legacy appendix breadth boundary Ant weak fragile not reliable historical_legacy appendix breadth boundary FrankaCubeStack not meaningful not meaningful not meaningful optional_extra_task_pilot_6x3090 all-failure boundary; not used for method ranking The cross-task results support a task-aware interpretation. RHyVE is most useful when the task is sparse or phase-sensitive, such that early and late rewards serve different roles. When a dense reward already provides sufficient guidance, or when reward rankings do not exhibit meaningful phase changes, warm-up may be unnecessary. D.3 Candidate-Pool Scaling Stress Test The main method is designed for small candidate sets. To test the boundary of this regime, we perform candidate-pool scaling stress tests with K\in\{3,5,10\} on FrankaCabinet and Cartpole . These results are marked as stress_tests and are not used for main performance claims. Table 27: Candidate-pool scaling stress test. Larger pools increase verification cost and can reduce top-1 reliability. This supports the small-candidate local-verification scope of RHyVE . Task K Top-1 hit rate Hit@3 Rank corr. Overhead (s) FrankaCabinet 3 1.00 1.00 0.833 1732 FrankaCabinet 5 1.00 1.00 0.950 2909 FrankaCabinet 10 0.50 1.00 0.911 5850 CartPole 3 0.50 1.00 0.500 60 CartPole 5 0.00 0.50 0.700 100 CartPole 10 0.00 0.50 0.600 201 Figure 11: Left: Candidate-pool reliability versus pool size. Top-1 reliability weakens as the candidate set grows, especially on the lightweight stress task. Right: Verification overhead grows with candidate-pool size. This motivates using RHyVE as a small-candidate local-verification protocol rather than exhaustive large-pool search. Takeaway. The pool-scaling experiment narrows the paper’s claim. RHyVE is currently best supported for small candidate sets. Extending it to larger pools will likely require coarse-to-fine search, uncertainty-aware ranking, clustering, or more efficient candidate pruning. D.4 Additional Limitations, Blocked Runs, and Negative Evidence We include negative and blocked evidence for transparency. These results are not used for main quantitative claims, but they clarify the boundary of the current method. Table 28: Support-only and negative evidence summary. These results are documented to clarify scope and prevent overclaiming. Evidence block Status Interpretation BallBalance blocked selectors documented, not rerun Two selector rows remain blocked; they are appendix-level support baselines and do not affect the locked BallBalance boundary conclusion. Trigger-based switching support only Competence-triggered switching does not replace the fixed wu=50 schedule on FrankaCabinet . Reactive selectors support only Selectors can fail by never switching, switching too late, or selecting unstable winners. Candidate-pool scaling stress test Larger pools weaken top-1 reliability and increase compute cost. Cartpole and Ant breadth boundary / negative These tasks show that phase dependence and warm-up gains are task-dependent rather than universal. Historical singleton runs legacy only Retained for traceability but excluded from main and support claims. Scope of the current evidence. The most defensible interpretation of the full experimental suite is as follows. RHyVE identifies and operationalizes a real deployment issue for LLM-generated rewards: reward hypotheses may become reliable and useful only after sufficient learner competence. On sparse phase-sensitive tasks, this can make a simple phase-aware schedule stronger than direct or one-shot deployment. However, the method is not a universal adaptive scheduler, not a large-pool reward-search algorithm, and not a guarantee that warm-up improves every task. Recommended use. In its current form, RHyVE is most appropriate when: 1. a small set of plausible reward hypotheses is available; 2. the task is sparse or phase-sensitive; 3. early reward comparisons are suspected to be unreliable; 4. the practitioner can afford sparse shared-checkpoint fork verification; 5. the deployment goal is to choose a stable phase-aware schedule rather than continuously chase a reactive selector. When the oracle-like reward is already dense, or when the candidate set is too large for reliable local ranking, RHyVE should be used as a diagnostic rather than as an automatic deployment rule. D.5 FrankaCubeStack Optional Pilot The reduced optional-scope FrankaCubeStack pilot is retained only as all-failure boundary evidence. It is not used for method ranking, and it does not change the main task-aware interpretation of the paper. Table 29: FrankaCubeStack optional pilot boundary table. All rows are optional_extra_task_pilot_6x3090 evidence and are retained only to document the all-failure boundary. Method n Max / consec_max \uparrow Final \uparrow Last-5 mean \uparrow Full AUC \uparrow Tail AUC \uparrow Collapse rate Status direct_late_task_faithful 3 0.0000 0.0000 0.0000 0.0000 0.0000 0.0 optional_extra_task_pilot_6x3090 fixed_wu100_hard 3 0.0028 0.0000 0.0000 0.000084 0.000005 0.0 optional_extra_task_pilot_6x3090 fixed_wu50_hard 3 0.0001 0.0000 0.0000 0.0000003 0.0000 0.0 optional_extra_task_pilot_6x3090 Figure 12: FrankaCubeStack optional pilot: all-failure boundary. All compared methods remain effectively flat under the reduced optional-scope budget, so this pilot is retained only as boundary evidence and not used for method ranking.