Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight
Can Jin, Jiakang Li, Rui Wu, Eddy Zhang, Dimitris N. Metaxas · 2026
Read Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight by Can Jin, Jiakang Li, Rui Wu, Eddy Zhang, Dimitris N. Metaxas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00424.txt.
Introduction
Jin and co-authors use weak critics to coach strong models past what weak judges can grade. As models grow strong, weak guides fail at firm tags or picks. Yet a weak guide may still nudge a strong reply the right way. That nudge needs no full solve of the task at hand.
They call this weak-critic strong oversight in their frame. Weak notes lift frozen strong models at use time, when notes are sound. Their plan, OPCD, sifts top notes from the weak critic pool. Those notes are then taught back into the strong model as self-guide marks. Marks adapt as the strong model's own line shifts. Tests on thought and fit sets show gains that grow with rounds.
Catalog record
| CREATOR | Can Jin, Jiakang Li, Rui Wu, Eddy Zhang, Dimitris N. Metaxas |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00424
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight Can Jin, Jiakang Li, Rui Wu, Eddy Zhang, Dimitris N. Metaxas Abstract As large language models become stronger, weak supervisors may fail to provide reliable labels, preferences, or final judgments for complex outputs, limiting both weak-to-strong generalization and scalable oversight. We study a more tractable form of weak supervision: using a weak model as a critic rather than as a labeler or judge. Instead of solving the task or selecting the correct answer, the weak critic only needs to provide a non-misleading revision direction that helps the strong model better use its own knowledge. We call this setting weak-critic strong oversight . We first show that weak critiques can improve frozen strong models at inference time, and that critique quality is key to this improvement. We then propose progressive on-policy critique distillation ( Opcd ), which filters high-quality critiques and distills critic-guided behavior into the strong model through adaptive self-teacher signals. Experiments on reasoning and alignment benchmarks show that our method improves strong models over training epochs, suggesting an effective path for scalable oversight with weak supervision. † † affiliationnotice: Equal contribution, † Equal Advising. 1 Introduction Modern large language models (LLMs) are commonly aligned with human supervision, such as task demonstrations, preference labels, reward models, and reinforcement learning from human feedback ( 13 ; 31 ; 8 ) . These methods work well when the supervisor can reliably judge the model output. However, as models become stronger, they may produce answers, plans, proofs, or code that are difficult for humans or weaker models to fully verify. This creates a central challenge for alignment: how can a weak supervisor guide a stronger model when the final task is too hard for the supervisor to solve or judge? Two related research directions study this challenge. Weak-to-strong generalization asks whether supervision from a weak model can elicit useful behavior from a stronger pretrained model ( 11 ) . Scalable oversight asks how a weak human or model can provide reliable supervision for a stronger system, often through assistance, interaction, or debate ( 2 ; 17 ; 10 ; 22 ) . Although these directions differ in their goals, many existing methods place a similar burden on the weak supervisor: the weak model must provide direct labels, soft logits, preference signals, or final judgments over complete answers. This can be too demanding when the task is beyond the weak supervisor’s full ability. In such cases, the supervision signal can be noisy, incomplete, or systematically wrong, which limits both weak-to-strong generalization and scalable oversight. In this paper, we study a different form of weak supervision: using the weak model as a critic rather than as a labeler or judge. A weak critic does not need to solve the task, provide the correct answer, identify every error, or give a detailed revision plan. It can be useful by giving a general but correct revision direction, such as suggesting that the reasoning is incomplete, a condition is missing, a boundary case should be checked, or the response should be safer. This form of feedback is often easier than labeling or judging complete answers, and is close to common human–AI interaction: users may not know the full solution, but they can still give feedback that helps a stronger model revise. As long as the critique is not misleading, it can help the strong model better use its own knowledge without requiring the weak supervisor to provide full supervision. We call this setting weak-critic strong oversight . We first test this idea at inference time. Given a question, the strong model produces an initial answer, the weak model critiques it, and the strong model then revises its answer conditioned on the question, the initial answer, and the critique. This directly evaluates whether weak critiques can improve a frozen strong model. Our results show that they can, even when the critique only gives a general revision direction rather than a detailed error analysis. This supports the core hypothesis of weak-critic strong oversight : weak supervisors may not need to solve or judge the full task to provide useful oversight. We also find that critique quality is central. Helpful critiques improve performance, while misleading critiques can hurt performance, even compared with using no critique. This motivates filtering useful critiques before using them for training. To internalize the inference-time improvement, we propose a progressive o n- p olicy c ritique d istillation method ( Opcd ). In each epoch, the current strong model generates on-policy answers, and the weak model critiques these answers. We then use an outcome- and rubric-based quality metric to keep only useful critiques. For each kept example, the critic-conditioned strong model serves as a self-teacher, using the critique as guidance to provide dense token-level signals. The student is the same strong model without access to the critique, trained by on-policy distillation. After each update, the strong model produces new answers with new error patterns, and the weak critic provides fresh critiques for the updated model. This process distills the useful critic-guided behavior observed at inference time while keeping the supervision adaptive to the current strong model. Our experiments show that weak-critic strong oversight improves strong-model performance in both inference-time and training-time settings. Compared with standard weak-to-strong methods that directly distill weak-model responses or logits, our method does not force the weak model to provide full supervision. Compared with ground-truth supervised finetuning, our method studies a more realistic oversight setting where reliable labels may be unavailable or too hard for weak supervisors to provide. Across reasoning and alignment benchmarks, progressive on-policy critique distillation improves the strong model over training epochs, showing that critique-based supervision is an effective path for scalable oversight and weak-to-strong generalization. Our main contributions are: ★ Critique-based weak supervision. We identify critiquing as a more tractable form of weak supervision than labeling or judging, and propose weak-critic strong oversight for scalable oversight and weak-to-strong generalization. ★ Inference-time validation. We show that weak critiques can improve strong-model performance at inference time, even when they provide only general revision directions, and find that critique quality is key to reliable improvement. ★ Progressive critique distillation. We introduce Opcd , a progressive on-policy critique distillation strategy that filters high-quality critiques and uses them as adaptive weak feedback to train the strong model. ★ Strong results. Across multiple benchmarks, Opcd progressively improves strong-model performance, showing that critique-based supervision can effectively use weak oversight for stronger models. 2 Related Works Scalable oversight and weak-to-strong generalization. Scalable oversight aims to develop methods to supervise AI systems on tasks that are difficult for humans ( 2 ; 10 ) . The primary focus has been on designing human-AI collaboration protocols that help humans evaluate AI outputs more accurately, for example through debate and consultancy ( 17 ; 21 ) , and on lowering the cognitive burden of evaluation through critique-assisted review and prover-verifier games ( 36 ; 30 ; 23 ) . In contrast, weak-to-strong generalization (W2S) ( 11 ) explores a complementary direction that designs learning algorithms which let a strong pretrained model generalize correctly from weak supervision as if it had been trained on higher-quality labels. A growing body of