Agentic Rubrics as Contextual Verifiers for SWE Agents
Mohit Raghavendra 1,∗, Anisha Gunjal 1,∗, Bing Liu 1, Yunzhong He 1 · 2026
Read Agentic Rubrics as Contextual Verifiers for SWE Agents by Mohit Raghavendra 1,∗, Anisha Gunjal 1,∗, Bing Liu 1, Yunzhong He 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04171.txt.
Introduction
Mohit Raghavendra, Anisha Gunjal, Bing Liu, and Yunzhong He test a way to grade code patches without running any tests. Checking patches matters because grades drive both learning by reward and gains from trying many candidates. Running code is grounded but costly to set up, while patch classifiers are cheaper yet less tied to the code base.
Their Agentic Rubrics method sends an expert agent into the repository to write a checklist grounded in that code. Candidate patches are then scored against the checklist alone. On SWE-Bench Verified with parallel test-time search, rubrics beat the strongest baseline compared by at least a few points on two Qwen coder models. Scores align with true test outcomes and also flag faults that tests miss, the four authors report.
Catalog record
| CREATOR | Mohit Raghavendra 1,∗, Anisha Gunjal 1,∗, Bing Liu 1, Yunzhong He 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04171
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agentic Rubrics as Contextual Verifiers for SWE Agents Mohit Raghavendra 1,∗, Anisha Gunjal 1,∗, Bing Liu 1, Yunzhong He 1 Abstract Verification is critical for improving agents: it provides the reward signal for Reinforcement Learning and enables inference-time gains through Test-Time Scaling (TTS). Despite its importance, verification in software engineering (SWE) agent settings often relies on code execution, which can be difficult to scale due to environment setup overhead. Scalable alternatives such as patch classifiers and heuristic methods exist, but they are less grounded in codebase context and harder to interpret. To this end, we explore Agentic Rubrics : an expert agent interacts with the repository to create a context-grounded rubric checklist, and candidate patches are then scored against it without requiring test execution. On SWE-Bench Verified under parallel TTS evaluation, Agentic Rubrics achieve a score of 54.2% on Qwen3-Coder-30B-A3B and 40.6% on Qwen3-32B , with at least a +3.5 percentage-point gain over the strongest baseline in our comparison set. We further analyze rubric behavior, showing that rubric scores are consistent with ground-truth tests while also flagging issues that tests do not capture. Our ablations show that agentic context gathering is essential for producing codebase-specific, unambiguous criteria. Together, these results suggest that Agentic Rubrics provide an efficient, scalable, and granular verification signal for SWE agents. 1 1 footnotetext: Equal contribution. [email protected] https://scale.com/research/agenticrubrics 1 Introduction Figure 1 : Agentic rubric pipeline. In the rubric-generation phase (left), a rubric agent inspects the codebase and PR description using repository tools, then produces a rubric.yaml organized along four rubric axes (File Change, Spec Alignment, Integrity, Runtime). In the verification phase (right), a SWE agent proposes a patch, which is graded against the rubric to yield an execution-free verifier score. Large Language Models (LLMs) have rapidly advanced on coding tasks, enabling increasingly capable software engineering (SWE) agents for realistic code editing and bug fixing [ 28 , 9 , 21 ] . A central bottleneck in training and evaluating such agents is verification : determining whether a candidate patch is correct, complete, safe, and aligned with the intended behavior. Verifier’s Law links the ease of training AI systems on a task to the efficiency and reliability of verifying candidate solutions [ 22 ] . In SWE agent, strong verification plays a dual role. It provides supervision for post-training with verifiable rewards [ 13 ] , and it improves inference through test-time scaling by sampling multiple candidates and selecting the best one using a verifier [ 3 ] . Current approaches use a range of verifiers, including unit tests (human or LLM-generated), learned patch classifiers, similarity metrics, and LLM judges [ 12 , 23 , 10 , 24 ] . Verification via code execution is environment-aware, but can be costly to scale due to per-instance setup (e.g., sandbox initialization), and may yield sparse or brittle signals, including limited distinguishability and test toxicity [ 6 , 10 ] . In contrast, execution-free signals are operationally lightweight, but can be less reliable [ 4 ] , less interpretable, and prone to shallow cues. As SWE agents expand to more open-ended, goal-driven tasks and long-tail repositories, verifiers must become both scalable and codebase-specific. To close this gap, we explore Agentic Rubrics . In our setup, illustrated in Figure 1 , an expert rubric agent first interacts with a sandboxed repository to synthesize context-grounded rubric criteria; after rubric generation, candidate patches are scored without executing code, enabling scalable verification. We build on rubric-based verification [ 16 , 25 ] , which decomposes correctness into interpretable criteria that capture partial progress and surface failure modes. For SWE, rubrics written from the problem statement alone are often under-specified because they lack repository-specific context. Our rubric generation is therefore agentic : the verifier actively explores the repository to ground criteria in relevant code paths, interfaces, and project conventions, yielding rubric items that are more specific and consistently gradable. We evaluate Agentic Rubrics via best-of- K selection under parallel test-time scaling on SWE-Bench Verified, ablate different design decisions and provide detailed analyses of rubric alignment and utility. Our contributions are: (1) We study Agentic Rubrics , a repository-grounded rubric generation paradigm with execution-free scoring for patch selection and post-training. (2) We show that Agentic Rubrics consistently outperform strong test-based and execution-free verifier baselines under parallel test-time scaling on SWE-Bench Verified. (3) We analyze why Agentic Rubrics work, demonstrating alignment with ground-truth tests and showing that rubrics surface diagnostic concerns (e.g., unnecessary edits or missing edge-case handling) even when tests pass. (4) We demonstrate that agentic rubric generation can be distilled into smaller open-weight models, enabling scalable deployment. 2 Preliminaries 2.1 Verification for SWE Agents We consider a verifier as a procedure that assigns a score to a candidate patch for a given issue, with the goal of selecting or training toward higher-quality solutions. Prior work in SWE Agent settings commonly uses two broad classes of verification signals. Execution-based methods verify patches by executing code, most often by running unit tests (human-authored ground-truth or LLM-generated) [ 6 ] . Execution-free methods assess patch quality without running the repository, by reranking candidates using learned patch classifiers/verifiers, similarity metrics, or LLM judges [ 23 ] . These approaches occupy different points in the trade-off space between repository grounding, operational cost, and reliability [ 10 ] . Execution-based verification is environment-aware but can require per-instance setup (e.g., sandbox initialization) and may yield sparse or brittle signals (e.g., limited distinguishability or test toxicity). Execution-free verification is operationally lightweight, but can be less reliable [ 4 ] , less interpretable, and sometimes sensitive to surface-level cues (e.g., stylistic patterns, non-semantic artifacts) rather than functional correctness. 2.2 Rubric-based Verification A rubric verifies a candidate patch by decomposing correctness into a small set of explicit criteria Arora et al. 2025 . Concretely, a rubric consists of criteria texts (optionally grouped by axes) with per-criterion weights, and a scoring rule that aggregates criterion-level judgments into a single verifier score. Given a problem and a candidate patch, a judge assigns each criterion a score (e.g., binary or graded) and aggregates them to obtain an overall patch score used for selection or learning. For SWE tasks, a key practical consideration is grounding . Criteria written solely from the problem statement can omit repository-specific interfaces, constraints, and conventions, which makes judgments less precise and less consistent across patches. This motivates verifiers whose criteria are grounded in the right task-relevant repository context, while still allowing lightweight scoring once criteria are generated. 3 Experimental Design 3.1 Agentic Rubrics Rubric Generation. We implement a rubric-generation agent on top of the SWE Agent scaffold, which provides tools for repository navigation, file inspection/editing, and shell command execution [ 28 , 21 ] . We modify the scaffold’s SYSTEM PROMPT , instructing the agent to explore the repository, gather task-relevant context, and produce a patch that adds a structured rubric file, rubrics.yaml (prompt in Appendix A.10 ). This workflow mirrors how developers validate fixes when comprehensive tests are unavailable: inspecting surrounding code and contracts, tracing call sites, and reasoning about edge cases. Each rubric item is a tuple (t_{i},w_{i}) consisting of a short natural-language criterion t_{i} and an importance weight w_{i}\in\{1,2,3\} (nice-to-have / important / must-have), and is assigned to one of