Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation
Tzu-Heng Huang, Harit Vishwakarma, Frederic Sala · 2025
Read Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation by Tzu-Heng Huang, Harit Vishwakarma, Frederic Sala (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10403.txt.
Introduction
Tzu-Heng Huang, Harit Vishwakarma, and Frederic Sala propose firing the model judge in a 2025 paper. Using one language model to grade another is costly, opaque, and biased, they argue. Their alternative, PAJAMA, has the model write an ordinary judging program once; the program then grades locally, cheaply, and with logic anyone can read and tweak.
Program judges agreed with themselves more and showed fewer biased verdicts than a mid-size model judge, by clear margins. When their verdicts were distilled into a smaller model, the student beat the model judge on a hard reward benchmark and two judge data sets. The cost per judgment fell by roughly a thousandfold. The authors pitch programs as evaluation's future.
Catalog record
| CREATOR | Tzu-Heng Huang, Harit Vishwakarma, Frederic Sala |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-10403
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation Tzu-Heng Huang, Harit Vishwakarma, Frederic Sala Abstract Large language models (LLMs) are widely used to evaluate the quality of LLM generations and responses, but this leads to significant challenges: high API costs, uncertain reliability, inflexible pipelines, and inherent biases. To address these, we introduce PAJAMA ( P rogram- A s-a- J udge for A utomated M odel A ssessment), a new alternative that uses LLMs to synthesize executable judging programs instead of directly scoring responses. These synthesized programs can be stored and run locally, costing orders of magnitude less while providing interpretable, and auditable judging logic that can be easily adapted. Program-based judges mitigate biases, improving judgment consistency by 15.83% and reducing biased responses by 23.7% on average compared to a Qwen2.5-14B-based LLM-as-a-judge. When program judgments are distilled into a model, PAJAMA outperforms LLM-as-a-judge on the challenging CHAT-HARD subset of RewardBench, outperforming metrics by 2.19% on Prometheus and 8.67% on the JudgeLM dataset, all at three orders of magnitude lower cost. 1 Introduction General-purpose large language models (LLMs) are now the standard for automated evaluation of generative model responses. In the typical LLM-as-a-judge setup, a judge LLM evaluates a user query with two LLM-generated answers, then selects the better one [ 1 ] . These judgments can be viewed as preference labels that enable ranking LLMs by performance [ 2 , 3 ] , efficiently verifying LLM agent behaviors [ 4 ] , curating datasets [ 5 ] , or distilling into specialized reward models [ 6 ] . Despite their promise, current LLM-as-a-judge approaches have several drawbacks: • High cost : Querying state-of-the-art models, e.g., Open AI o3, or Claude 3 Opus [ 7 ] , can run into thousands of dollars for large evaluation sets. For example, researchers spent about $4,000 to generate 100K high-quality judgments using GPT-4 [ 8 ] . • Uncertain reliability : Although LLMs can explain their verdicts, their outputs may exhibit inconsistent reliability in adhering to rubrics [ 9 ] . • Inflexible pipelines : Any minor change to the evaluation rubric requires re - running the entire pipeline, incurring additional expenses. • Inherent biases : LLMs trained on web datasets encode social and stylistic biases (e.g., gender preferences, or favoring emojis), which can skew decisions [ 10 , 11 , 12 ] . We address these by relying on a simple but powerful notion. Instead of prompting an LLM for preferred answers, we ask it to generate the judging logic it would use and encode it into an executable program. In other words, the model is asked to synthesize a compact function, e.g., a few lines of Python, that encodes its evaluation criteria. These are executed to obtain a quality score for each response. This program synthesis -style strategy offers several advantages. First , API costs now scale with the number of generated programs, not the dataset size, significantly reducing expenses. Once generated, judging programs can be stored and executed locally for any new query at no additional cost. Second , synthesized programs are interpretable, allowing practitioners to audit each line, refine rubrics, or insert heuristic rules to minimize bias. Third , by exposing the full decision logic, this approach turns a black-box judging model into a transparent, inspectable evaluation process. While promising, translating LLM judgments into executable code raises new challenges. It is common for synthesized programs to be repetitive, reusing similar criteria with minor variations. To encourage diversity, we introduce six distinct criteria— each expressible as code —to guide LLMs in generating useful programs. Additionally, individual program outputs can be noisy, and different programs may capture complementary signals. We address this by combining our approach with the weak supervision framework [ 13 , 14 , 15 , 16 ] . By modeling program outputs, we aggregate program judges into a collective signal that can outperform naïve ensembling approaches like majority voting. Results and Contributions. To tackle the above, we propose PAJAMA ( P rogram- A s-a- J udge A utomated M odel A ssessment), a lower-cost and lower-bias evaluation system that relies on synthesized judging programs. Each program can serve directly as a judge or have its judgments distilled into a reward model for improved generalization. For example, when distilled into a reward model, PAJAMA outperforms LLM - as - a-judge - distilled reward models on the challenging CHAT-HARD subset of RewardBench [ 17 ] , achieving +2.19% on Prometheus [ 18 ] and +8.67% on JudgeLM dataset [ 8 ] , while decreasing API costs by approximately 3500 \times and 2500 \times , respectively. We also test PAJAMA on high-bias responses, observing that it produces consistent correct responses and improved error rate over LLM-as-a-judge. Across four common bias types, it enhances consistency by 15.83% and reduces the biased - answer win rate by 23.7% relative to Qwen2.5 - 14B. 2 Evaluation System: PAJAMA We start with an overview of PAJAMA’s general workflow, followed by the problem setup (Sec. 2.1 ), then describe program synthesis for judging code (Sec. 2.2 ) and discuss how we combine multiple judgments to produce an aggregated evaluation decision (Sec. 2.3 ). Figure 1 : PAJAMA’s General Workflow. General Workflow. The workflow of PAJAMA is illustrated in Fig. 1 . First, users collect queries and LLM generation pairs, then create prompts to guide LLMs in generating programs with judging logic. Prompts can vary in their design and specified evaluation criteria, enabling the creation of multiple judging programs. Using weak supervision, aggregated preference labels are inferred from program outputs. Finally (and optionally), these judgments can be used to train a distilled reward model for local use. 2.1 Problem Setup We consider user queries and responses drawn from the space of free-form text, \Sigma^{\star} , e.g., all possible natural language strings. We denote the space of queries as \mathcal{X} and the space of responses as \mathcal{Y} . For any query x\in\mathcal{X} , we evaluate two responses, y_{1},y_{2}\in\mathcal{Y} , generated by the same or different LLMs. Other approaches directly score each candidate generation, i.e., assign a score to each y . Ground-truth assessments for reward or evaluation models are accurate but costly and slow due to human annotations. Using LLM-as-a-judge is faster but incurs high inference costs, and may embed LLM biases into ground truth and downstream models. We address these issues with synthesized judging programs as an alternative. 2.2 Program Synthesis We propose a prompting template to query LLMs for synthesizing programmatic judges. Using GPT-4o [ 19 ] , we generate these judges as follows: Prompt used to synthesize judging programs. You are a judge tasked with evaluating LLM-generated responses to a given question. Write your evaluation logic as Python code, returning a numeric score for a response where higher values indicate better quality. Use third-party libraries (e.g., embedding models, nlp metrics) as needed. def judging_function(query, response) : We chose this approach for its simplicity, but we note that more sophisticated approaches to program synthesis can be seamlessly swapped in . Next, we propose six distinct criteria that can be incorporated into the program synthesis prompt and thus be encoded into executable Python code. We describe each criterion below. We likewise note that these can be easily