When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges

Parth Darshan Abhishek Divekar · 2026

Read When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges by Parth Darshan Abhishek Divekar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26046.txt.

Introduction

Darshan and Divekar probe what breaks when a language-model judge is tuned for several criteria at once. Textual gradient methods write critiques in words, not numbers, so classic multi-task fixes do not apply. The authors extend TextGrad to the multi-objective case and vary how much the loss, gradient and optimiser models share across criteria. When one gradient model must judge all criteria jointly, its task focus falls 59 percent, from 9.0 to 3.7 out of 10. Simply merging singly tuned prompts also degrades performance, the authors find.

Catalog record

CREATORParth Darshan Abhishek Divekar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39229
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26046

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Gradients Collide: Failure Modes of Multi-Objective Prompt Optimization for LLM Judges Parth Darshan Abhishek Divekar Abstract Customizing an LLM judge to a specific problem or domain often involves optimizing its prompt across multiple evaluation criteria simultaneously. Textual gradient methods automate this for a single judge criterion, however they produce natural-language critiques, not numerical vectors. Thus, the conflict-resolution toolkit of multi-task learning (PCGrad, MGDA) does not apply to this multi-objective textual gradient setting. We extend TextGrad to the multi-objective setting and test four decomposition modes of textual gradient optimizers by varying how much cross-objective information the loss, gradient and optimizer LLMs share. We find the gradient’s task-focus drops by 59% (9.0 to 3.7 out of 10) when the gradient LLM must provide feedback on multiple criteria jointly. Separately, we observe that naively combining single-objective optimized instructions into a single prompt degrades Spearman ρ \rho from 0.305 to 0.220 ( − 0.085 -0.085 ). These results identify two separable failure modes: optimization-time gradient dilution and inference-time instruction interference , which together constrain the design space for multi-objective judge optimization using textual feedback. 1 1 1 Code, data and prompts available at https://github.com/adivekar-utexas/when-gradients-collide 1 Introduction Figure 1: Overview of the optimization pipeline. Each step consists of four stages: (1) the task model predicts scores, (2) the loss LLM critiques predictions against ground truth, (3) the gradient LLM converts critiques into instruction edits, and (4) the optimizer LLM rewrites the prompt. The decomposition mode determines whether each stage operates per-task (Separate) or over all tasks jointly (Combined). Only the per-task instruction text is updated; the prompt skeleton and output format remain frozen throughout optimization. Modern judges, whether human or LLMs, are used to evaluate text along multiple quality dimensions at once. SummEval ( 14 ) scores summaries on four dimensions simultaneously; MT-Bench ( 43 ) spans eight categories from coding to roleplay. 18 developed Prometheus 2 specifically because existing evaluators “do not possess the ability to evaluate based on custom evaluation criteria ”. Prompt optimization methods like TextGrad ( 41 ) , OPRO ( 38 ) , and GEPA ( 1 ) can improve prompts for LLM judges automatically, but they optimize a single objective. Whether they extend to multi-objective judges, where one prompt must score several evaluation dimensions at once, remains less studied. The core challenge is structural. When multi-task deep learning models encounter conflicting task gradients, methods like PCGrad ( 39 ) and MGDA ( 29 ) resolve conflicts via projection or constrained optimization. The conflict-resolution tools of numerical multi-task learning do not apply directly, because textual gradients lack the vector-space structure these methods require. As textual gradients are natural-language strings, they do not have equivalent concepts of magnitude, inner products, or linear subspaces. The instruction “Make the coherence rubric more specific” cannot be numerically projected against “simplify the fluency criteria”. Prior work addresses related problems but not this intersection. For example, intra-task feedback quality has been studied: recently 7 identify rule dilution , when heterogeneous error modes are aggregated within a single optimization step, and 23 show that the gradient analogy does not accurately explain why Automatic Prompt Optimization (APO) methods succeed. Multi-objective prompt optimization also exists: 17 find that MGDA underperforms volume-based alternatives for discrete prompt tokens. Multi-task judge training is studied extensively ( 35 ; 37 ; 34 ) , but these methods update weights, not prompts. To our knowledge, no prior work studies multi-objective judge prompt optimization using textual gradients. We propose and test four plausible decomposition modes of textual gradient optimization for multi-objective judges: SSS , SSC , SCC , CCC , where the naming encodes whether each pipeline stage (loss, gradient, optimizer) processes tasks separately (S) or combined (C). We also introduce two process-level diagnostics: gradient specificity (how targeted each gradient is to a single task) and feedback adherence (whether the optimizer follows the gradient). We evaluate on SummEval ( 14 ) with four criteria, using two validation gate settings (MAE gating and no gating). Our results reveal a consistent structure in how and why multi-criteria prompt optimization stagnates. In 6 of 10 configurations with Qwen3 judges, optimization never exceeds the initial generic prompt (§ 5 , Table 1 ); only Single-Task with val=mae improves ( +0.031 Spearman at step 2). Our diagnostics localize this effect: gradient specificity drops by 59% ( 9.0 to 3.7 ) when the gradient LLM processes all tasks jointly § 6.1 , Table 4 ). Feedback adherence remains high (7.8 to 8.8), indicating that the optimizer incorporates the gradient’s suggestions regardless of their specificity. A separate oracle experiment shows that even independently optimal per-task instructions degrade from 0.305 to 0.220 ( -0.085 Spearman) when combined into one prompt (§ 6.3 , Table 3 ). These results identify two separable failure modes: optimization-time gradient dilution and inference-time instruction interference . Our contributions are as follows: (1) we present an empirical study of multi-criteria textual gradient optimization for LLM judges across four decomposition modes (§ 4 , § 5 , Table 1 ); (2) we propose two process-level diagnostics ( gradient specificity and feedback adherence ) that localize issues in optimization to gradient quality, not optimizer compliance (§ 6.1 , Figure 3 , Table 2 ). (3) we conduct an oracle-instruction experiment which isolates inference-time instruction interference as a failure mode distinct from optimization-time dilution (§ 6.3 , Table 3 ). 2 Related Work Textual Gradient Methods. Early methods in Prompt optimization used scalar signals: OPRO ( 38 ) rewrites prompts by conditioning on (prompt, score) histories, and APE ( 44 ) generates candidates from demonstrations and selects the highest-scoring variant. ProTeGi ( 27 ) replaced scalar signals with textual gradients : natural-language critiques that guide prompt rewrites. TextGrad ( 41 ) extended this to multi-component computation graphs, propagating critiques through LLM pipelines in analogy to backpropagation. Subsequent work has questioned the gradient analogy. GPO ( 32 ) decomposed textual optimization into two factors (update direction and update method), drawing analogies to gradient, momentum, and learning rate, but found that adding a reflection step hurts performance. 23 showed empirically that the gradient decomposition (the chain-rule structure central to the analogy) does not consistently improve prompt optimization, and that the gradient metaphor does not accurately explain why APO methods succeed. GEPA ( 1 ) evolves language programs via reflective mutation with Pareto selection. To our knowledge, all methods above optimize a single objective and none provide a mechanism to observe or control how per-task feedback interacts during optimization. Multi-Objective Prompt Optimization. Multi-objective prompt optimization is a nascent area with two distinct approaches. Population-based methods maintain a Pareto front of candidate prompts: MOPO ( 24 ) applies NSGA-II ( 9 ) with LLM-based mutation to affective text generation, and ParetoPrompt ( 42 ) decomposes objectives into scalarized subproblems. MORL-Prompt ( 17 ) adapts multi-objective reinforcement learning to discrete prompt tokens but found that MGDA ( 29 ) underperforms the simpler product-of-rewards at balancing competing objectives. Evolutionary approaches include EvoPrompt ( 15 ) for single-objective and 4 for multi-objective sentiment balancing. These methods operate at the candidate-selection level: they choose which prompts to keep from a population. To our knowledge, none of the above studies how per-task feedback interacts within a single textual gradient trajectory,

Read the complete text