Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges
Maxim Khomiakov & Jes Frellsen · 2026
Read Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges by Maxim Khomiakov & Jes Frellsen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17172.txt.
Introduction
Khomiakov and Frellsen propose a calibration protocol for language model judges based on controlled noise. Model judges are stochastic and often overconfident, yet ground truth is scarce where they are used. Their idea: as input noise grows, task performance should degrade in a statistically clear trend. They test this slope with repeated trials, using signal noise for tables and word perturbations for text. Across benchmarks, text judges degraded predictably. Most tabular datasets showed no significant decline, even under heavy noise.
Catalog record
| CREATOR | Maxim Khomiakov & Jes Frellsen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47084 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17172
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges Maxim Khomiakov & Jes Frellsen Abstract Large language models (LLMs) are increasingly used as automated judges and synthetic labelers, especially in low-label settings. Yet these systems are stochastic and often overconfident, which makes deployment decisions difficult when external ground truth is limited. We propose a practical calibration protocol based on controlled input interventions: if noise severity increases, task performance should exhibit a statistically significant deterioration trend. We operationalize this with a slope-based hypothesis test over repeated trials, using signal-to-noise-ratio (SNR) perturbations for tabular data and lexical perturbations for text data. Across UCI tabular benchmarks and four text classification datasets, we find clear modality-dependent behavior. Our results reveal a modality gap: while text-based judges degrade predictably, the majority of tabular datasets show a lack of statistically significant performance deterioration even under significant signal-to-noise reduction. Interestingly we find that model performance is lower on datasets that are insensitive to noise interventions. We present a reproducible methodology and reporting protocol for robust LLM-judge calibration under distribution shift. 1 Introduction High-quality labels remain costly and difficult to produce: annotation policy design, expert recruitment, and quality control are all substantial bottlenecks ( Gebru et al. 2021 ; Snow et al. 2008 ) . In parallel, LLM-based pipelines now support two high-value workflows: (1) synthetic labeling, and (2) LLM-as-a-Judge decision systems for automated triage and ranking ( Liu et al. 2023 ; Zheng et al. 2023 ) . The practical failure mode is not only low average quality, but unreliable behavior under shift: small input changes can create non-trivial output volatility, especially when large trusted holdout labels are unavailable ( Lin et al. 2022 ; Huang et al. 2025 ) . Core question. Can we reject an LLM judge for a target task before deployment by stress-testing its response to controlled interventions? We frame this as a causal question in the interventionist sense of Pearl 2009 : apply a controlled perturbation to the input and test whether performance degrades as predicted. We formalize a monotone-deterioration hypothesis over a calibrated noise schedule. Let \mathcal{P} denote a task-specific performance score (e.g., accuracy for classification, R^{2} for regression) measured on clean inputs, and \mathcal{P}_{k} the same score measured under perturbation magnitude k\in K , where K is ordered from weak to strong noise. Define \Delta_{k}=\mathcal{P}_{k}-\mathcal{P} . Our conceptual intervention hypothesis is \displaystyle H_{0} \displaystyle:\exists\,k_{1}<k_{2}\text{ such that }\mathbb{E}[\mathcal{P}_{k_{2}}]>\mathbb{E}[\mathcal{P}_{k_{1}}], (1) \displaystyle H_{1} \displaystyle:\mathbb{E}[\mathcal{P}_{k_{1}}]\geq\mathbb{E}[\mathcal{P}_{k_{2}}]\quad\forall k_{1}<k_{2}, (2) with at least one strict inequality under H_{1} . Here, H_{0} (the null hypothesis) states that the expected performance is not monotone under increasing perturbation i.e., there exists at least one pair of noise magnitudes where performance improves despite stronger noise. H_{1} (the alternative) encodes the expected deterioration property: as noise increases, the expected performance does not increase, and it decreases for at least one step. The expectation \mathbb{E}[\mathcal{P}_{k}] is taken over the randomness in the procedure at severity level k , including (i) the sampled perturbations/noise realizations and (ii) any stochasticity in the LLM inference; In practice, we operationalize this ordered alternative using a one-sided linear trend test (negative slope) as a pragmatic surrogate; thus sensitive indicates significant deterioration with increasing noise, which is consistent with but not equivalent to strict monotonicity. Intuitively, if increasing perturbation does not induce expected deterioration, the model is either exploiting artifacts or is insufficiently sensitive to task signal, both of which reduce trust in judge-style deployment. Conversely, when the expected deterioration holds with statistical significance, we obtain a useful calibration signal about where the model can be trusted. This paper presents a reproducible methodology around this idea: a pipeline for perturbation design, repeated inference, and hypothesis testing across tabular and text settings, presenting findings which indicate that employing such a methodology prior to deployment could provide an indicator of trustworthy LLM results. Overview of Contributions 1. We propose a practical noise-response calibration protocol for deciding whether an LLM judge is reliable for a specific task, based on intervention schedules and slope-based hypothesis testing. 2. We report large-scale empirical findings spanning UCI tabular tasks and four widely used text classification datasets, performing experiments applying both correlated and uncorrelated Gaussian noise for tabular data and lexical noise for text. 3. We demonstrate that for the majority of included UCI tabular tasks, performance is largely insensitive to the tested noise schedules (i.e., it does not exhibit statistically significant deterioration as noise increases). We discuss how this insensitivity can serve as a practical warning signal for LLM-judge deployment, complementing average-performance reporting. 2 Related works Hallucination, alignment, and judge reliability. Recent work has studied hallucination mechanisms and mitigation strategies in depth, including retrieval augmentation, verification, and calibration-oriented prompting ( Huang et al. 2025 ) . In parallel, alignment methods such as RLHF and constitutional training have improved instruction-following behavior, but do not eliminate robustness failures under distribution shift ( Ouyang et al. 2022 ; Bai et al. 2022 ) . For LLM-as-a-Judge pipelines, reporting and evaluation protocol quality is itself a major source of variance. A central recommendation in recent methodology work is that judge results should be reported with stronger controls for prompt sensitivity, variance, and statistical uncertainty ( Lee et al. 2026 ) . Perturbation sensitivity in reasoning and NLP. Our work is closely related to perturbation-based robustness analysis. In mathematical reasoning, Mirzadeh et al. 2025 show that seemingly benign symbolic variations can induce substantial performance changes. More broadly, LLM reasoning has been shown to degrade under distractor-like or superficially irrelevant perturbations ( Shi et al. 2023 ) . For text corruption, lexical perturbation studies demonstrate that character-level noise can strongly hurt model behavior even when semantics are mostly preserved ( Belinkov & Bisk 2018 ; Pruthi et al. 2019 ) . Our text intervention family is inspired by this line of work but is used for trust calibration rather than adversarial training. Causal framing and benchmark context. We adopt an intervention-based causal framing: the perturbation operator is treated as an intervention on inputs, and the downstream performance response is the measured effect ( Pearl 2009 ; Peters et al. 2017 ) . This perspective makes explicit what is being manipulated and what constitutes expected behavior under increasing uncertainty. For tabular benchmarks, we use datasets from the UCI repository ( Asuncion & Newman 2007 ) , one of the largest tabular datasets designed for ML prediction tasks, and include canonical datasets such as Iris ( Fisher 1936 ) and Wine Quality ( Cortez et al. 2009 ) . For text experiments we use IMDB ( Maas et al. 2011 ) , Yelp Review Full ( Zhang et al. 2015 ) , SST-2 ( Socher et al. 2013 ) , and Financial PhraseBank ( Malo et al. 2014 ) , datasets that all revolve around sentiment classification tasks. 3 Method Algorithm 1 Noise-response calibration protocol 1: Dataset D , noise schedule N=\{n_{k}\}_{k=1}^{K} , repetitions R , significance level \alpha 2: Initialize LLM Judge for the prediction task and compute baseline performance \mathcal{P} 3: for each n_{k}\in N do 4: for each repetition r\in\{1,\dots,R\} do 5: Generate noisy inputs, query the LLM, record performance \mathcal{P}_{k,r} 6: end for 7: end for 8: Fit linear model \mathcal{P}_{k,r}=\beta_{0}+\beta_{1}n_{k}+\epsilon 9: Perform one-sided slope test H_{0}:\beta_{1}\geq 0 vs. H_{1}:\beta_{1}<0 10: