The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
Jikai Jin, Vasilis Syrgkanis · 2026
Read The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice by Jikai Jin, Vasilis Syrgkanis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01311.txt.
Introduction
Jikai Jin and Vasilis Syrgkanis tackle a bias that creeps into scoring language models from real usage logs. Users who pick a given model may differ from those who pick another, and those same traits can sway how outputs get rated, so raw log scores mix unlike groups.
The study mixes three sources: a large biased log, a small randomized trial where the model is assigned by chance, and a simulator that replays models on saved prompts. The main theorem shows the trial plus the simulator are enough to recover true causal scores for each model. The log then only sharpens the estimates. Six estimator families are tested on shaped data and on real saved tasks in summarizing and coding, and none wins in all settings.
Catalog record
| CREATOR | Jikai Jin, Vasilis Syrgkanis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01311
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice Jikai Jin, Vasilis Syrgkanis Abstract Offline evaluation of language models from usage logs is biased when model choice is confounded: the same user-side factors that influence which model is used can also influence how its output is judged, so raw comparisons of logged scores mix self-selected populations rather than estimating a common quantity of interest. A small randomized experiment can break this bias by overriding model choice, but in practice such experiments are scarce and costly. We study a three-source design that combines a large confounded observational log (OBS) for scale, a small randomized experiment (EXP) for unconfounded scoring, and an offline simulator (SIM) that replays candidate models on cached contexts. Our main result is an identification theorem showing that the randomized experiment and the simulator are together enough to recover causal model values; the observational log enters only afterward, to reduce estimation error rather than to make the causal comparison valid. Six estimator families are evaluated in a controlled semi-synthetic validation and in two real-task cached benchmarks for summarization and coding. No family dominates every regime; relative performance depends on the amount of unbiased EXP supervision and on how closely the target reward aligns with OBS-derived structure. 1 Introduction Offline evaluation of language models increasingly relies on deployment logs, preference data, and judge annotations collected in the wild ( Zheng et al. 2024 ; Zhao et al. 2024 ; Zheng et al. 2023 ; Chiang et al. 2024 ) . Once model choice is user-driven, those logs are confounded: the same unobserved factors that influence which model is used can also influence how its output is judged ( Kallus et al. 2018 ; Rosenman et al. 2023 ; Cheng & Cai 2021 ) . Raw comparisons of logged outcomes therefore mix different self-selected subpopulations rather than a common target population. In summarization, a reader who prefers concise prose may select a model known for that style and rate it more favorably; in coding assistance, repository difficulty or developer familiarity may affect both which assistant is invoked and whether the resulting patch succeeds. A randomized experiment breaks this link by overriding model choice. In practice, such experiments are costly and disruptive, so they remain small. Prior work studies how to combine a large confounded observational sample with a small randomized one ( Kallus et al. 2018 ; Rosenman et al. 2023 ; Cheng & Cai 2021 ; Lin et al. 2025 ; Yang et al. 2025 ; Colnet et al. 2024 ) . The problem has two parts: outcome generation and outcome scoring. Outcome generation asks which output a model would have produced on a given context; in many deployments an offline simulator (SIM) handles this by replaying models on cached contexts. Outcome scoring asks how the realized output should be evaluated without inheriting the bias in logged model choice. Separating the two yields a three-source design: a large observational log (OBS) with self-selected model choices, a small randomized experiment (EXP) with unconfounded outcome labels, and SIM outputs under alternative model choices. The goal is to estimate how each generative model would perform when logged model choice is confounded, randomized outcome labels are limited, and offline replay can regenerate candidate outputs. Contributions. The paper makes three contributions. (C1) Identification (Theorem 1 ). Under the structural assumptions stated in Section 2 , the simulator and the randomized experiment together identify causal model values. The observational log is not required for identification. (C2) Estimator role of OBS (post-identification only). We separate identification from estimation and compare six post-identification estimator families that differ in how they use OBS. After identification, OBS improves estimation but never serves as an identifying source. A theory-to-evidence map explains why different families help in different data regimes. (C3) Empirical evaluation. A controlled semi-synthetic validation and two real-task cached benchmarks (summarization, coding) compare the estimator families on held-out recommendation regret. The empirical pattern is not universal: performance depends mainly on how much unbiased experimental supervision is available, and on how well observational structure matches the reward that ultimately matters. The cached benchmarks evaluate fixed cached targets rather than the full latent population quantity one would ideally estimate. Section 2 defines the causal graph, data sources, target estimands, and assumptions. Section 3 states and proves the SIM-plus-EXP identification result. Section 4 introduces the post-identification estimator families and the theory-to-evidence map. Section 5 describes the controlled validation and cached benchmarks; Sections 5.3.1 and 5.4 report the cached summarization and cached coding probes. Sections 6 , 7 , and 8 cover related work, scope and limitations, and conclusions. 2 Problem setting OBS provides scale and auxiliary supervision from self-selected usage, EXP provides unbiased outcome labels, and SIM provides counterfactual outputs. Theorem 1 shows that EXP and SIM identify the target, while OBS only improves statistical efficiency afterward. Figures 1 and 2 illustrate the causal structure and the division of labor. 2.1 Variables and causal structure X U A M Y Observed context Latent user state (unobserved) Chosen model Mediator (output) Outcome (user utility) Confounding paths Causal paths Figure 1: Causal graph with confounded model choice. The latent state U affects both the chosen model A and the outcome Y , so OBS comparisons of logged outcomes are biased. Assumption A1 (randomization in EXP) breaks the U\to A link inside EXP, and Assumption A4 (no latent mediator confounding) blocks any direct U\to M path; together they enable the SIM-plus-EXP identification of Theorem 1 . We study a five-variable graph (X,U,A,M,Y) shown in Figure 1 , where X is the observed context, U\in\mathbb{R}^{d_{u}} is an unobserved user state, A\in\mathcal{A} indexes the chosen generative model, M is the generated output, and Y\in[0,1] is the scalar outcome. Confounding arises because the latent state U affects both model choice and outcome. Beyond ordinary confounding, the graph imposes two structural restrictions: conditional on (X,A) , the latent user state U does not directly affect the mediator M , and A affects the outcome Y only through M . These assumptions are natural in stateless generative systems, where the model’s behavior is determined by its input and internal randomness rather than by who is asking. 2.2 Accessible data sources Three data sources are available, each contributing something the others cannot provide alone. Observational log (OBS). OBS is a large observational dataset D_{\mathrm{OBS}}=\{(X_{i},A_{i},M_{i},Y_{i},Z_{i})\}_{i=1}^{n_{\mathrm{OBS}}}, where A_{i} may depend on the corresponding latent state U_{i} and Z_{i}\in[0,1]^{K} are K auxiliary labels that may or may not be available. The auxiliary labels Z_{i} provide extra supervision for learning a compact proxy representation of the context–output pair; user feedback ratings and pairwise preferences are natural sources and can often be collected at scale ( Christiano et al. 2017 ; Ouyang et al. 2022 ; Chiang et al. 2024 ) . Randomized experiment (EXP). EXP is a much smaller dataset D_{\mathrm{EXP}}=\{(X_{j},A_{j},M_{j},Y_{j})\}_{j=1}^{n_{\mathrm{EXP}}}, in which model choice is randomized and therefore unconfounded. Since A_{j} is drawn from the experimental action set \mathcal{A}_{\mathrm{EXP}} according to a fixed distribution, A\perp U and A\perp U\mid X , so EXP outcomes are causally interpretable. Randomizing model choice disrupts user experience, which is why these experiments remain small in practice. Offline simulator (SIM). The simulator reruns any generative