From Stochasticity to Signal: A Bayesian Latent State Model for Reliable Measurement with LLMs

Yichi Zhang, Ignacio Martinez · 2025

Read From Stochasticity to Signal: A Bayesian Latent State Model for Reliable Measurement with LLMs by Yichi Zhang, Ignacio Martinez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23874.txt.

Introduction

Yichi Zhang and a co-author measure a hidden flaw in how language models are used as text classifiers. Ask the same model the same question ten times and its answers are not independent. Teams often feed repeated judgments to latent class models, which assume each rating is a fresh draw given the true label. The authors test that assumption on 52,500 judgments from seven open models.

The ratings turned out correlated, so ten judgments carried the information of only about one to two independent ones. Under that correlation, the standard binomial latent class model failed badly: its ninety-five percent intervals never contained the true false acceptance rate in forty-two settings. Their fix is a beta-binomial version with one correlation parameter, proven identifiable and extensible. Its intervals recovered proper coverage near ninety percent. Recovery quality tracked a gap in acceptance rates, measured by Youden's J statistic.

Catalog record

CREATORYichi Zhang, Ignacio Martinez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Stochasticity to Signal: A Bayesian Latent State Model for Reliable Measurement with LLMs Yichi Zhang, Ignacio Martinez Abstract Large Language Models (LLMs) are increasingly used to automate classification tasks in business, such as analyzing customer satisfaction from text. However, the inherent stochasticity of LLMs, in terms of their tendency to produce different outputs for the same input, creates a significant measurement error problem that is often neglected with a single round of output, or addressed with ad-hoc methods like majority voting. Such naive approaches fail to quantify uncertainty and can produce biased estimates of population-level metrics. In this paper, we propose a principled solution by reframing LLM variability as a statistical measurement error problem and introducing a Bayesian latent state model to address it. Our model treats the true classification (e.g., customer dissatisfaction) as an unobserved latent variable and the multiple LLM ratings as noisy measurements of this state. This framework allows for the simultaneous estimation of the LLM’s false positive and false negative error rates, the underlying base rate of the phenomenon in the population, the posterior probability of the true state for each individual observation, and the causal impact of a business intervention, if any, on the latent state. Through simulation studies, we demonstrate that our model accurately recovers true parameters where naive methods fail. We conclude that this methodology provides a general and reliable framework for converting noisy, probabilistic outputs from LLMs into accurate and actionable insights for scientific and business applications. \paperurl https://arxiv.org/abs/2510.23874 \uselogo \reportnumber 0001 keywords Large Language Models (LLMs),Measurement Error, Bayesian Latent Variable Model, Text Classification, Causal Inference, Customer Satisfaction, Noisy Labels 1 Introduction The business imperative for scalable customer satisfaction monitoring is clear, yet manual review of qualitative data like call transcripts remains a significant bottleneck. Gathering human expert ratings is expensive and time-consuming. Large Language Models (LLMs) have emerged as a natural and powerful tool for automating this classification task, exhibiting good alignment with human judgments on certain tasks ( Bavaresco et al. 2024 ) . However, a fundamental characteristic of LLMs, their probabilistic nature, presents a critical challenge. The stochasticity, while a feature for generating diverse text, becomes a source of measurement error in classification contexts where consistency is paramount. For example, if an LLM provides five ratings for a single call transcript and returns the set \{1,0,1,1,0\} , the true satisfaction state is ambiguous. Naive aggregation methods, such as using a single round of LLM output as the ground truth, or taking the majority vote or the average from multiple rounds of outputs, are insufficient. They fail to model the underlying error process, cannot quantify uncertainty, and do not easily incorporate prior knowledge or external factors. When the objective of the classification task is an important business outcome (e.g., user satisfaction) on which the impact of a business product or feature is of interest (e.g., a customer service workflow), the inability to characterize the measurement error by the naive methods hinders the causal evaluations of high-stake decisions. In this paper, we address these challenges by proposing a Bayesian latent state model. We reframe the problem of LLM variability as a solvable measurement error problem. Our model achieves three key goals: (1) it estimates the population’s base dissatisfaction rate and the LLM’s intrinsic error rates directly from the data; (2) it quantifies the uncertainty for each individual classification; and (3) it provides a framework for measuring the impact of business interventions on customer satisfaction. This paper proceeds as follows: Section 2 reviews related work, Section 3 details our model, Sections 4 and 5 provide validations via simulation, and Section 6 concludes with discussion. 2 Background & Related Work 2.1 The Proliferation of Large Language Models in Automated Text Classification The advent of Large Language Models (LLMs), powered by the transformer architecture, represents a paradigm shift in natural language processing (NLP). Pre-trained on vast text corpora, these models have demonstrated a remarkable capacity to understand and generate human-like text, enabling them to perform a wide array of tasks, including text classification, with minimal task-specific training. The application of LLMs for classification typically falls into three regimes: zero-shot learning, where the model classifies text based on a prompt without examples; few-shot learning, which improves performance by including a handful of examples in the prompt; and fine-tuning, which achieves state-of-the-art results by updating model weights on a larger labeled dataset ( Kostina et al. 2025 ) . While fine-tuning is powerful, many practical applications in business and science rely on the flexibility and low deployment cost of prompting-based methods (zero-shot and few-shot learning), making the variability inherent in this approach a particularly salient issue ( Lu et al. 2021 ; Liu et al. 2023 ) . In business analytics and customer intelligence, LLMs are widely used to automate the analysis of unstructured text from sources like customer reviews, support tickets, and social media ( Ipa et al. 2024 ; Ghatora et al. 2024 ; Wang et al. 2024 ) . This includes tasks such as sentiment analysis to gauge customer opinion, theme extraction to identify recurring issues like "product quality" or "customer service," and competitive intelligence to monitor brand perception. By moving beyond simple star ratings to a deeper, qualitative understanding of customer feedback, businesses can make more informed, customer-centric decisions ( Bharadwaj 2023 ) . Similarly, in the social sciences, LLMs are emerging as a tool to accelerate traditionally labor-intensive qualitative data analysis ( Bano et al. 2024 ) . Researchers are using LLMs to automate the "coding" of interview transcripts and open-ended survey responses, which promises to increase the scale and speed of qualitative inquiry. While this automation offers benefits such as reducing researcher subjectivity, the literature also highlights critical challenges, including a heavy reliance on prompt engineering and the risk of model "hallucinations," which points toward an underlying problem of reliability and measurement ( Barros et al. 2025 ) . The widespread adoption of LLMs in these applied domains has created a quiet measurement crisis: the models are often treated as deterministic black boxes, with little attention paid to the statistical implications of their inherent variability. 2.2 A Foundational Challenge: Probabilistic Outputs and Inherent Stochasticity as Measurement Error At their core, LLMs are probabilistic models that learn a distribution over a vocabulary of tokens ( Wang et al. 2023 ) . The variability in their output stems from the decoding strategy applied to this distribution. Techniques such as setting the temperature parameter and top- k sampling introduce randomness to generate diverse and creative text ( Kostina et al. 2025 ) . While this stochasticity is a feature for generative tasks, it becomes a significant challenge in classification contexts where consistency and accuracy are paramount ( Bender et al. 2021 ; Wang et al. 2022 ) . When an LLM is tasked with classifying an item that has a single, true underlying state (e.g., a customer is either dissatisfied or not), its output variability ceases to be a feature and must be treated as a source of measurement error ( Atil et al. 2024 ; Ludwig et al. 2025 ; Herrera-Poyatos et al. 2025 ) . Addressing this probabilistic nature of outputs from LLMs is analogous to resolving the conflicts from

Read the complete text