Quantifying Uncertainty in AI Visibility A Statistical Framework for Generative Search Measurement
Ronald Sielinski · 2026
Read Quantifying Uncertainty in AI Visibility A Statistical Framework for Generative Search Measurement by Ronald Sielinski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08924.txt.
Introduction
Ronald Sielinski argues that AI search visibility scores need uncertainty bands. Identical queries return different answers and cites across runs, yet current measures report single-run citation share as fixed. His study of three platforms finds citation distributions follow a power law with wide spread, so many apparent domain gaps sit inside measurement noise and rankings shift across samples.
Catalog record
| CREATOR | Ronald Sielinski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91564 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08924
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantifying Uncertainty in AI Visibility A Statistical Framework for Generative Search Measurement Ronald Sielinski Abstract AI-powered answer engines are inherently non-deterministic: identical queries submitted at different times can produce different responses and cite different sources. Despite this, current approaches to measuring domain visibility in generative search rely on single-run point estimates of citation share and prevalence, treating them as fixed values. This paper argues that citation visibility metrics should be treated as sample estimators of an underlying response distribution. We conduct an empirical study of citation variability across three generative search platforms—Perplexity Search, OpenAI SearchGPT, and Google Gemini—using repeated sampling across three consumer product topics under two sampling regimes: daily collections over nine days and high-frequency sampling at ten-minute intervals. Citation distributions follow a power-law form and exhibit substantial variability across repeated samples. Bootstrap confidence intervals reveal that many apparent differences between domains fall within the noise floor of the measurement process. Distribution-wide rank stability analysis further demonstrates that citation rankings are unstable across samples, not only among top-ranked domains but throughout the frequently cited domain set. These findings show that single-run visibility metrics provide a misleadingly precise picture of domain performance in generative search. We argue that citation visibility must be reported with uncertainty estimates and provide practical guidance on sample sizes required to achieve interpretable confidence intervals. CCS Concepts: Information systems \rightarrow Information retrieval \rightarrow Evaluation of retrieval results; Information systems \rightarrow World Wide Web \rightarrow Web searching and information discovery. Additional Keywords and Phrases: AI visibility, answer engine optimization, generative engine optimization. 1 Introduction Generative search engines have transformed how users encounter information online. Rather than returning a ranked list of hyperlinks, platforms such as Perplexity Search, OpenAI SearchGPT, and Google Gemini synthesize a natural-language response to each query and attribute claims within that response to a set of cited web sources. For brand managers, digital marketers, and content strategists, these citations have become a new form of visibility: appearing in a cited source confers authority and drives traffic in a way that a traditional search ranking no longer guarantees. Practitioners have responded by developing measurement frameworks that report, for example, citation counts, citation share, and citation prevalence as metrics of AI visibility. These reports typically derive from a single batch of queries submitted on a single occasion. The resulting numbers are treated as ground truth: domain A has 12% citation share, domain B has 8%, and A is therefore more visible. Optimization campaigns are launched, interventions are measured, and conclusions are drawn—all on the basis of single-run point estimates. This paper argues that such measurements are fundamentally unreliable, because the systems they measure are non-deterministic. The same query submitted twice to the same platform can produce a different cited source set. This is not a bug or an edge case: it is a design property of generative AI systems, which use stochastic sampling at generation time. The practical consequence is that citation visibility metrics are random variables, not fixed values, and any single measurement of them carries unquantified uncertainty that may be large enough to invalidate common analytical conclusions. 1.1 Motivating Example Consider a concrete decision a practitioner faces every day. A brand manager measuring domain visibility on SearchGPT for the topic of running gear collects a sample of 200 queries and observes that tomsguide.com has a citation share of approximately 9.5%, while runnersworld.com has a citation share of approximately 6.0%. The conclusion seems clear: Tom’s Guide is the dominant source for this topic on this platform, outperforming Runner’s World by more than three percentage points. Resources are allocated accordingly. The 95% bootstrap confidence intervals tell a different story. The interval for tomsguide.com spans approximately 5.5% to 12.5%; the interval for runnersworld.com spans approximately 4.0% to 8.0%. These intervals overlap substantially: the entire runnersworld.com interval falls within the tomsguide.com interval. The apparent difference of 3.5 percentage points is well within the range of sampling noise. A practitioner cannot conclude from this sample that tomsguide.com outperforms runnersworld.com with any statistical confidence. This is not a pathological edge case. Across the platforms and topics studied in this paper, overlapping confidence intervals of this kind are the norm rather than the exception for domains that appear to differ in citation share by less than 5-7 percentage points. The bootstrap confidence interval makes the uncertainty explicit and actionable: it transforms an apparently decisive ranking into a result that is statistically indistinguishable from a tie. Citation visibility metrics must be accompanied by uncertainty estimates, or they are liable to mislead. This paper reframes the measurement of domain visibility in generative search as a statistical estimation problem for stochastic information systems and demonstrates empirically that reliable visibility measurement requires repeated sampling and explicit uncertainty quantification. 1.2 Contributions This paper makes the following contributions: • An empirical characterization of citation variability in three generative search platforms across repeated sampling, demonstrating that stochastic system behavior produces material differences in measured domain visibility. • A formal framework distinguishing system-level stochasticity (the answer engine produces different outputs on repeated calls) from measurement uncertainty (estimates derived from finite samples are imprecise) and showing that both are present and consequential. • A demonstration that citation count distributions follow a power-law form, and that variability differs structurally between the head and tail of the distribution—with implications for how frequently and infrequently cited domains should be analyzed. • An application of bootstrap confidence intervals to citation visibility metrics, showing that they are both necessary (intervals are wide relative to observed differences) and computationally tractable. • A distribution-wide rank stability analysis using weighted Spearman rank correlation, demonstrating that citation rank instability extends across the full frequently-cited domain set and is not confined to the top-ranked domains, establishing that confidence intervals are necessary for inference at any rank position in the distribution. • A methodological control using content checksums to verify that observed variability is attributable to the answer engines rather than to changes in the content of cited pages. 2 Related Work 2.1 Citation Quality in Generative Search A growing body of work examines the accuracy and faithfulness of citations in generative search outputs. Liu et al. establish that generative systems frequently cite sources that do not support the attributed claims and introduce metrics for citation recall and precision [ 12 ] . Gao et al. present the ALCE benchmark for automatic citation evaluation, which operationalizes citation quality as the degree to which cited passages entail generated claims [ 7 ] . Xu et al. extend this framework with CiteBench, introducing fine-grained criteria for evaluating citation faithfulness across multiple dimensions [ 15 ] . Li et al. propose AttributionBench, targeting the attribution verification problem specifically [ 11 ] . This literature treats citation accuracy as the central measurement challenge: given a response and its citations, do the citations support the claims? Implicit in this framing is that each measurement is a reliable observation about a well-defined system state. None of these works address the variability of citation selection across repeated runs of the same query—the question of whether the set of cited sources itself is stable. Our work complements this literature by characterizing the stochastic properties of the citation process rather than the quality of individual citations. 2.2 Visibility and Optimization