Behavioral Fingerprints for LLM Endpoint Stability and Identity

Jonah Leshin, Manish Shah, Ian Timmis, Daniel Kang · 2026

Read Behavioral Fingerprints for LLM Endpoint Stability and Identity by Jonah Leshin, Manish Shah, Ian Timmis, Daniel Kang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19022.txt.

Introduction

Leshin and co-authors build a monitor that tracks whether an AI model endpoint stays the same over time. Uptime and speed checks miss silent changes to weights, token sets, or serving stacks. Their tool samples outputs from a fixed prompt set on a schedule. It compares output spreads across time with a distance score and shuffle tests. In trials, it spots shifts in model family, version, and settings. Watching one model across hosts, they find wide gaps in stability between and within providers.

Catalog record

CREATORJonah Leshin, Manish Shah, Ian Timmis, Daniel Kang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17980
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Behavioral Fingerprints for LLM Endpoint Stability and Identity Jonah Leshin, Manish Shah, Ian Timmis, Daniel Kang Abstract . The consistency of AI-native applications depends on the behavioral consistency of the model endpoints that power them. Traditional reliability metrics such as uptime, latency and throughput do not capture behavioral change, and an endpoint can remain “healthy” while its effective model identity changes due to updates to weights, tokenizers, quantization, inference engines, kernels, caching, routing, or hardware. We introduce Stability Monitor , a black-box stability monitoring system that periodically fingerprints an endpoint by sampling outputs from a fixed prompt set and comparing the resulting output distributions over time. Fingerprints are compared using a summed energy distance statistic across prompts, with permutation-test p-values as evidence of distribution shift aggregated sequentially to detect change events and define stability periods. In controlled validation, Stability Monitor detects changes to model family, version, inference stack, quantization, and behavioral parameters. In real-world monitoring of the same model hosted by multiple providers, we observe substantial provider-to-provider and within-provider stability differences. 1. Introduction 1.1. Reliability is not stability The software deployment intuition “if it’s up and fast, it’s fine” is insufficient for AI-native systems: an endpoint can pass standard health checks while its outputs drift, shifting formatting, tone, or tool behavior in ways that break downstream parsers and guardrails. Stability failures are particularly impactful in multi-step agentic workflows, where small variance in early steps can compound into large workflow variance. This motivates stability as a distinct operational metric: behavioral consistency of a model endpoint under a fixed interface contract. Even if users fix visible settings (e.g., temperature), endpoint behavior can vary due to system level nondeterminism “beneath the surface”: variance in inference engines, kernels, caching, batch sizes, and hardware—and providers may route requests across heterogeneous environments. The values of these factors at run time may be a function of overall system load, which the user cannot control, thus making the endpoint nondeterministic from the user’s perspective ( 6 ) . These system-level factors, which can result in nondeterministic behavior even at temperature 0, must be considered on top of discrete, deliberate updates such as a model version change, undermining reproducibility in both production and evaluation environments. Evaluation instability and provider effects are now recognized as a major source of variance in benchmarking results ( 1 ) . 1.2. Contribution and scope We introduce two components: Stability Monitor , which continuously fingerprints endpoints and detects behavioral change events, and Stability Arena , a web application that publishes and visualizes the data Stability Monitor produces. Together, the goal is to operationalize continuous stability monitoring under realistic constraints: • Black-box : observe only public API I/O; no privileged access to weights or provider internals. • Lightweight and frequent : monitor at high cadence versus heavy periodic evals (motivating the notions of stability periods and change events). • Actionable : produce audit trails and change event alerts usable by engineering, security, and compliance. Stability Arena exposes the system’s monitoring capabilities—built on its fingerprinting methodology and sequential change-detection pipeline—through a live web interface that lets users inspect these capabilities against real endpoint data. We contribute: (1) A practical fingerprinting and change-detection pipeline based on energy-distance permutation testing and sequential evidence aggregation. (2) Controlled validation across multiple real-world change types. (3) Real-world provider comparisons for the same nominal model showing large cross-provider and within-provider stability differences, motivating “same model, different behavior” as an operational reality. 2. Background and related work Prior work on model fingerprinting includes frameworks for intellectual property protection– verifying whether a deployed model is a copy or derivative of a protected model ( ( 2 ; 11 ) ). These methods typically require white-box access or adversarial inputs crafted to distinguish specific architectures. Our work differs in both goal and setting: rather than verifying model ownership, we detect behavioral change at an endpoint over time, using only standard black-box API access with natural-language prompts. Chen et al. ( 4 ) demonstrated that the behavior of GPT-3.5 and GPT-4 changed substantially over several months, with measurable shifts in accuracy, formatting, and safety behavior, providing direct motivation for continuous behavioral monitoring of production endpoints. More recent work on black-box change detection includes B3IT ( 3 ) , which uses per-endpoint “border inputs”—prompts where multiple tokens are nearly tied as the top prediction at low temperature. Like our approach, B3IT frames the problem as a hypothesis test on the output distribution and targets continuous monitoring. A key difference is that B3IT’s probes are tailored to each endpoint during an initialization phase and may require re-discovery after a change event, since the model’s decision boundaries may have shifted. Our system uses a fixed, model-agnostic prompt set that persists across change events and endpoints, avoiding per-endpoint initialization. We use an energy distance statistic as the core metric comparing samples of response embeddings. This metric is a nonparametric distance between distributions and is zero if and only if the distributions match under mild conditions (as a metric on distributions of random vectors) ( 10 ) . Energy distance is closely related to maximum mean discrepancy (MMD) ( 5 ) ; we chose energy distance for its interpretability and parameter-free formulation. We use sequential evidence accumulation based on e-values (expectation-bounded test statistics) to enable continuous monitoring with optional stopping behavior, suited for streaming detection settings ( 12 ; 9 ) . Our statistical approach to measuring behavioral change is complementary to projects that run domain-specific evaluations over time ( 7 ; 8 ) . These efforts offer a signal about specific abilities over time, but are typically expensive to run, limiting the feasibility of coverage across a range of models and providers. 3. Problem definition In the context of text-to-text models, we define endpoint stability as distributional consistency of responses under a fixed prompt set and fixed user-visible settings. We treat the endpoint as a black box. Endpoint instability may occur due to changes to: • model identity (family, version, tokenizer, weights), • deployment (inference engine, stack), • optimization techniques (quantization, caching, kernels, speculative decoding), • configurations (sampling parameters, system prompts) Our objective is to detect when there has been a change in the endpoint’s effective output distribution and to create an audit trail of change events and stability periods . 4. Methodology: fingerprints and change detection 4.1. Fingerprint construction A fingerprint is constructed by selecting a fixed set of prompts and sampling multiple responses per prompt; responses are embedded as real-valued vectors. A fingerprint thus consists of a set of sets of vectors —one sample set per prompt. Our implementation requires a total of 800 inference requests, each consisting of a few tokens, to be made to an endpoint to generate a fingerprint. 4.2. Pairwise fingerprint comparisons Given fingerprints X and Y , we compute an energy distance statistic per prompt between the prompt-specific embedding samples, and sum across prompts to produce an aggregate distance E=E(X,Y) . To test the null hypothesis that two fingerprints are drawn from the same underlying distribution, we compute a p-value via a permutation test on the aggregate statistic E . To compute the p-value, the embedding vectors for each prompt are pooled and randomly re-split into two groups of the original sizes according

Read the complete text