When Is the Same Model Not the Same Service? A Measurement Study of Hosted Open-Weight LLM APIs
Haorui Li Zhenghui He Xuanzi Liu Yang Xu Dongsheng Liu Jiakang Ma Lupan Wu Yangjie Wu Xiongchao Tang Tianhui Shi * QingCheng.AI AIPing Team * Corresponding author: Tianhui Shi ( [email protected] ) · 2026
Read When Is the Same Model Not the Same Service? A Measurement Study of Hosted Open-Weight LLM APIs by Haorui Li Zhenghui He Xuanzi Liu Yang Xu Dongsheng Liu Jiakang Ma Lupan Wu Yangjie Wu Xiongchao Tang Tianhui Shi * QingCheng.AI AIPing Team * Corresponding author: Tianhui Shi ( [email protected] ) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02821.txt.
Introduction
Haorui Li and nine co-authors measure how the same open-weight language model becomes a different service depending on who hosts it. Users rarely run the raw model; they call a provider's endpoint, which varies in variant, protocol behavior, context size, price, speed, reliability, and what tasks it can actually do. The paper argues the service, not the artifact, is the real unit.
Data come from a monitoring platform's request logs, provider records, compatibility probes, price snapshots, and steady latency measures over one quarter. Demand concentrates in a few model families, yet old versions stay active after new releases. A provider listing many models does not mean users adopt them, and prices stay steadier than speed, context, or error behavior. Task mix shifts token patterns, so the best provider depends on the job.
Catalog record
| CREATOR | Haorui Li Zhenghui He Xuanzi Liu Yang Xu Dongsheng Liu Jiakang Ma Lupan Wu Yangjie Wu Xiongchao Tang Tianhui Shi * QingCheng.AI AIPing Team * Corresponding author: Tianhui Shi ( [email protected] ) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02821
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Is the Same Model Not the Same Service? A Measurement Study of Hosted Open-Weight LLM APIs Haorui Li Zhenghui He Xuanzi Liu Yang Xu Dongsheng Liu Jiakang Ma Lupan Wu Yangjie Wu Xiongchao Tang Tianhui Shi * QingCheng.AI AIPing Team * Corresponding author: Tianhui Shi ( [email protected] ) Abstract Open-weight large language models (LLMs) are usually named as model artifacts, but production users often consume them as hosted API services. This paper argues that the operational unit is a service object : a provider-specific, time-varying endpoint defined by model variant, protocol behavior, context capacity, listed price, latency and throughput distribution, reliability, and task feasibility. Using sampled request logs, provider metadata, compatibility probes, pricing snapshots, and continuous latency measurements collected by AI Ping during Q4 2025, we study how this service layer changes the meaning of “the same model.” Three empirical patterns emerge. First, observed demand is concentrated but persistent across versions: in the displayed family aggregate, the largest family carries 32.0% of relative demand and the top five carry 87.4%, with a Gini coefficient of 0.693, while older variants remain active after newer releases. Second, supply and use separate: provider listing breadth does not imply realized adoption, and listed prices are more anchored than latency, throughput, context length, protocol support, and error semantics. Third, task mix matters: applications induce different token-length regimes, so provider choice is a constrained decision over provider-model-task-time tuples rather than a lookup by model name. In two representative counterfactuals under observed feasibility constraints, routing lowers Qwen3-32B cost by 37.8% and raises DeepSeek-V3.2 average throughput by about 90% relative to direct official access. The results support a measurement view of hosted open-weight LLMs as heterogeneous services, not static catalog entries. We open-source the measurement methodology and reproduction artifacts at https://github.com/haoruilee/llm_api_measurement_study to support result reproduction. 1 Introduction Modern LLM deployment no longer ends at a model release. Open-weight models such as DeepSeek, Qwen, Kimi, GLM, and MiniMax are not only artifacts for local inference; they also become shared endpoints served by multiple providers with different prices, capacity plans, protocol implementations, context windows, safety wrappers, quantization choices, and latency profiles. The same nominal model can therefore expose different operational behavior depending on where and when it is called. This paper focuses on the cross-provider API layer that sits between model capability and application behavior. We define a hosted open-weight LLM service object as S=(v,p,\pi,W,C,L,R,t), where v is the model variant or checkpoint label, p is the provider, \pi is the protocol surface, W is the effective context policy, C is the price schedule, L is the latency and throughput distribution, R is reliability and error recoverability, and t is measurement time. This definition separates a model artifact from the service actually consumed by applications. The measurement target is not whether one model is intrinsically better than another, but how nominally identical or closely related open-weight models behave once they are exposed through independently operated API endpoints. We distinguish three identity layers throughout the paper: checkpoint-level claims require evidence about the exact released weights; variant-level claims concern named provider variants such as instruction, thinking, or dated releases; family-level claims aggregate related variants only for distributional summaries. This distinction is central because many public dashboards and API catalogs collapse these layers, while production behavior often depends on the finer service object. Using the Q4 2025 AI Ping measurement sample, we study request volume, provider coverage, listed prices, time to first token (TTFT), tokens per second (TPS), context length, protocol compatibility, slow-response behavior, task category, and routing policy. The dataset is a sampled measurement view from AI Ping rather than a census of all LLM API activity; the results should therefore be read as evidence about the observed service layer, not as global market shares. The main finding is that model name is not a sufficient operational unit. Demand is concentrated, but older variants remain in use; provider coverage is broad, but realized usage is uneven; prices are often anchored, but latency, throughput, protocol behavior, context capacity, and error recoverability vary enough to change application outcomes. Routing has measurable value in the sample because it exploits cross-provider dispersion under feasibility constraints, not because any provider is universally superior. Throughout the paper, directly observed aggregate facts are separated from mechanism-level interpretations suggested by the data. 2 Background and Related Work Recent model technical reports document the rapid improvement of open-weight LLM capability. The open-weight families appearing in or adjacent to this study are documented by DeepSeek-V2 [ 1 ] , DeepSeek-V3 [ 2 ] , DeepSeek-R1 [ 3 ] , Qwen [ 4 ] , Qwen2.5 [ 5 ] , Qwen3 [ 6 ] , Kimi K2 [ 7 ] , MiniMax-01 [ 8 ] , MiniMax-M1 [ 9 ] , and GLM-4.5 [ 10 ] . These reports primarily describe model architectures, training data, post-training procedures, benchmark performance, context capability, or reasoning behavior. Their natural unit of analysis is the model family or checkpoint. This paper instead asks what happens after those models enter a multi-provider API market, where the same nominal model can be exposed through different infrastructure and protocol layers. The systems literature studies how LLM serving systems manage batching, scheduling, memory, and throughput. Orca introduced iteration-level scheduling and selective batching for transformer serving [ 11 ] ; PagedAttention improves KV-cache memory management for high-throughput serving [ 12 ] ; SGLang optimizes structured generation workloads through a co-designed frontend and runtime [ 14 ] ; and DistServe separates prefill and decode resources to optimize latency-constrained serving [ 15 ] . Recent surveys summarize the broader design space of LLM inference serving [ 13 ] . This paper is complementary: it does not propose a serving system, but measures externally visible consequences of heterogeneous serving deployments. A related line of work studies cost-quality trade-offs and model routing. FrugalGPT formulates cascades for reducing LLM inference cost while preserving quality [ 16 ] , and RouteLLM learns to route requests between models using preference data [ 17 ] . Our setting differs in that routing occurs not only across models but also across providers for the same model family. The decision boundary therefore includes context-window feasibility, protocol compatibility, latency tails, throughput, error semantics, and time-varying provider behavior. Two other lines of work are especially relevant to our measurement boundary. First, Model Equality Testing formulates the question “which model is this API serving?” as a two-sample testing problem, motivated by the possibility that providers may quantize, watermark, fine-tune, or otherwise modify an advertised model [ 18 ] . We do not claim checkpoint-level equality for every provider endpoint in this paper. Instead, we treat model identity as a measured boundary condition and report family-level or variant-level summaries only where the evidence supports them. Second, large-scale usage studies such as the OpenRouter 100T token study and BurstGPT emphasize that real production traffic differs from benchmark workloads and that request metadata, timing, model/provider identifiers, token counts, and cancellation behavior define the observable sample frame [ 19 , 20 ] . Our methodology follows this measurement style but focuses specifically on hosted open-weight API service heterogeneity. 3 Measurement Methodology 3.1 Dataset and Units of Analysis The study combines several observation units over the Q4 2025 measurement