KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing

Yijia Fang, Yiqing Feng, Bingyu Li, Mingxun Zhou † · 2026

Read KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing by Yijia Fang, Yiqing Feng, Bingyu Li, Mingxun Zhou † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29524.txt.

Introduction

Yijia Fang and three colleagues present KBF, a black-box audit checking which model serves a relay or reseller endpoint. Users cannot see behind such endpoints, so KBF probes steady recall near the knowledge boundary, counting wrong answers that repeat. It calibrates decisions against the reference model's own variation. Across sixteen production endpoints it catches all 155 relevant substitutions without rejecting any same-reference controls. Field audits flag seven of twenty-eight endpoints as inconsistent, at under a dollar per audit after enrollment.

Catalog record

CREATORYijia Fang, Yiqing Feng, Bingyu Li, Mingxun Zhou †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE98040
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29524

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing Yijia Fang, Yiqing Feng, Bingyu Li, Mingxun Zhou † Abstract Relay and reseller APIs increasingly intermediate access to large language models (LLMs), but users have no direct way to verify that a claimed endpoint is actually serving the advertised model. We introduce KBF , a low-cost black-box auditing protocol that fingerprints model APIs using stable numerical recall near the knowledge boundary. Across 16 production LLM endpoints, KBF flags all 155 economically relevant substitutions without rejecting any same-model controls, remains stable under deployment variation, detects high-separation mixed-routing attacks when only 5–10% of traffic is substituted, and finds that 7 of 27 platform–model cells in a six-platform shadow API audit are statistically inconsistent with their reference endpoints, with inconsistencies concentrated on premium Claude endpoints. † † footnotetext: Corresponding author. I Introduction Relay and reseller APIs are now a common access path for large language models (LLMs). Users buy credits from a third-party service and send requests through a black-box endpoint, while the relay chooses the upstream model provider. Established aggregators offer legitimate benefits, such as unified billing and simple endpoint switching across models. At the same time, the growing token demand of agentic workloads has pushed users toward newer and less-audited resellers that advertise lower prices or broader model availability. This convenience creates a basic trust problem: the endpoint may not be serving the model it claims to serve . A relay can silently replace an expensive advertised model with a cheaper backend, or mix reference-consistent and substituted traffic to reduce cost while making the behavior harder to notice. The incentive is direct: flagship models can be orders of magnitude more expensive than budget models, and recent measurement work shows that deceptive model claims in shadow APIs already occur at meaningful scale, affecting both research reproducibility and service reliability [ 1 ] . The practical need for auditing has already produced an ecosystem of informal checks. Users ask the model to identify itself, pose ad hoc ‘‘tricky’’ questions, or rely on community auditing services with opaque methodologies 1 1 1 For example, hvoy.ai and llmtest.cn. This is not an endorsement: the authors have no connection with any existing auditors and/or relay services. . These checks are not a sound basis for security decisions. Self-identification is unreliable because LLMs often misstate their own identity even in benign settings [ 2 ] . Hand-crafted challenge prompts can be brittle, model-specific, and easy for a provider to evade by special-casing; public auditing services often expose neither their decision rule nor their conflict-of-interest posture. This motivates a concrete technical question: given black-box access to an official reference API and a suspect relay API, can an auditor test whether the relay is serving the claimed model? We deliberately study the access level available to an ordinary relay user. The auditor can send prompts to the official endpoint and the suspect endpoint, but cannot inspect the served model, observe token probabilities, read hidden system prompts, or see server-side routing logs. This is a setting strictly more challenging and yet more practical than the “white-box” or the “grey-box” setting addressed by many model-provenance and fingerprinting proposals, where the verifier can inspect model internals, observe richer outputs such as logits, or embed secret triggers or watermarks during model preparation [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . These mechanisms are valuable when the model owner or serving platform cooperates, but they do not apply to relay users facing a potentially uncooperative intermediary. We envision that a useful audit protocol in this setting should satisfy four requirements: • High true-positive rate : the protocol should detect economically meaningful model substitution while maintaining a very low false-positive rate; in this setting, a false accusation against an honest provider is costly. • Robustness to deployment variation : the suspect endpoint may use unknown system prompts, decoding parameters, output-length limits, RAG wrappers, or agent-style interfaces. The audit should not require the auditor to recover the exact deployment configuration. • Benign and low-friction queries : the protocol should avoid prompt injection, harmful-content requests, or other security-sensitive probes that may be blocked, rate-limited, or answered adaptively. Audit traffic should resemble ordinary user traffic. • Economic practicality : the protocol should be cheap for auditors to run repeatedly, while making selective evasion expensive for dishonest relays even when the protocol is public. Existing black-box techniques do not satisfy these requirements simultaneously. Output-distribution tests such as MET [ 10 ] and ZeroPrint [ 11 ] compare response distributions over many queries, but their fingerprints are sensitive to deployment context and show high variance under wrappers and system-prompt changes. Behavioral fingerprinting methods such as LLMmap and LLMPrint show that few-query model identification is possible [ 12 , 13 ] , but they rely on fixed behavioral probes, including prompt-injection-style or safety-sensitive interactions, that are poorly suited to routine third-party auditing. Standard benchmark evaluation is conceptually straightforward but too expensive and too slow to serve as a practical relay audit. Given the status quo, we ask: Can we build a black-box relay audit that is discriminative, robust to deployment variation, benign to run, and economically practical? I-A Our Contribution Main observation. Our key observation is that useful audit signal appears near the knowledge boundary . When queried about numerical facts near the edge of factual recall, models often do not produce arbitrary noise. Instead, they produce stable, model-distinct values, including repeatable wrong answers. Operationally, these responses behave like persistent model-specific parametric associations. This makes boundary numerical recall a stronger audit signal than self-identification, style, or high-variance continuation behavior. It is also operationally attractive: the probes can be ordinary-looking factual queries, and the comparison does not require embeddings or semantic similarity models, which can introduce their own instability [ 14 , 15 ] . Technical Contribution: KBF auditing protocol. Based on this observation, we design Knowledge-Boundary Fingerprinting (KBF), a black-box relay-auditing protocol built around stable numerical recall near the knowledge boundary. KBF has three stages. First, in an offline probe-generation stage, the auditor uses the official reference API to construct a model-specific candidate set. Second, KBF screens candidates for stability across reference configurations and, when useful, for contrast against cheaper substitute models. Third, in the online audit, the auditor queries the suspect endpoint, compares the returned numerical values against the reference consensus under domain-specific tolerances, and applies a statistical test to decide whether the suspect endpoint is consistent with the claimed model. The main technical challenge is to generate probes that are simultaneously stable for the reference model, discriminative against likely substitutes, and cheap to obtain. KBF addresses this challenge with an adaptive frontier search . For each knowledge domain, the reference model is asked to propose candidate numerical facts from domain-specific themes. Across rounds, the generation prompts move toward increasingly obscure and specialist-only facts. Each candidate is then re-queried in a short audit format under multiple deployment configurations; only candidates with stable recall are retained. To remove probes that are stable but weakly discriminative, KBF optionally performs contrastive screening against a smaller or cheaper model (e.g., a 9-billion parameter model) and discards candidates on which the contrast model agrees with

Read the complete text