Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation

Ziyao Wang, Guoheng Sun, Yexiao He, Zheyu Shen, Bowei Tian, Ang Li · 2025

Read Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation by Ziyao Wang, Guoheng Sun, Yexiao He, Zheyu Shen, Bowei Tian, Ang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00912.txt.

Introduction

Ziyao Wang, Guoheng Sun, Yexiao He, Zheyu Shen, Bowei Tian, and Ang Li tackle a billing worry with paid language model services in work published in 2025. Services often hide their inner reasoning steps yet still charge users for every token made, including the hidden ones. Users have no easy way to check whether those hidden counts are fair.

The team builds a tool called PALACE that guesses the hidden reasoning length from just the prompt and the final answer. A small routing part sorts tasks by type so the guess can be tuned for math, coding, medical, and general questions. Tests across those fields showed low error rates. The authors present the tool as a first step toward open checks on token billing.

Catalog record

CREATORZiyao Wang, Guoheng Sun, Yexiao He, Zheyu Shen, Bowei Tian, Ang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00912

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation Ziyao Wang, Guoheng Sun, Yexiao He, Zheyu Shen, Bowei Tian, Ang Li Abstract Commercial LLM services often conceal internal reasoning traces while still charging users for every generated token, including those from hidden intermediate steps, raising concerns of token inflation and potential overbilling. This gap underscores the urgent need for reliable token auditing, yet achieving it is far from straightforward: cryptographic verification ( e.g., hash-based signature) offers little assurance when providers control the entire execution pipeline, while user-side prediction struggles with the inherent variance of reasoning LLMs, where token usage fluctuates across domains and prompt styles. To bridge this gap, we present PALACE ( Predictive Auditing of LLM APIs via Reasoning Token Count Estimation ), a user-side framework that estimates hidden reasoning token counts from prompt–answer pairs without access to internal traces. PALACE introduces a GRPO-augmented adaptation module with a lightweight domain router, enabling dynamic calibration across diverse reasoning tasks and mitigating variance in token usage patterns. Experiments on math, coding, medical, and general reasoning benchmarks show that PALACE achieves low relative error and strong prediction accuracy, supporting both fine-grained cost auditing and inflation detection. Taken together, PALACE represents an important first step toward standardized predictive auditing, offering a practical path to greater transparency, accountability, and user trust. 1 Introduction Large language model (LLM) services have made significant progress in recent years, incorporating capabilities such as multi-step reasoning Guo et al. 2025 , external tool use Chen et al. 2024 ; Zhuang et al. 2023 , and agent coordination Zhao et al. 2024 . These advancements enhance generation quality and expand the applicability of LLMs to a wider range of real-world domains. However, these reasoning LLMs also produce long internal outputs that often include verbose and noisy content, such as backtracking traces or speculative reasoning branches Aggarwal and Welleck 2025 , which may not benefit end users and can hinder comprehension. Moreover, revealing such internal traces poses security and intellectual property risks: malicious users could extract model behaviors or replicate proprietary agent workflows, threatening the commercial value of LLM systems. To mitigate these risks, LLM service providers like OpenAI (ChatGPT) OpenAI 2025 , Google (Gemini) Google Gemini 2025 , and Anthropic (Claude) typically withhold internal tokens generated during reasoning or tool execution. Nonetheless, users are still billed for the full sequence of tokens, including both hidden internal tokens and visible final outputs. Such services have been formally defined as Commercial Opaque LLM Services (COLS) Sun et al. 2025b . This opaque design raises serious transparency concerns : users are charged for token usage they cannot observe or verify. All usage data is reported solely by the service provider, whose financial incentives may conflict with those of the user. This creates the risk of token inflation , where COLS may over-report token usage to increase billing. Empirical studies show that in many state-of-the-art models, over 90% of tokens are consumed in hidden reasoning, with only a small portion forming the final answer Sun et al. 2025a . Even minor inflation under such conditions can result in substantial overcharging, particularly for large-scale API users engaged in synthetic data generation, annotation, or document processing. For example, a single high-efficiency ARC-AGI evaluation run using OpenAI’s o3 model consumed 111 million tokens, resulting in a cost of $66,772 Chollet 2025 , in which more than 60% are the cost of hidden reasoning tokens, implying that a 30% misreporting of hidden tokens could cost more than $12,000. Auditing token usage under these conditions is challenging for two reasons. First, cryptographic approaches Sun et al. 2025a offer little assurance when providers control the entire execution pipeline and can selectively expose data. Second, user-side heuristics based on input–output lengths are unreliable, because reasoning token usage varies drastically with domain, prompt style, and task complexity Jin et al. 2024 . Together, these challenges create a critical need for an auditing solution that works without access to hidden traces and remains robust to reasoning variability . Figure 1: The transparency concern of COLS. To address these challenges, we introduce PALACE ( Predictive Auditing of LLM APIs via Token Count Estimation ), the first user-side framework to estimate hidden reasoning token usage directly from prompt–answer pairs. PALACE requires COLS to release lightweight auxiliary datasets that are verifiable and guaranteed to be entirely generated by COLS’s own LLMs, fulfilling their obligation to provide certifiable data for auditing. Each sample in these datasets includes a user prompt, the model’s reasoning process, and the final answer. PALACE addresses the trace inaccessibility challenge by learning to infer reasoning length from observable input–output semantics, eliminating any dependency on internal execution data. It addresses the reasoning variability challenge through two key innovations: a GRPO Shao et al. 2024 (Group Relative Policy Optimization)-driven adaptation module that improves reasoning length inference for complex tasks, and a lightweight domain router that dynamically calibrates predictions across diverse usage scenarios. For each auditing instance, the user provides her prompt and the corresponding response returned by COLS, along with the API configuration. PALACE invokes the router to select the appropriate GRPO adapted module, and performs the token count prediction. PALACE returns both per-sample predictions and aggregated estimates across multiple queries to support comprehensive auditing of the service. Different from prior auditing methods such as Sun et al. 2025a , PALACE neither relies on hidden tokens or intermediate embeddings during inference, nor requires users to issue additional queries or requests during the auditing process. This makes PALACE a lightweight and user-transparent auditing solution for COLS. Our main contributions are summarized as follows: • We introduce reasoning length estimation as a foundation for user-side auditing of hidden reasoning token usage in commercial COLS. • We present PALACE , which combines a GRPO-augmented adaptation module with a lightweight domain router to infer hidden reasoning length directly from prompt–answer pairs, without access to internal execution traces. • We construct multi-domain benchmark datasets and show that PALACE achieves low relative error and stable cumulative accuracy across reasoning-heavy tasks, enabling fine-grained cost auditing and reliable detection of token inflation. 2 Background and Motivation 2.1 COLS and Hidden Tokens The development of reasoning LLMs Muennighoff et al. 2025 ; Hao et al. 2024 and agentic LLMs Chan et al. 2023 rapidly changes the service module of LLM service providers. Set reasoning LLM API as an example, OpenAI now provides multiple reasoning model inference APIs such as o3, o4-mini, o4-mini-high, etc Jaech et al. 2024 ; OpenAI 2025 . When a user submits a prompt to one of these APIs, the reasoning model typically generates a reasoning trace and the final answer. While most major providers return only the final answer in the API response, they charge based on the total number of tokens consumed by both the reasoning trace and the answer. As a form of compensation, service providers such as OpenAI, Gemini, and Claude offer users only a summarized version of the reasoning content for inspection, while the full reasoning trace remains hidden. Nevertheless, the total number of reasoning tokens, including those not visible to the user, is still accounted for in the billing. OpenAI explicitly states in its documentation: While reasoning tokens are not visible via

Read the complete text