FEval-TTC: Fair Evaluation Protocol for Test-Time Compute
Pavel Rumiantsev, Soumyasundar Pal, Yingxue Zhang, Mark Coates · 2025
Read FEval-TTC: Fair Evaluation Protocol for Test-Time Compute by Pavel Rumiantsev, Soumyasundar Pal, Yingxue Zhang, Mark Coates (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01203.txt.
Introduction
Pavel Rumiantsev, Soumyasundar Pal, Yingxue Zhang, and Mark Coates propose a fair way to evaluate test time compute methods. These techniques spend extra computation at answer time, often through chains of thought, to improve language model results. Model performance and API prices drift over time, which can silently invalidate comparisons published earlier.
Their protocol, FEval-TTC, standardizes evaluation across several models and a diverse set of mathematical and commonsense reasoning data sets. Few shot prompting and answer extraction are fixed across data sets, cutting researcher overhead, and a cost model estimates both token usage and dollar cost per query. The goal is comparisons that stay equitable as prices and models change. The protocol is released as open source software.
Catalog record
| CREATOR | Pavel Rumiantsev, Soumyasundar Pal, Yingxue Zhang, Mark Coates |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01203
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FEval-TTC: Fair Evaluation Protocol for Test-Time Compute Pavel Rumiantsev, Soumyasundar Pal, Yingxue Zhang, Mark Coates Abstract The performance of Large Language Models (LLMs) and the associated dollar costs of API calls can fluctuate over time, potentially invalidating conclusions drawn in prior research. To address this, we propose a F air Eval uation protocol for T est- T ime C ompute (FEval-TTC), designed to ensure consistent assessment of test-time compute (TTC) methods, regardless of such fluctuations. FEval-TTC focuses on the evaluation of TTC methods that utilize underlying Chains-of-Thought (CoT). It supports evaluations across multiple LLMs on a diverse set of mathematical and commonsense reasoning datasets. The few-shot prompting and answer extraction processes are standardized across datasets, reducing both time and monetary overhead for researchers. Furthermore, we provide a cost modelling procedure that estimates both the token and dollar cost per query, facilitating equitable comparisons of prevalent TTC methods. We open-source FEval-TTC for public use at https://github.com/networkslab/feval_ttc . \workshoptitle Evaluating the Evolving LLM Lifecycle Benchmarks, Emergent Abilities, and Scaling Figure 1: FEval-TTC comprises of three distinct groups of datasets, each consisting of question–answer pairs (see Section 2.1 ). Each dataset is queried by multiple LLMs from different families with a standardized query format. We provide 40 sampled Chains-of-Thoughts (CoTs) with extracted answers, number of tokens, and the corresponding dollar cost of inference per question (see Section 2.2 ). 1 Introduction The emergence of System-2 thinking in Large Language Models (LLMs) ( Ji et al. 2025 ) has introduced a new paradigm that leverages inference-time computation to enhance reasoning capabilities ( Snell et al. 2025 ) . This paradigm involves allocating additional computational resources during inference, such as extended token generation, to improve performance on complex reasoning tasks ( Yang et al. 2025 ) . The additional computation may originate from a single LLM ( Snell et al. 2025 ) or from coordination among multiple LLMs ( Qi et al. 2025 ) . However, this increase in generation leads to substantial time and financial costs, posing practical challenges that hinder rapid experimentation and broader adoption. In case a researcher is using an API, the monetary cost is primarily determined by commercial API usage fees, which are typically charged by API providers based on token usage. For self-hosted LLMs, monetary costs arise from the electricity consumed to operate GPUs. The time costs in both cases arise primarily from the latency between initiating a request and receiving the response, since the LLM generates tokens sequentially. Concurrently, the LLM landscape evolves rapidly, with frequent model updates, new releases, and revisions to API pricing. Such volatility can undermine the validity of prior research or create unfair advantages for newer methods if experimental setups do not carefully control for differences in model performance and cost. Reusing results from published work without accounting for these changes can further exacerbate these issues, leading to inaccurate comparisons. This F air Eval uation protocol for T est- T ime C ompute ( FEval-TTC ) addresses these challenges by enabling researchers to substantially reduce both computational and time costs, while preserving fair and reproducible comparisons with prior work. FEval-TTC includes a comprehensive set of pre-recorded model queries and responses, along with extracted answers and associated metadata. For instance, applying self-consistency with 20 samples on the GSM8K dataset ( Cobbe et al. 2021a ) using Mixtral 8×22B can take up to seven hours due to inference latency. In contrast, FEval-TTC allows this evaluation to be completed in seconds by eliminating the need for live LLM calls. We provide standardized requests and responses for sixteen datasets covering both commonsense and mathematical reasoning tasks. Additionally, we introduce a unified cost model to ensure consistent and fair estimation of both query and response costs across different methods and models. The uniqueness of FEval-TTC lies in the following features: • It supports several groups of reasoning tasks and multiple LLM model families. • It is extensible in a trivial way to incorporate additional models, datasets, and prompting techniques. • It ensures a fair comparison of test-time algorithms by using a standardized set of LLM responses and a unified monetary/token cost model. • FEval-TTC significantly reduces the evaluation time and cost of common test-time inference methods by leveraging pre-recorded LLM responses instead of issuing live queries. 2 FEval-TTC package overview ⬇ from feval_ttc import load , DatasetType , LLMType dataset , [ llm1 , llm2 ] = load ( DatasetType . SVAMP , \ [ LLMType . LLaMA3B32 , LLMType . Qwen72B25 ]) for question_id , dataentry in dataset : print ( "Question: " , dataentry . question ) print ( "True answer: " , dataentry . answer ) llm1_response = llm1 ( question_id , N =20) print ( "1st CoT answer: " , llm1_response . cots [0]. answer ) print ( "Token cost: " , llm1_response . cots [0]. tokens ) print ( "USD Cost: " , llm1_response . cots [0]. dollar_cost ) Listing 1: Example of an interaction with the FEval-TTC package This section provides the architectural overview of the FEval-TTC. FEval-TTC is composed of two main parts: Dataset module and LLM module. The Dataset module holds the list of questions and answers and an interface to iterate over them (see Section 2.1 ). The LLM module stores multiple Chain-of-Thoughts (CoTs) responses along with their extracted answers. The design of both modules employs key-value dictionaries to facilitate seamless access to cached data. The main package features an interface to load a Dataset module instance and a set of corresponding LLM module instances. A usage example for research purposes is provided in Listing 1 . 2.1 Dataset module Dataset module instance contains a list of Dataentries (see Listing 2 ). Each Dataentry includes a question and its ground-truth answer, collected from the corresponding datasets. We did not change questions and answers, but the answer format was standardized across the package. For each dataset, we provide a system prompt that was used to obtain LLM responses. ⬇ class DatasetEntry ( BaseModel ): answer : str question : str class Dataset ( BaseModel ): data : List [ DatasetEntry ] datatype : DatasetType system_prompt : str Listing 2: Dataset module in FEval-TTC FEval-TTC features datasets from three different reasoning categories: commonsense reasoning, arithmetic reasoning, and mathematical reasoning. The commonsense reasoning group includes tasks designed to assess inference capabilities using commonsense knowledge, such as CommonSenseQA ( Talmor et al. 2019 ) and 11 BIG-Bench-Hard ( Suzgun et al. 2023 ) tasks. The arithmetic reasoning group contains datasets such as GSM8K ( Cobbe et al. 2021a ) , SVAMP ( Patel et al. 2021 ) , and AQuA ( Ling et al. 2017 ) , which require basic calculation skills. The mathematical reasoning category targets advanced problem-solving ability, represented by the MATH-500 ( Hendrycks et al. 2021 ) , which consists of competition-style mathematical questions requiring rigorous algebraic and geometric manipulation. 2.2 LLM module ⬇ class CoTMetadata ( BaseModel ): dollar_cost : float tokens : int class CoT ( BaseModel ): raw_text : str answer : Optional [ str ] metadata : CoTMetadata class LLMRequest ( BaseModel ): raw_text : str dollar_cost : float tokens : int class LLMResponse ( BaseModel ): cots : List [ CoT ] request : LLMRequest answers