Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflection
Jack Butler, Nikita Kozodoi, Zainab Afolabi, Brian Tyacke, Gaiar Baimuratov · 2025
Read Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflection by Jack Butler, Nikita Kozodoi, Zainab Afolabi, Brian Tyacke, Gaiar Baimuratov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20653.txt.
Introduction
Jack Butler and four co-authors compare two ways to get better answers from large language models at run time without retraining them. One path lets a model reflect on its own answer and try again, while the other simply gives it a larger budget of steps or tokens to work with.
Tests span maths reasoning and translation across model families from three major makers, under varied depths of reflection and budgets. Reflection helps most in maths, in some cases more than tripling scores, while results in translation depend more on the market and language pair. A live marketing system built for an online fashion shop confirms that gains differ by setting, so teams should test before they spend.
Catalog record
| CREATOR | Jack Butler, Nikita Kozodoi, Zainab Afolabi, Brian Tyacke, Gaiar Baimuratov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Finding the Sweet Spot: Trading Quality, Cost, and Speed During Inference-Time LLM Reflection Jack Butler, Nikita Kozodoi, Zainab Afolabi, Brian Tyacke, Gaiar Baimuratov Abstract As Large Language Models (LLMs) continue to evolve, practitioners face increasing options for enhancing inference-time performance without model retraining, including budget tuning and multi-step techniques like self-reflection. While these methods improve output quality, they create complex trade-offs among accuracy, cost, and latency that remain poorly understood across different domains. This paper systematically compares self-reflection and budget tuning across mathematical reasoning and translation tasks. We evaluate prominent LLMs, including Anthropic Claude, Amazon Nova, and Mistral families, along with other models under varying reflection depths and compute budgets to derive Pareto optimal performance frontiers. Our analysis reveals substantial domain dependent variation in self-reflection effectiveness, with performance gains up to 220% in mathematical reasoning. We further investigate how reflection round depth and feedback mechanism quality influence performance across model families. To validate our findings in a real-world setting, we deploy a self-reflection enhanced marketing content localisation system at Lounge by Zalando, where it shows market-dependent effectiveness, reinforcing the importance of domain specific evaluation when deploying these techniques. Our results provide actionable guidance for selecting optimal inference strategies given specific domains and resource constraints. We open source our self-reflection implementation for reproducibility at https://github.com/aws-samples/sample-genai-reflection-for-bedrock . \workshoptitle Efficient Reasoning 1 Introduction The deployment of large language models (LLMs) in production systems has created new challenges for practitioners seeking to optimise performance under real-world constraints. Recent advances in inference-time computation scaling Snell et al. 2025 offer promising solutions, allowing deployed systems to dynamically allocate computational resources based on task difficulty, available budget, and latency requirements without model retraining. Two established approaches for adjusting inference-time resources are multi-step inference and budget tuning. Budget tuning, available for select LLMs such as Anthropic Claude 3.7 Sonnet and OpenAI o1, enables users to configure inference parameters like maximum thinking tokens or reasoning tiers (e.g., low or high), allocating greater computational effort to more challenging inputs. Multi-step approaches such as self-reflection Madaan et al. 2023 are model-agnostic and involve prompting a model to revise its responses through sequential follow-up calls to the model. Prior work demonstrates that self-reflection improves LLM performance in tasks with clearly defined evaluation criteria Madaan et al. 2023 and structured domains with informative feedback signals, such as programming or math Chen et al. 2024 . For instance, executing generated code and providing the outputs back to the LLM as context creates concrete feedback signals for accurate self-reflection. However, most production applications such as translation or classification involve more ambiguous objectives and weaker feedback signals. The effectiveness of self-reflection on such tasks remains underexplored, making it unclear whether additional inference-time computation consistently yields performance gains across diverse domains. This uncertainty is compounded by the rapidly evolving LLM landscape, where practitioners navigate dozens of model options across providers, each with different capabilities, pricing structures, and inference features. Production teams frequently face critical decisions: Should they deploy a smaller, cost-effective model with advanced inference strategies, or invest in larger models with simpler inference pipelines? How do these choices impact not just accuracy, but operational costs, latency requirements, and system reliability? This paper addresses these practical deployment challenges through comprehensive evaluation of inference-time optimisation strategies across real-world applications. We make three primary contributions. First, we benchmark self-reflection and budget tuning across multiple LLMs and application domains including mathematical reasoning, text-to-SQL generation, sentiment classification, and translation. The derived Pareto-optimal frontiers illustrate accuracy-latency trade-offs of different strategies and provide actionable recommendations for selecting a suitable inference method based on domain-specific requirements, resource constraints, and the base model. Second, we analyse self-reflection trajectories across different LLMs and feedback mechanisms, revealing how reflection depth and feedback quality critically influence self-reflection performance. To the best of our knowledge, this work presents the first direct performance comparison between these two approaches. Third, we demonstrate our findings through a production deployment at Lounge by Zalando, where we implemented self-reflection for marketing content localisation across 17 European markets. 2 Related Work 2.1 Inference-Time Compute LLMs such as Anthropic’s Claude family Anthropic 2024 have been increasing in size over recent years, which has brought profound improvements in performance across a wide range of applications while simultaneously increasing training time and compute requirements Hoffmann et al. 2022 . Inference-time compute optimisation techniques avoid modifying the pre-trained model and instead enable dynamic allocation of computational resources depending on the specific requirements of each input. This offers the ability to tune performance according to task demands, scaling performance at inference time Snell et al. 2025 . One of the prominent inference optimisation approaches is self-reflection Madaan et al. 2023 , which performs sequential follow-up LLM calls, allowing it to revise initial responses. Other studies have explored drawing parallel samples from the LLM and implementing sampling and verification procedures such as tree-of-thoughts Yao et al. 2023 and graph-of-thoughts Besta et al. 2024 . Recent work has also leveraged techniques such as temporary fine-tuning Akyürek et al. 2025 and nearest neighbour retrieval-based fine-tuning Hübotter et al. 2025 where model’s parameters are temporarily updated during inference. In this paper, we explore trade-offs between two established inference strategies: model-agnostic self-reflection Madaan et al. 2023 and built-in reasoning capabilities exposed through model provider APIs. Our goal is to provide insights for practitioners who lack resources to conduct large-scale per-sample fine-tuning or complex multi-step inference processes with feedback loops. 2.2 LLM Post-Training Another research area aimed at LLM reasoning capabilities is the use of reinforcement learning on specialised reasoning datasets. These approaches implement reinforcement learning with access to either outcome supervision Trung et al. 2024 ; Xi et al. 2024 , step-by-step process supervision Lightman et al. 2023 ; Setlur et al. 2025 or LLM-driven feedback mechanisms Lee et al. 2024 . Reasoning models are commonly deployed via API interfaces with configuration settings that allow users to adjust computational resources allocated per sample (e.g. Anthropic Claude 3.7 Sonnet thinking tokens budget). Crucially, the reasoning in these models happens as internal processing tokens, and discrete proposed solutions are not validated using external feedback during generation. 2.3 LLM Evaluation As LLMs have become more capable and businesses increasingly incorporate them into their products and services, robust evaluation frameworks have become essential. Various evaluation platforms exist across different domains, such as HELM Liang et al. 2023 and Chatbot Arena Chiang et al. 2024 , where models undergo evaluation and receive automated or human feedback scores depending on the task and domain. While these platforms provide standardised evaluation of different LLMs, including base models and fine-tuned or quantised variants, there is a lack of comparable evaluation frameworks for inference techniques. This gap makes it challenging for practitioners to understand and navigate trade-offs when combining LLMs with various inference-time compute methods. Throughout our experimentation, we demonstrate these trade-offs across model families, task domains, and inference budgets. 3 Experimental Setup 3.1 Datasets The empirical study splits into two stages: testing on established benchmarks followed by evaluation on real-world deployment application. First, we perform experiments across four distinct domains using established benchmarks: • Flores-200 (Translation) Costa-Jussà et al. 2022 : Multilingual translation benchmark spanning 200 languages, allowing assessment of cross-lingual capabilities. • Math500 (Mathematical reasoning) Lightman et al. 2024 : Dataset with