TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs

Duygu Nur Yaldiz, Yavuz Bakman, Sungmin Kang, Alperen Ozis, Hayrettin Eren Yildiz, Mitash Shah, Zhiqi Huang, Anoop Kumar, Alfy Samuel, Daben Liu, Sai Praneeth Karimireddy, Salman Avestimehr · 2025

Read TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs by Duygu Nur Yaldiz, Yavuz Bakman, Sungmin Kang, Alperen Ozis, Hayrettin Eren Yildiz, Mitash Shah, Zhiqi Huang, Anoop Kumar, Alfy Samuel, Daben Liu, Sai Praneeth Karimireddy, Salman Avestimehr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08203.txt.

Introduction

Duygu Yaldiz and co-authors release a toolbox for catching language models in a lie, in a 2025 paper. Big models state falsehoods with confidence, and knowing when an answer is likely untrue matters most where errors are costly.

Their open-source Python library, TruthTorchLM, gathers more than thirty truthfulness prediction methods under one interface. The methods range widely: some read the model's inner states, some need only its text, some lean on source documents, and some learn from labels. The library plugs into both locally hosted models and commercial application interfaces. By making the methods easy to swap and compare, the team hopes to speed research on honesty checks and put them within reach of ordinary builders.

Catalog record

CREATORDuygu Nur Yaldiz, Yavuz Bakman, Sungmin Kang, Alperen Ozis, Hayrettin Eren Yildiz, Mitash Shah, Zhiqi Huang, Anoop Kumar, Alfy Samuel, Daben Liu, Sai Praneeth Karimireddy, Salman Avestimehr
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08203

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs Duygu Nur Yaldiz, Yavuz Bakman, Sungmin Kang, Alperen Ozis, Hayrettin Eren Yildiz, Mitash Shah, Zhiqi Huang, Anoop Kumar, Alfy Samuel, Daben Liu, Sai Praneeth Karimireddy, Salman Avestimehr Abstract Generative Large Language Models (LLMs) inevitably produce untruthful responses. Accurately predicting the truthfulness of these outputs is critical, especially in high-stakes settings. To accelerate research in this domain and make truthfulness prediction methods more accessible, we introduce TruthTorchLM 1 1 1 https://github.com/Ybakman/TruthTorchLM , 2 2 2 https://www.youtube.com/watch?v=dgovBgUYz3w , 3 3 3 https://pypi.org/project/TruthTorchLM/ an open-source, comprehensive Python library featuring over 30 truthfulness prediction methods, which we refer to as Truth Methods . Unlike existing toolkits such as Guardrails guardrails-ai , which focus solely on document-grounded verification, or LM-Polygraph Fadeeva et al. 2023 , which is limited to uncertainty-based methods, TruthTorchLM offers a broad and extensible collection of techniques. These methods span diverse trade-offs in computational cost, access level (e.g., black-box vs. white-box), grounding document requirements, and supervision type (self-supervised or supervised). TruthTorchLM is seamlessly compatible with both HuggingFace and LiteLLM, enabling support for locally hosted and API-based models. It also provides a unified interface for generation, evaluation, calibration, and long-form truthfulness prediction, along with a flexible framework for extending the library with new methods. We conduct an evaluation of representative truth methods on three datasets, TriviaQA, GSM8K, and FactScore-Bio. 1 Introduction Generative Large Language Models (LLMs) have been widely adopted in many real-world applications due to their remarkable performance across a range of tasks, from code generation to conversational agents Band et al. 2021 . Despite these successes, LLMs inevitably produce outputs that are factually or logically incorrect, commonly referred to as hallucinations Ravi et al. 2024 . Detecting such untruthful outputs is particularly crucial in high-stakes applications where reliability and correctness are essential. In response, numerous methods have been proposed to assess the truthfulness of LLM-generated content to support reliable decision-making. These include uncertainty estimation techniques, agentic tool use, multi-LLM collaboration strategies, supervised classification models, and document-based verification approaches. Each method varies in terms of computational cost, required access to model internals, and reliance on external resources. As LLM usage continues to grow, developing new techniques and refining existing ones remains crucial, given that truthfulness is a core requirement for trustworthy language generation. Figure 1: Overview of TruthTorchLM functionalities. To support research in this domain, an open-source library that consolidates existing methods and offers a flexible development framework is essential. Current software tools only partially meet this need. For instance, Guardrails guardrails-ai is an open-source library that focuses on document-based guardrails to assess the truthfulness of LLM outputs. However, it lacks support for a broader range of methods that do not rely on external documents, such as uncertainty estimation techniques. Similarly, LM-Polygraph Fadeeva et al. 2023 provides implementations of uncertainty estimation methods, but it does not include supervised approaches, document-checking techniques, or tool-based strategies. A more comprehensive and extensible toolkit is needed to facilitate systematic evaluation and innovation across the full spectrum of truthfulness prediction methods. To facilitate research and address the limitations of existing software in the domain of truthfulness prediction, we introduce TruthTorchLM (TTLM), an open-source library that currently implements over 30 truth methods with diverse algorithmic ideas. The library provides an intuitive and extensible interface for integrating new methods, enabling researchers to prototype and evaluate novel approaches with ease. TruthTorchLM is seamlessly compatible with both HuggingFace Wolf et al. 2020 and LiteLLM BerriAI , two of the most widely used frameworks for deploying LLMs in local and hosted environments. Beyond method implementation, the library offers comprehensive evaluation tools for benchmarking performance and includes calibration utilities to produce more interpretable truthfulness scores. Importantly, TruthTorchLM supports the application of truth methods to long-form generations, where multiple factual claims may be present and each claim requires individual assessment. This long-form setting represents a challenging and underexplored area of research, which is currently underserved by existing libraries. Our contributions can be summarized as follows: 1. We release TruthTorchLM (TTLM) , an open-source library that implements over 30 truthfulness prediction methods, fully compatible with both HuggingFace and LiteLLM frameworks. 2. TruthTorchLM provides a unified interface for generation, evaluation, calibration, and long-form extension of existing truth methods, along with a flexible framework for adding new methods. 3. We conduct a comprehensive evaluation of a representative truth methods across three diverse datasets, TriviaQA, GSM8K, and FactScore-Bio, using both an open-weight model (LLaMA-3-8B) and a closed-weight model (GPT-4o-mini). 2 System Design and Features of TTLM TTLM is designed around a central abstraction: truth methods , which are methods for predicting the truthfulness of LLM-generated outputs. Using TTLM, users can generate responses for any input query and apply one or more truth methods to assess the reliability of these outputs, whether they are short-form answers containing a single claim or long-form responses with multiple factual assertions. In addition to prediction, TTLM enables users to evaluate and calibrate the outputs of truth methods with just a few lines of code. In the following sections, we detail each of TTLM’s core features and explain how they support robust and scalable research in truthfulness assessment. Table 1: Categorization of a representative subset of available methods in TruthTorchLM. Truth Methods Document-Grounding Supervised Access Level Sampling-Required LARS Yaldiz et al. 2025 ✗ ✓ Grey-box ✗ MARS Bakman et al. 2024 ✗ ✗ Grey-box ✗ SelfDetection Zhao et al. 2024 ✗ ✗ Black-box ✓ PTrue Kadavath et al. 2022 ✗ ✗ Grey-box ✗ AttentionScore Sriramanan et al. 2024 ✗ ✗ White-box ✗ CrossExamination Cohen et al. 2023 ✗ ✗ Black-box ✓ Eccentricity Lin et al. 2024 ✗ ✗ Black-box ✓ GoogleSearchCheck Chern et al. 2023 ✓ ✗ Black-box ✗ Inside Chen et al. 2024 ✗ ✗ White-box ✓ KernelLanguageEntropy Nikitin et al. 2024 ✗ ✗ Black-box ✓ MiniCheck Tang et al. 2024 ✓ ✗ Black-box ✗ Matrix-Degree Lin et al. 2024 ✗ ✗ Black-box ✓ SAPLMA Azaria and Mitchell 2023 ✗ ✓ White-box ✗ SemanticEntropy Kuhn et al. 2023 ✗ ✗ Grey-box ✓ MultiLLMCollab Feng et al. 2024 ✗ ✗ Black-box ✓ SAR Duan et al. 2024 ✗ ✗ Grey-box ✓ VerbalizedConfidence Tian et al. 2023 ✗ ✗ Black-box ✗ DirectionalEntailmentGraph Da et al. 2024 ✗ ✗ Black-box ✓ 2.1 Truth Methods Truth Methods are methods designed to estimate the truthfulness or correctness of an LLM’s response to a given query. These methods operate in an off-the-shelf manner, meaning they do not interfere with the generation process itself, but instead assign a post hoc truthfulness score (referred to as a truth value ) after the response has been produced. Each Truth Method can optionally define its own parameters and must implement a standardized forward function, which takes as input the relevant generation-time information, such as generated token ids, the LLM and tokenizer objects, and returns a truth value. All methods inherit from the TruthMethod base class and follow a consistent interface, making the library easily extensible for users to implement custom methods. Truthfulness estimation can be approached in a variety of ways, each with distinct trade-offs. The first major axis of variation is the use of external context: for example, Natural Language Inference (NLI) Lei et al. 2023 methods assess truthfulness relative to supporting

Read the complete text