SciTrust 2.0: A Comprehensive Framework for Evaluating Trustworthiness of Large Language Models in Scientific Applications CCS: Computing methodologies Natural language generation

Emily Herron, Junqi Yin, Feiyi Wang · 2025

Read SciTrust 2.0: A Comprehensive Framework for Evaluating Trustworthiness of Large Language Models in Scientific Applications CCS: Computing methodologies Natural language generation by Emily Herron, Junqi Yin, Feiyi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25908.txt.

Description

Emily Herron, Junqi Yin, and Feiyi Wang introduce a framework to judge whether language models can be trusted in science work. Their system, SciTrust 2.0, scores models on four sides: truthfulness, strength against adversarial attacks, scientific safety, and scientific ethics. The ethics part covers eight areas, including dual use research, where findings could help or harm, and bias.

Herron and the team build open ended truth tests through a checked tuning pipeline with expert review. They test seven models, four built for science and three general industry models. General models scored higher overall, with one mini model leading on truth and attack resistance, while science models showed clear gaps in logic and ethics.

Catalog record

CREATOREmily Herron, Junqi Yin, Feiyi Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25908

Browse the full library · About the Tower of Babel