SciTrust 2.0: A Comprehensive Framework for Evaluating Trustworthiness of Large Language Models in Scientific Applications CCS: Computing methodologies Natural language generation
Emily Herron, Junqi Yin, Feiyi Wang · 2025
Read SciTrust 2.0: A Comprehensive Framework for Evaluating Trustworthiness of Large Language Models in Scientific Applications CCS: Computing methodologies Natural language generation by Emily Herron, Junqi Yin, Feiyi Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25908.txt.
Description
Emily Herron, Junqi Yin, and Feiyi Wang introduce a framework to judge whether language models can be trusted in science work. Their system, SciTrust 2.0, scores models on four sides: truthfulness, strength against adversarial attacks, scientific safety, and scientific ethics. The ethics part covers eight areas, including dual use research, where findings could help or harm, and bias.
Herron and the team build open ended truth tests through a checked tuning pipeline with expert review. They test seven models, four built for science and three general industry models. General models scored higher overall, with one mini model leading on truth and attack resistance, while science models showed clear gaps in logic and ethics.
Catalog record
| CREATOR | Emily Herron, Junqi Yin, Feiyi Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25908