HalluMat: Detecting Hallucinations in LLM-Generated Materials Science Content Through Multi-Stage Verification
Bhanu Prakash Vangala, Sajid Mahmud, Pawan Neupane, Joel Selvaraj, Jianlin Cheng · 2025
Read HalluMat: Detecting Hallucinations in LLM-Generated Materials Science Content Through Multi-Stage Verification by Bhanu Prakash Vangala, Sajid Mahmud, Pawan Neupane, Joel Selvaraj, Jianlin Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22396.txt.
Description
Bhanu Prakash Vangala and four co-authors build tools to catch language models fabricating materials science. Hallucinated facts in AI-generated research content threaten research integrity, and detection methods need benchmarks of their own. The team introduces HalluMatData, a benchmark dataset for grading hallucination spotting in materials science text, and HalluMatDetector, a multi-stage system that checks generated content several ways.
The detector combines intrinsic checking, lookup from multiple sources, contradiction graph analysis, and metric-based scoring. Hallucination levels varied widely across materials subdomains, with high-entropy alloy queries showing the most factual mismatch. The pipeline cut hallucination rates by about thirty percent versus standard model outputs. A new consistency score also measures how answers drift across reworded versions of the same question.
Catalog record
| CREATOR | Bhanu Prakash Vangala, Sajid Mahmud, Pawan Neupane, Joel Selvaraj, Jianlin Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-22396