Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers

Wooseok Seo Seungju Han Jaehun Jung, Benjamin Newman Seungwon Lim Seungbeen Lee Ximing Lu, Yejin Choi Youngjae Yu, [1ex] Yonsei University · 2025

Read Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers by Wooseok Seo Seungju Han Jaehun Jung, Benjamin Newman Seungwon Lim Seungbeen Lee Ximing Lu, Yejin Choi Youngjae Yu, [1ex] Yonsei University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13342.txt.

Description

Seo, Yu and their co-authors evaluate fact verification systems themselves, in work from 2025. Checking facts is vital for reliable language model use, yet the checkers are rarely checked with the same care.

The team tested many large models and one specialist verifier on examples from a wide set of fact-checking benchmarks. About a sixth of the data proved ambiguous or wrongly labeled, enough to shift model rankings. Big models with a few examples did surprisingly well and should be standard baselines, they argue. Small tuned verifiers still lag on hard reasoning cases, but synthetic multi-step training data helped them.

Catalog record

CREATORWooseok Seo Seungju Han Jaehun Jung, Benjamin Newman Seungwon Lim Seungbeen Lee Ximing Lu, Yejin Choi Youngjae Yu, [1ex] Yonsei University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13342

Browse the full library · About the Tower of Babel