FinNuE: Exposing the Risks of Using BERTScore for Numerical Semantic Evaluation in Finance Conference: CIKM 2025 Workshop on Advances in Financial AI: Innovations, Risk, and Responsibility in the Era of LLMs (Non-archival); November 14, 2025; Seoul, Korea CCS: Computing methodologies Lexical semanti

Yu-Shiang Huang, Yun-Yu Lee, Tzu-Hsin Chou, Che Lin, Chuan-Ju Wang · 2025

Read FinNuE: Exposing the Risks of Using BERTScore for Numerical Semantic Evaluation in Finance Conference: CIKM 2025 Workshop on Advances in Financial AI: Innovations, Risk, and Responsibility in the Era of LLMs (Non-archival); November 14, 2025; Seoul, Korea CCS: Computing methodologies Lexical semanti by Yu-Shiang Huang, Yun-Yu Lee, Tzu-Hsin Chou, Che Lin, Chuan-Ju Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09997.txt.

Description

Yu-Shiang Huang, Yun-Yu Lee, Tzu-Hsin Chou, Che Lin, and Chuan-Ju Wang expose a weakness in BERTScore, a popular metric that grades text similarity using embeddings. In finance, small number changes flip meaning entirely: a two percent gain and a twenty percent loss are opposites, not near-duplicates. The authors built FinNuE, a diagnostic dataset that perturbs numbers in controlled ways across earnings calls, filings, social posts, and news.

Tested on FinNuE, BERTScore repeatedly gave high similarity scores to text pairs whose financial meaning diverged sharply, showing low sensitivity to numerical variation. The authors argue this is a fundamental limit of embedding-based metrics for financial text. Their dataset and findings motivate evaluation frameworks that treat numbers as first-class meaning, not decoration.

Catalog record

CREATORYu-Shiang Huang, Yun-Yu Lee, Tzu-Hsin Chou, Che Lin, Chuan-Ju Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-09997

Browse the full library · About the Tower of Babel