J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain

Yiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L.A. Clarke, Weixing Shen · 2025

Read J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain by Yiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L.A. Clarke, Weixing Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18360.txt.

Description

Yiran Hu and co-authors probe whether large language models truly reason with legal knowledge, in a 2025 paper. Models are now used to judge legal tasks, yet they might lean on surface words rather than legal logic. If so, using them as judges could be risky.

The team built a test that injects mistakes into legal texts: typos, swapped legal terms, and wrongly retrieved statutes. Human experts tend to see past such slips and judge by logic. The models, both general and law-focused, were often misled by the planted errors. The authors also tried several fixes meant to harden the models, and compared how well each worked. Their framework gives the field a way to check whether legal models reason, or merely match patterns.

Catalog record

CREATORYiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L.A. Clarke, Weixing Shen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18360

Browse the full library · About the Tower of Babel