J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain
Yiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L.A. Clarke, Weixing Shen · 2025
Read J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain by Yiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L.A. Clarke, Weixing Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18360.txt.
Description
Yiran Hu and co-authors probe whether large language models truly reason with legal knowledge, in a 2025 paper. Models are now used to judge legal tasks, yet they might lean on surface words rather than legal logic. If so, using them as judges could be risky.
The team built a test that injects mistakes into legal texts: typos, swapped legal terms, and wrongly retrieved statutes. Human experts tend to see past such slips and judge by logic. The models, both general and law-focused, were often misled by the planted errors. The authors also tried several fixes meant to harden the models, and compared how well each worked. Their framework gives the field a way to check whether legal models reason, or merely match patterns.
Catalog record
| CREATOR | Yiran Hu, Huanghai Liu, Qingjing Chen, Ning Zheng, Chong Wang, Yun Liu, Charles L.A. Clarke, Weixing Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18360