When Tables Go Crazy: Evaluating Multimodal Models on French Financial Documents
Virginie Mouilleron, Théo lasnier, Djamé Seddah · 2026
Read When Tables Go Crazy: Evaluating Multimodal Models on French Financial Documents by Virginie Mouilleron, Théo lasnier, Djamé Seddah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10384.txt.
Description
Vision-language models handle many document tasks well, but their reliability in specialized, non-English finance documents is far less tested. A 2026 paper led by Virginie Mouilleron, with three co-authors, introduces Scribe Finance, the first multimodal benchmark for French finance documents. The dataset holds over twelve hundred expert-validated questions on text extraction, table reading, chart interpretation, and multi-turn reasoning. Six open-weight models, from eight to one hundred twenty-four billion parameters, face a language-model judge.
Models score strongly on text and table tasks, at about eighty-five to ninety percent. They struggle with charts, landing between about a third and three-fifths. Multi-turn dialogue exposes a sharper failure: early mistakes spread across turns, driving accuracy down to roughly half regardless of model size. Current models thus serve well-defined extraction tasks but stay brittle in interactive, multi-step financial analysis, the authors conclude.
Catalog record
| CREATOR | Virginie Mouilleron, Théo lasnier, Djamé Seddah |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-10384