When Tables Go Crazy: Evaluating Multimodal Models on French Financial Documents

Virginie Mouilleron, Théo lasnier, Djamé Seddah · 2026

Read When Tables Go Crazy: Evaluating Multimodal Models on French Financial Documents by Virginie Mouilleron, Théo lasnier, Djamé Seddah (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10384.txt.

Description

Vision-language models handle many document tasks well, but their reliability in specialized, non-English finance documents is far less tested. A 2026 paper led by Virginie Mouilleron, with three co-authors, introduces Scribe Finance, the first multimodal benchmark for French finance documents. The dataset holds over twelve hundred expert-validated questions on text extraction, table reading, chart interpretation, and multi-turn reasoning. Six open-weight models, from eight to one hundred twenty-four billion parameters, face a language-model judge.

Models score strongly on text and table tasks, at about eighty-five to ninety percent. They struggle with charts, landing between about a third and three-fifths. Multi-turn dialogue exposes a sharper failure: early mistakes spread across turns, driving accuracy down to roughly half regardless of model size. Current models thus serve well-defined extraction tasks but stay brittle in interactive, multi-step financial analysis, the authors conclude.

Catalog record

CREATORVirginie Mouilleron, Théo lasnier, Djamé Seddah
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-10384

Browse the full library · About the Tower of Babel