NoTeS-Bank : Benchmarking Neural Transcription and Search for Scientific Notes Understanding
Aniket Pal, Sanket Biswas, Alloy Das, Ayush Lodh, Priyanka Banerjee, Soumitri Chattopadhyay, Dimosthenis Karatzas, Josep Lladós, C.V. Jawahar · 2025
Read NoTeS-Bank : Benchmarking Neural Transcription and Search for Scientific Notes Understanding by Aniket Pal, Sanket Biswas, Alloy Das, Ayush Lodh, Priyanka Banerjee, Soumitri Chattopadhyay, Dimosthenis Karatzas, Josep Lladós, C.V. Jawahar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-09249.txt.
Introduction
Aniket Pal and co-authors built a test set in 2025 for how well AI can read and search handwritten study notes. Notes mix equations, sketches, and loose handwriting, which breaks tools built for printed pages. Their benchmark, NoTeS-Bank, gathers hard notes from many fields.
Two tasks anchor the set: answer questions with boxed proof from the notes, and first guess the field, then fetch the right page and answer. Scoring covers ranking quality, recall, box overlap, and answer match. Top vision-language models and search tools were run against the set, and their transcription and reasoning gaps showed plainly. The authors offer the benchmark as a stiff new yardstick for document AI. It pushes tools toward the messy notes students really keep.
Catalog record
| CREATOR | Aniket Pal, Sanket Biswas, Alloy Das, Ayush Lodh, Priyanka Banerjee, Soumitri Chattopadhyay, Dimosthenis Karatzas, Josep Lladós, C.V. Jawahar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-09249
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NoTeS-Bank : Benchmarking Neural Transcription and Search for Scientific Notes Understanding Aniket Pal, Sanket Biswas, Alloy Das, Ayush Lodh, Priyanka Banerjee, Soumitri Chattopadhyay, Dimosthenis Karatzas, Josep Lladós, C.V. Jawahar Abstract . Understanding and reasoning over academic handwritten notes remains a challenge in document AI, particularly for mathematical equations, diagrams, and scientific notations. Existing visual question answering (VQA) benchmarks focus on printed or structured handwritten text, limiting generalization to real-world note-taking. To address this, we introduce NoTeS-Bank , an evaluation benchmark for Neural Transcription and Search in note-based question answering. NoTeS-Bank comprises complex notes across multiple domains, requiring models to process unstructured and multimodal content. The benchmark defines two tasks: (1) Evidence-Based VQA , where models retrieve localized answers with bounding-box evidence, and (2) Open-Domain VQA , where models classify the domain before retrieving relevant documents and answers. Unlike classical Document VQA datasets relying on optical character recognition (OCR) and structured data, NoTeS-BANK demands vision-language fusion, retrieval, and multimodal reasoning. We benchmark state-of-the-art Vision-Language Models (VLMs) and retrieval frameworks, exposing structured transcription and reasoning limitations. NoTeS-Bank provides a rigorous evaluation with NDCG@5, MRR, Recall@K, IoU, and ANLS , establishing a new standard for visual document understanding and reasoning. Figure 1. Comparison of OCRs on a challenging handwritten scientific note sample. Powerful commercial OCR engines (eg. Textract, Google-OCR) fail to accurately transcribe the content, often losing mathematical symbols, structure, and semantic meaning. In contrast, open-source OCRs (e.g. Nougat, Got 2.0, OLM-OCR) struggle to extract even the textual content for the full document. This highlights the necessity for multimodal reasoning beyond OCR in handwritten document understanding. 1. Introduction “What we know is a drop, what we do not know is an ocean.” – Isaac Newton. The pursuit of knowledge often begins with handwritten notes — scribbled equations, diagrams, and annotations that serve as the foundation of scientific discovery, engineering breakthroughs, and academic learning. However, despite the fundamental role of handwritten notes, their automated understanding remains a formidable challenge in visual document understanding (VDU). Prior datasets ( Mathew et al. 2021 ) cover either neatly rendered handwriting (HW-SQuAD) or historical letters (BenthamQA), but none address modern lecture or notebook pages with complex content. Academic notes often mix prose with formulas, contain shorthand or abbreviations, and include sketches, flowcharts, or diagrams drawn by the note-taker. Current state-of-the-art (SOTA) OCR-based VDU models ( Xu et al. 2020 ; Huang et al. 2022 ; Appalaraju et al. 2021 ) assume a well-defined layout or reading order as in DocVQA ( Mathew et al. 2020 ) . They can falter when content is scattered or non-linear. For instance, models in such datasets ( Mathew et al. 2020 ; Van Landeghem et al. 2023b ) performed poorly on questions where the answer required interpreting the document’s structure (tables, columns, or aligned layout elements). In handwritten lecture notes, the “layout” might include diagrams next to text or equations below explanatory text, which is not trivial for models to parse, as shown in Figure 1 . Most existing models are not equipped to jointly analyze visual drawings and text. They either ignore non-text elements or treat the problem as pure text extraction or OCR tasks. This is a limitation when notes contain sketches, scientific diagrams, or mathematical equations. For example, a flowchart or a geometric diagram might be essential to answer a question, but generic text-based VQA models will not interpret a drawing of a triangle or a circuit diagram. Similarly, handwritten equations or symbols (e.g. \Sigma , integrals, chemical structures) can be misread by OCR or not understood in context (e.g., distinguishing a handwritten “z” from a “2” in an equation as shown in Figure 1 ). Existing benchmarks like InfographicVQA ( Mathew et al. 2022 ) and MathVista ( Lu et al. 2023 ) explicitly show that joint reasoning over text and graphics is needed, and current SOTA baselines perform modestly on such tasks. In summary, today’s VQA systems tend to be brittle outside the domains they were trained in - they struggle with messy handwriting, unstructured layouts, and mixed modalities prevalent in note-based documents. The information is there—somewhere—but buried beneath uneven ink, equations in the margins, half-drawn diagrams, and scribbled corrections. To answer a question, they don’t just recall the fact—they search, they scan, they locate the exact line, the boxed formula, the circled term. That process of not only knowing the answer but being able to localize to it is at the heart of our evidence-based VQA task. Evidence-based document VQA challenges models to look at the document like a human would, navigating through visual cues, spatial arrangements, and noisy handwriting to retrieve and ground the answer as shown in Figure 2 . This isn’t just about accuracy—it’s about understanding. Grounding the response builds trust, explains reasoning, and proves that the model isn’t just parroting patterns but truly reading the document. In our benchmark, this grounding becomes the litmus test: can your model not only answer the question but also show its work? The key challenge is to answer questions by identifying the correct region inside a visual academic note (via bounding box), while also classifying both the local content type (e.g. flowchart, chemical formulae, graph, table, math equation, etc.) and the global domain (e.g., physics, computer science) of the answer—encouraging a full spectrum of document understanding, from retrieval to reasoning. Now, picture a student preparing for an oral exam, trying to answer a complex, open-ended question like, " Why does underflow occur in floating point representation? " The answer probably lives somewhere in a series of lecture notes, in different subjects, on different days. To respond, the student first narrows it down: this sounds like computer science or numerical methods. Then, they dive into their notes, flipping through the relevant sections and scanning for definitions, diagrams, or formulas. Only then do they piece together an answer—sometimes from multiple fragments, maybe with a margin sketch or an annotated equation to help. This is the essence of open-domain question answering over academic notes. But unlike vanilla open-domain QA systems ( Marino et al. 2019 ) or M3DocVQA ( Cho et al. 2024 ) that pull structured facts from curated sources like Wikipedia, our task demands that the model acts like a student, knowing where to look, what to extract, and how to connect the dots visually and semantically. The challenge isn’t just to answer but to find a filter and reason across noisy, visual content. In our setup, models must do three things: first, predict the domain (e.g., physics, mathematics, chemistry) from the question to guide retrieval; second, retrieve relevant pages of handwritten notes—not structured web articles or knowledge bases; and third, perform multimodal reasoning across diagrams, equations, and prose, often scattered across the page. Figure 2. Illustrative examples from the NoTeS-Bank showing diverse local reasoning categories such as equations, flowcharts, structural formulas, and textual answers. Center: Distribution of question types across eight task-level categories. Surrounding: Sample questions and annotated evidence regions from handwritten notes, highlighting the visual and semantic complexity handled in the Evidence-Based VQA task. Though significant advancements have been witnessed in recent times with the emergence of Vision-Language Models (VLMs) ( Laurençon et al. 2024 ; Wang et al. 2024 ) and retrieval-augmented approaches (