Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch

Katja Berčič, Slobodan Stanojevikj · 2026

Read Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch by Katja Berčič, Slobodan Stanojevikj (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28815.txt.

Introduction

Bercic and Stanojevikj describe Mathswitch, an open-source project gathering records of mathematical concepts from sources such as Wikidata, Wikipedia, MathWorld and several specialist encyclopedias. Records that name the same concept are linked, but each source keeps its own structure and wording. Because Wikidata is edited collaboratively and queried broadly, the imported set is noisy, with some items not mathematical at all and others ambiguous.

To clean the set, the pair test a voting ensemble of language model judges, meaning several models vote on whether each item truly belongs. They check the voters against items known to have MathWorld entries, and rerun the test with database identifiers hidden to see if labels sway the vote. Where judges disagreed with MathWorld, the cases fell into three groups, including thin descriptions and scope mismatches, each pointing to a different fix.

Catalog record

CREATORKatja Berčič, Slobodan Stanojevikj
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28815

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Categorizing Mathematical Concepts with LLM Voting Ensembles in Mathswitch Katja Berčič, Slobodan Stanojevikj Abstract Mathswitch is an open-source project that imports mathematical concept records from sources such as Wikidata, Wikipedia, MathWorld, Encyclopedia of Mathematics, nLab, ProofWiki, and Agda-Unimath, and links records that refer to the same concept. It does not reorganize or redefine the imported content; each source retains its own structure. The current focus is on importing high-quality concept data from Wikidata and the resources it links to, with plans to expand to further sources and better concept linking. Because the concept set is approximated through queries over Wikidata’s collaboratively edited graph, the imported data is noisy: some items are non-mathematical, while others are ambiguous. In this paper, we test whether a voting ensemble of LLM judges can filter this noise. We evaluate it on Wikidata items with known MathWorld identifiers as a positive control, and examine how classification changes when database identifiers are removed from context. We then inspect the cases where the judges disagree with MathWorld and group these disagreements into three categories (degenerate descriptions, narrow scope bias, and editorial-scope mismatches) that suggest different remediation strategies. 1 Introduction Mathematical concepts can be found in many kinds of resources. Encyclopedias such as MathWorld and the Encyclopedia of Mathematics provide curated reference entries. Collaborative platforms such as Wikipedia, nLab, and ProofWiki offer community-maintained expositions. Wikidata provides a structured knowledge graph, the LMFDB documents its objects through knowls, and libraries of formalized mathematics such as Mathlib, Mizar, and Agda-Unimath contain machine-checked definitions and proofs. Each resource uses its own vocabulary, level of formality, and organizational scheme, and a concept described in one source cannot be systematically located in another. Mathswitch is an open-source project that works toward connecting mathematical concepts across such resources. It imports concept records from several sources and groups records that appear to refer to the same idea. It does not reorganize or redefine the imported content; each source retains its own structure, and Mathswitch makes the connections between them explicit. The current focus is on importing high-quality concept data from Wikidata and the resources it links to, but the set of covered sources and the sophistication of concept linking are both intended to grow. Cross-referencing at this scale depends on the quality of the imported data. There is no authoritative registry of all mathematical concepts, so Mathswitch approximates the concept set through hand-tuned queries over Wikidata’s collaboratively edited graph. The resulting data is noisy: some items are clearly non-mathematical but match the query patterns, while others are genuinely ambiguous terms that carry both mathematical and non-mathematical meanings. This noise degrades the reliability of the cross-reference network. Mathswitch is an open-source web application, available at mathswitch.xyz , that serves pages about mathematical concepts, linking records across external mathematical resources. The landing page offers a search interface over the concept set (Figure 1 ), and each concept page collects links to Wikipedia, nLab, MathWorld, ProofWiki, the Encyclopedia of Mathematics, and the formal mathematics library Agda-Unimath for each concept (when these links are available), together with the Wikidata label and description (Figure 2 ). The current prototype is built around Wikidata and the identifiers it exposes. Figure 1: The Mathswitch landing page with a search box and a list of sources together with the number of concepts imported from each. Figure 2: The Mathswitch concept page for Stone duality. In this paper, we describe the Mathswitch data pipeline and evaluate an automated categorization approach based on a voting ensemble of LLM judges. The pipeline consists of three stages: 1. preparing the SPARQL queries (Section 3 ), 2. fetching data from Wikidata and Agda-Unimath (Section 4 ), 3. filtering concepts with the voting ensemble and building the concept network (Section 5 ). We evaluate the ensemble on Wikidata items with known MathWorld identifiers as a positive control and examine how classification changes when database identifiers are removed from context. As a negative control, we run the same ensemble on a sample of physics concepts from Wikidata. We then inspect the cases where the judges disagree with MathWorld and group them into three categories (Section 6.3 ). 2 Related work Several systems organize and connect mathematical knowledge across sources. Some build unified formal representations: OMDoc provides a markup language for mathematical documents [ 7 ] , the MMT framework generalizes this with a foundation-independent module system [ 13 ] , MathHub uses MMT for hosting and cross-referencing libraries [ 5 ] , and the Formal Abstracts project proposes stating theorems in a controlled formal language [ 2 ] . Gauthier and Kaliszyk [ 3 ] and Müller et al. [ 9 ] align concepts across proof assistant libraries without a common formalism. Wikidata is collaboratively edited, and its quality varies across topics [ 12 , 18 ] . Instance-of and subclass-of assignments are often incomplete or inconsistent, which affects downstream queries. For mathematical content, the MaRDI initiative builds a Wikibase-powered knowledge graph from sources such as DLMF, zbMATH, and arXiv [ 17 ] . Scharpf, Schubotz, and Gipp link mathematical formulae to Wikidata entities through the AnnoMathTeX system [ 15 , 16 ] . The Wikidata WikiProject Mathematics maintains curated property lists and modeling conventions [ 20 ] . The OntoMath - PRO ontology provides a curated vocabulary for mathematical concepts [ 1 ] . Work on Wikidata subsetting [ 4 , 11 , 21 ] shows that strict domain-based extraction is hard, since concepts do not separate cleanly into domains. Mathswitch faces a narrower version of this problem: given a candidate set from SPARQL queries, it needs to decide which items are mathematical concepts. Recent work uses large language models as automated evaluators, asking them to judge whether a given output satisfies a quality criterion or matches a reference answer. Zheng et al. show that strong language models approximate human preference judgments and document systematic biases (position, verbosity, self-preference) in single-model judges [ 22 ] . Sampling multiple responses from a single model and majority-voting over them improves reliability [ 19 ] , but does not address biases shared by all samples from the same model. Querying several different models goes further: LLM-Blender ranks and fuses outputs from a model pool [ 6 ] and Li et al. report accuracy gains from majority voting across multiple agents [ 8 ] . Most of this work targets open-ended generation and preference ranking. 3 Wikidata, SPARQL queries and their results Wikidata is a collaboratively edited knowledge graph. Its main building blocks are items , properties , and lexemes . Items represent entities and are identified by Q identifiers: for example, Q11348 is the mathematical concept function and Q395 is mathematics (as a field of study). Properties represent relations and are identified by P identifiers: P31 is instance of , P2579 is studied by , and P2812 is MathWorld identifier . Lexemes (L identifiers) represent words and phrases; Mathswitch does not use them. A statement (subject–predicate–object) links an item to a value through a property. For instance, the statement Q11348 P2579 Q395 means that function (s) are studied by mathematics . External identifiers such as P2812 link Wikidata items to records in other databases. Mathswitch queries this knowledge graph using SPARQL to form

Read the complete text