Self-hosted Lecture-to-Quiz: Local LLM MCQ Generation with Deterministic Quality Control
Seine A. Shintani · 2026
Read Self-hosted Lecture-to-Quiz: Local LLM MCQ Generation with Deterministic Quality Control by Seine A. Shintani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08729.txt.
Introduction
Seine Shintani builds a self-hosted pipeline that turns lecture PDFs into multiple-choice quizzes with no outside service. A local model drafts questions, but the shipped banks are plain text with a full quality trace and need no model at run time. In a seed sweep over three short dummy lectures, all 120 accepted items pass hard checks, while a warning layer flags eight items for softer risks like doubled distractors.
Catalog record
| CREATOR | Seine A. Shintani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35639 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08729
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-hosted Lecture-to-Quiz: Local LLM MCQ Generation with Deterministic Quality Control Seine A. Shintani Abstract We present an end-to-end self-hosted (API-free) pipeline—API-free in the sense that lecture content is not sent to any external LLM service—that converts lecture PDFs into multiple-choice questions (MCQs) using a local LLM plus deterministic quality control (QC). The pipeline is designed for black-box minimization : LLMs may assist drafting, but the final released artifacts are plain-text question banks with an explicit QC trace and without any need to call an LLM at deployment time. We run a seed sweep on three short “dummy lectures” (information theory, thermodynamics, and statistical mechanics), collecting 15 runs × \times 8 questions = 120 accepted candidates (122 attempts total under bounded retries). All 120 accepted candidates satisfy hard QC checks (JSON schema conformance, a single marked correct option, and numeric/constant equivalence tests); however, the warning layer flags 8/120 items (spanning 8 runs) that expose residual quality risks such as duplicated distractors or missing rounding instructions. We report a warning taxonomy with concrete before → \rightarrow after fixes, and we release the final 24-question set (three lectures × \times 8 questions) as JSONL/CSV for Google Forms import (e.g., via Apps Script / API tooling) (included as ancillary files under anc/ ). Finally, we position the work through the AI to Learn (AI2L) rubric lens and argue that self-hosted MCQ generation with explicit QC supports privacy, accountability, and Green AI in educational workflows. 1 Introduction Large language models (LLMs) can draft explanations, examples, and questions that support learning. For assessment authoring, however, instructors often face constraints that make “prompt-and-publish” workflows unsafe: materials may be proprietary, the correctness of generated items must be auditable, and repeated use of large models can be costly in both time and energy. This paper focuses on one practical scenario: converting lecture PDFs into multiple-choice questions (MCQs) in a self-hosted (API-free) manner, meaning generation that does not transmit lecture content to external LLM services. Throughout, we use this narrow API-free definition; data residency depends on the execution environment (e.g., fully local vs. hosted notebooks). Using a self-hosted model reduces privacy risk, but local generation raises its own challenges: outputs can contain duplicated options, multiple correct answers, or numerically equivalent distractors that silently break the question. We present Self-hosted Lecture-to-Quiz (L2Q) , a local-LLM pipeline with an explicit verification layer. L2Q (i) extracts a compact topic plan from the lecture, (ii) generates MCQs in a strict JSON schema, and (iii) applies deterministic quality-control (QC) checks (duplicate detection, numeric/constant and sampled equivalence tests, and unique-correct verification) with bounded retries. The end product is a static question set that can be imported into systems such as Google Forms; learners do not need access to an LLM at deployment time. Positioning with AI2L (AI to Learn). AI2L is a set of human-centered guidelines that constrain generative AI to the learning-support stage and aim to eliminate AI-derived black-box components from final deliverables, emphasizing (1) model transparency, (2) accountability via human verification, (3) privacy protection, and (4) Green AI through “use big models only when needed” and deploy lightweight artifacts [ 12 ] . Our study is not a direct application of AI2L in the strict sense, because MCQ drafting still relies on a black-box LLM. Instead, we treat L2Q as a concrete case study of black-box minimization : we run locally to address privacy (pillar 3), we provide QC logs and a rubric-friendly workflow to support human accountability (pillar 2), and we export static quizzes that no longer require repeated LLM calls during learning activities (pillar 4). We discuss what is achieved and what remains unresolved with respect to pillar 1 (full transparency) in the Discussion (Table 8 ). Contributions. • We propose an end-to-end self-hosted (API-free) pipeline (L2Q) for converting lecture PDFs into deployable MCQs using a local LLM. • We integrate deterministic QC checks (duplicate detection, equivalence testing, unique-correct verification) with bounded retries to reduce structural failure modes. • We report a 15-run case study on an entropy lecture pack, producing a curated set of 24 deployable MCQs and summarizing runtime and QC statistics. • We position the workflow with respect to the AI2L guidelines, clarifying both the AI2L-compatible aspects (privacy, accountability, reduced runtime dependence) and the remaining black-box limitations. 2 Related Work Automatic question generation. Automatic question generation (AQG) has been studied for decades, ranging from rule- and template-based methods to neural approaches [ 5 , 3 ] . Benchmark datasets such as RACE [ 6 ] and SQuAD [ 11 ] helped standardize evaluation, although classroom deployment typically requires additional constraints (format stability, answer uniqueness, and pedagogical review). LLM-based MCQ generation workflows. With instruction-tuned LLMs, MCQ generation is increasingly treated as a workflow problem: prompting strategies, multi-stage generation, and filtering are combined with human review to reach deployable quality. Recent studies explore multi-stage prompting for MCQs [ 7 ] , reliability analyses and educator perspectives [ 2 ] , and cognitively diverse MCQ generation with psychometric-style evaluation [ 13 ] . Education-specific deployments also emphasize teacher revision loops (e.g., stepwise MCQ generation for English education) [ 14 ] . These works mainly optimize question quality given access to powerful models (often cloud-hosted). Our focus is complementary: we emphasize self-hosted (API-free) generation and deterministic QC gates, aiming to reduce privacy risk and to avoid a permanent black-box dependency during learning sessions. Structured outputs and verification. Reliable downstream use benefits from constrained decoding and post-hoc verification. Constrained decoding has a substantial prior literature (e.g., lexically constrained decoding) that improves format and content controllability during generation [ 8 ] . We employ llama.cpp grammar-constrained decoding to stabilize JSON outputs [ 4 ] and add deterministic checks for uniqueness and numeric/constant equivalence. This supports black-box minimization by turning model outputs into auditable artifacts, while acknowledging that the drafting model itself remains opaque. 3 Method: Self-hosted Lecture-to-Quiz (L2Q) Pipeline L2Q is designed as a deterministic outer loop around a generative model. The pipeline has five stages: 1. PDF ingestion and segmentation. Extract text from a lecture PDF and split into coherent chunks with page references. 2. Topic planning. Produce a short topic plan (e.g., key definitions and properties) to encourage coverage and reduce redundancy. 3. MCQ drafting. Generate MCQs with five options (A–E), a single correct option, and a brief explanation. 4. Automatic QC and bounded retries. Apply a suite of tests; if a test fails, re-prompt the model with targeted feedback up to a fixed retry budget. 5. Export. Emit JSONL/CSV suitable for LMS ingestion and Google Forms import (e.g., via Apps Script / API tooling). 3.1 Quality-control tests We implement QC as a two-level rule set: hard constraints (reject + retry) and warning flags (accept but log for human review). The hard checks target machine-verifiable failures that break MCQs in practice; warning checks capture “editorial” risks that are inexpensive for humans to fix. Hard QC (reject + retry). • Schema compliance. Output must be valid JSON with required keys and exactly five options (A–E). • Single-answer structure. Exactly one labeled correct option; the correct option must be in {A,…,E}. • De-duplication. Reject exact duplicates and near-duplicates (string similarity \geq 0.92 ) within a run. • Equivalence and uniqueness. For options that are numeric