Prober.ai: Gated Inquiry-Based Feedback via LLM-Constrained Personas for Argumentative Writing Development
Ran Bi, Shiyao Wei, Yuanyiyi Zhou, [4pt] Florida State University · 2026
Read Prober.ai: Gated Inquiry-Based Feedback via LLM-Constrained Personas for Argumentative Writing Development by Ran Bi, Shiyao Wei, Yuanyiyi Zhou, [4pt] Florida State University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05598.txt.
Introduction
AI tutors that write essays for students can quietly erode the thinking the essay was meant to build. Ran Bi, Shiyao Wei, and Yuanyiyi Zhou present Prober.ai, a writing environment that refuses to generate text. It constrains a language model, through personas and structured output formats, to ask pointed questions about weak arguments instead.
Suggestions unlock only after the student reflects, a deliberate friction in the design. The approach draws on Toulmin's argumentation theory and research on questioning as feedback. A working prototype was built in thirty-six hours at a New York education hackathon, where it took second place.
Catalog record
| CREATOR | Ran Bi, Shiyao Wei, Yuanyiyi Zhou, [4pt] Florida State University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05598
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prober.ai: Gated Inquiry-Based Feedback via LLM-Constrained Personas for Argumentative Writing Development Ran Bi, Shiyao Wei, Yuanyiyi Zhou, [4pt] Florida State University Abstract The proliferation of large language models (LLMs) in educational settings has paradoxically undermined the cognitive processes they purport to support. Students increasingly outsource critical thinking to AI assistants that generate polished text on demand, resulting in measurable cognitive debt and diminished argumentative reasoning skills. We present Prober.ai , a web-based writing environment that inverts the conventional AI-tutoring paradigm: rather than generating or rewriting student text, the system constrains an LLM (Gemini 3 Flash Preview) through persona-specific system prompts and structured JSON output schemas to produce only targeted, inquiry-based questions about argumentative weaknesses. A two-phase interaction architecture— Challenge and Unlock —implements a pedagogical friction mechanism whereby revision suggestions are gated behind mandatory student reflection. The system’s design is grounded in Toulmin’s argumentation theory, research on peer feedforward questioning mechanisms, and evidence on AI-supported feedback in writing instruction. A functional prototype was developed in 36 hours during the NY EdTech Hackathon (March 2026), where it was awarded second place. We describe the system architecture, the prompt engineering methodology for constraining LLM output to pedagogically aligned JSON schemas, and discuss implications for scalable, cognition-preserving AI integration in writing education. Keywords: AI-assisted writing feedback, argumentative writing, inquiry-based learning, LLM prompt constraints, cognitive scaffolding, educational technology 1 Introduction The rapid integration of large language models (LLMs) into student writing workflows has created a pedagogical paradox. Tools such as ChatGPT, Gemini, and QuillBot offer immediate, fluent text generation and revision capabilities that students readily adopt. However, emerging neuroscience and behavioral evidence demonstrates that this convenience comes at a significant cognitive cost. Kosmyna et al. Kosmyna et al. 2025 measured EEG alpha-band activity during LLM-assisted essay writing and found statistically significant reductions in directed transfer function (dDTF) connectivity—a neural marker of active cognitive engagement—compared to both search-assisted and unassisted writing conditions. This phenomenon, characterized as cognitive outsourcing , describes the systematic offloading of higher-order thinking processes to AI systems. The consequences extend beyond neural metrics. Behaviorally, students who rely on AI-generated text produce essays that are superficially polished but structurally shallow: claims are asserted without adequate warrant, counterarguments are treated as perfunctory acknowledgments rather than genuine dialectical engagements, and the reasoning chains linking evidence to conclusions are frequently absent or circular Bi and Yan 2026 . Existing writing support tools exacerbate rather than address this problem. Grammar-focused platforms (Grammarly, QuillBot) operate at the surface level—correcting syntax, word choice, and tone—without engaging the argumentative structure of the text. General-purpose AI agents (ChatGPT, Gemini) provide direct answers, are systematically agreeable (“Sounds great! You have built a really strong, cohesive argument”), and never deliver the rigorous, challenging feedback that strengthens critical thinking. This paper presents Prober.ai , a system designed to occupy a fundamentally different position in the AI-assisted writing landscape. The core design principle is that the AI should never write for the student . Instead, the system constrains an LLM to function exclusively as a structured questioner—a “devil’s advocate” that identifies logical weaknesses in the student’s argumentative essay and poses targeted, open-ended questions that force the student to defend, clarify, and strengthen their own reasoning. Concrete revision suggestions are deliberately gated behind a mandatory reflection step: the student must first articulate a written defense of their argument before the system unlocks specific, actionable feedback. The technical contributions of this work are threefold: 1. Persona-constrained LLM output. We demonstrate a methodology for constraining a general-purpose LLM (Gemini 3 Flash Preview) to produce only pedagogically aligned, structured JSON output through carefully engineered system prompts and explicit output schema specifications, eliminating the model’s default tendency toward evaluative or generative responses. 2. Gated feedback architecture. We introduce a two-phase API design ( /challenge \rightarrow /unlock ) that implements pedagogical friction as a first-class architectural primitive, ensuring that cognitive effort precedes the delivery of revision support. 3. Multi-persona questioning framework. We operationalize two complementary critical personas— Reviewer #2 (expert-level logical scrutiny) and Confused Reader (novice-perspective clarity probing)—each producing distinct question taxonomies mapped to specific dimensions of argumentation quality. 2 Related Work 2.1 AI-Assisted Feedback in Education Ba et al. Ba et al. 2025 conducted a systematic literature review of AI-assisted feedback in education, identifying that the majority of existing systems provide directive feedback (explicit corrections and rewrites) rather than facilitative feedback (questions and prompts that guide self-regulation). Their meta-analysis revealed that facilitative feedback mechanisms are more strongly associated with long-term learning gains, particularly in writing domains where the development of metacognitive awareness is a primary instructional goal. Prober.ai is explicitly designed as a facilitative feedback system, producing only questions in its initial interaction phase and gating directive suggestions behind student reflection. 2.2 Argumentation Theory and Writing Assessment The question taxonomy employed by Prober.ai is grounded in Toulmin’s model of argumentation Kinnear et al. 2022 , which decomposes arguments into claims, data (evidence), warrants (reasoning links), backing, qualifiers, and rebuttals. Kinnear et al. Kinnear et al. 2022 demonstrated how this framework can inform assessment validity in educational settings, providing a principled basis for identifying specific argumentative weaknesses. Our system operationalizes Toulmin’s categories as distinct question modules: the claim question targets the clarity and precision of the central thesis, the reasoning question probes the warrant linking evidence to conclusion, the counterargument question examines the depth of dialectical engagement, and the scope/implication question addresses qualifiers and broader stakes. 2.3 Peer Feedback and Question-Based Scaffolding Latifi et al. Latifi et al. 2021 investigated the distinction between peer feedback and peer feedforward in argumentative writing, finding that question-based feedforward—where reviewers pose questions rather than make evaluative statements—significantly enhanced both the quality of argumentation and the depth of the learning process compared to traditional feedback approaches. Their work demonstrated that questions activate different cognitive processes than statements: questions require the writer to generate rather than merely evaluate, shifting the locus of cognitive effort from the reviewer to the writer. Prober.ai extends this principle by replacing the human peer reviewer with a persona-constrained LLM, enabling on-demand, scalable question-based feedforward. Noroozi et al. Noroozi et al. 2016 further established that scripted online peer feedback processes—where the feedback interaction is structured through predefined protocols—produce higher-quality argumentative essays than unscripted interactions. The structured JSON output schemas employed in Prober.ai serve an analogous function: they script the LLM’s feedback behavior according to a pedagogically principled protocol, ensuring consistency and alignment with argumentation quality dimensions. 2.4 Cognitive Outsourcing and AI Dependency Kosmyna et al. Kosmyna et al. 2025 provided the first neuroimaging evidence of cognitive debt accumulation during LLM-assisted writing. Their EEG study demonstrated that using ChatGPT for essay writing produced significantly lower alpha-band dDTF connectivity—indicating reduced active cognitive processing—compared to both search-engine-assisted and brain-only conditions. Critically, this effect persisted even when participants were instructed to use the AI as a “thinking partner” rather than a ghostwriter, suggesting that the mere availability of generated text suppresses independent reasoning. Gao et al. Gao et al. 2024 extended this analysis to peer feedback contexts, finding that students’ uptake of online peer feedback in argumentative essay writing was mediated by the cognitive effort required to process and integrate the feedback. These findings collectively motivate Prober.ai ’s core design decision: by refusing to generate or rewrite text,