METIS: Mentoring Engine for Thoughtful Inquiry & Solutions
Abhinav Rajeev Kumar, Dhruv Trehan, Paras Chopra · 2026
Read METIS: Mentoring Engine for Thoughtful Inquiry & Solutions by Abhinav Rajeev Kumar, Dhruv Trehan, Paras Chopra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13075.txt.
Introduction
Abhinav Rajeev Kumar, Dhruv Trehan, and Paras Chopra ask if an AI guide can coach young scholars. Many students have no access to an expert guide for research. The team asks if a smart tool can move an undergraduate from a rough idea to a full paper.
They built a tool named METIS for that task. It is aware of the stage of work and can call tools as it helps. Its parts include search of past work, picked guides, checks of method, and a memory of past steps. The team tests METIS head to head with two strong chat models, GPT-5 and Claude Sonnet 4. The study probes where an AI guide helps most and where a human guide still counts.
Catalog record
| CREATOR | Abhinav Rajeev Kumar, Dhruv Trehan, Paras Chopra |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13075
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
METIS: Mentoring Engine for Thoughtful Inquiry & Solutions Abhinav Rajeev Kumar, Dhruv Trehan, Paras Chopra Abstract Many students lack access to expert research mentorship. We ask whether an AI mentor can move undergraduates from an idea to a paper. We build METIS , a tool‑augmented, stage‑aware assistant with literature search, curated guidelines, methodology checks, and memory. We evaluate METIS against GPT‑5 and Claude Sonnet 4.5 across six writing stages using LLM‑as‑a‑judge pairwise preferences, student‑persona rubrics, short multi‑turn tutoring, and evidence/compliance checks. On 90 single‑turn prompts, LLM judges preferred METIS to Claude Sonnet 4.5 in 71% and to GPT‑5 in 54%. Student scores (clarity/actionability/constraint‑fit; 90 prompts × \times 3 judges) are higher across stages. In multi‑turn sessions (five scenarios/agent), METIS yields slightly higher final quality than GPT‑5. Gains concentrate in document‑grounded stages (D–F), consistent with stage‑aware routing and grounding; failure modes include premature tool routing, shallow grounding, and occasional stage misclassification. Introduction Large language models can explain, suggest, and critique. But can one mentor a student or early-career researcher from an initial, rough idea to a publishable paper? We study this question with METIS , an AI research mentor whose primary goal is to guide undergraduates from ideation to a publishable conference paper. METIS grounds its advice in conference instructions, research guides, and literature (arXiv/OpenReview) and keeps context across sessions so learners can make steady progress over weeks. This paper contributes: 1. a practical mentoring workflow and stage‑aware evaluation that turns ”research mentorship” into concrete tasks; 2. a simple, inspectable system with tools for literature search, guidelines retrieval, methodology checks, and memory; 3. an empirical comparison to GPT‑5 and Claude Sonnet 4.5 on single‑turn judgments and multi‑turn tutoring; and 4. open materials like prompts, logs, and scripts, to reproduce results. Related Work Tool‑augmented assistants combine reasoning with external tools (search, calculators, code) via modular designs such as ReAct ( 21 ) , MRKL ( 4 ) , and Toolformer ( 12 ) . METIS follows this line but targets research mentorship tasks (idea critique, planning, writing feedback) rather than generic tool use. For grounding, retrieval‑augmented generation ( 5 ) is paired with lightweight self‑critique ( 15 ) to encourage evidence and methodology checks. Autonomous scientific-process agents target end-to-end automation of the research lifecycle. Denario ( 17 ) uses a multi-agent pipeline to produce full manuscripts. FutureHouse’s Robin ( 2 ) and the Kosmos announcement ( 11 ) emphasize long-horizon discovery, and Aviary ( 10 ) provides a training environment for such agents. Our objective differs: we evaluate interactive mentorship and learner progress under constraints, not autonomous discovery throughput. In education, LLM tutoring agents frame learning as goal-oriented progression with personalization. GenMentor ( 19 ) proposes a multi-agent framework that maps learner goals to skills and sequences personalized paths. We share the emphasis on structured progression and personalization, but focus on research-writing mentorship and stage-aware evaluation. Evaluation of assistants often mixes pairwise preferences and rubric scores. LLM‑as‑a‑judge has been systematized (G‑Eval ( 7 ) , MT‑Bench/Chatbot Arena ( 22 ) ) alongside reliability concerns (position bias ( 14 ) , calibration ( 13 ) ). In our setting, we combine LLM‑judge preferences with student‑focused rubric trends and report uncertainty estimates for all summary metrics. Agent benchmarks highlight multi‑turn capability and task success (e.g., GAIA ( 9 ) , AgentBench ( 6 ) ); domain perspectives discuss AI for scientific practice ( 16 ) . Our scope is education‑oriented: stage‑aware research mentoring with a systems‑level comparison against strong chat baselines. Stage Summary A (Pre idea) Orientation and constraint-aware on-ramps before a concrete idea exists. B (Idea) Novelty, feasibility, and risk checks for early project ideas. C (Research plan) Multi-step research roadmaps with timelines, ablations, and resource/ethics constraints. D (First draft) Experiment design, ablations, and robustness checks for an attached technical paper. E (Second draft) Discussion, limitations, and rebuttal support grounded in an existing NLP paper/dataset. F (Final) Venue choice, artifact/release planning, and compliance checklists near submission time. Table 1: Writing stages A–F used in our stage-aware evaluation. Prompts are grouped by these stages for both pairwise preferences and student-persona rubric scores. Figure 1: METIS architecture. Stage detector and tool router select tools (Research Guidelines, web/document search, attachment search, methodology checks) based on writing stage. The agent synthesizes a reply and surfaces two self‑explanations ( Intuition , Why this is principled ), plus next steps and citations. Session memory maintains stage and constraints across turns. System Overview METIS is a tool-augmented, stage-aware assistant designed for research writing. At a high level, a lightweight router mediates between the user interface (CLI/TUI) and a set of tools: Research Guidelines (curated advice retrieval), Literature Search (arXiv/OpenReview retrieval + citation checks), Methodology Checks (sanity checks for metrics/baselines/ablations), a session memory maintains context across turns, and all interactions are logged for evaluation. Each reply includes two short self-checks— Intuition and Why this is principled —to expose the mental model and the justification behind the guidance; these are presented to the learner as explicit output blocks and are used by our rubrics for clarification/evidence metrics. METIS operates across six writing stages that reflect typical research progression: (A) Pre idea, (B) Idea, (C) Research plan, (D) First draft, (E) Second draft, and (F) Final. The stage detector (Figure 1 ) infers the current stage from conversation context to tailor tool selection and guidance depth. METIS’s behavior is controlled by a structured system prompt (Appendix C) that defines its research mentor persona, specifies tool-calling protocols, and templates the Intuition and Why this is principled blocks. Stage detection and tool routing are implemented via prompt instructions that analyze conversation context, rather than separate algorithmic modules. The mentor explicitly states the inferred current stage in responses, tracks progress across turns, and nudges the student toward the next stage with concrete actions when appropriate. Metric Scale Description Actionability 0.0–1.0 (scalar) Concrete, executable next steps (commands, parameters, timelines); 0.0 = abstract advice only. Clarification quality 0.0–2.0 Specific follow-ups tied to the user’s context; 2.0 = targeted questions, 0.0 = none. Citation quality 0.0–2.0 Cited sources are real, recent, and directly support guidance; 0.0 = missing or irrelevant. Citation validity 0/1 (binary) Whether citations resolve to real sources (no hallucinated references). Evidence integrity 0/1 (binary) Whether key claims are supported by cited evidence without contradictions or cherry-picking. RAG fidelity 0.0–2.0 Faithfulness of the response to retrieved context (low hallucinations or over-claims). Citation relevance 0.0–2.0 Topical match between cited sources and the claims they support. Source fit 0.0–2.0 Appropriateness of source type (e.g., primary paper vs commentary) for the claim. Persona compliance scaled (0.0–2.0) Adapts tone and detail to the stated learner persona and constraints. Stage awareness scaled (0.0–2.0) Matches advice to the current stage A–F; penalizes jumping too far ahead or behind. Tone constructive scaled (0.0–2.0) Critique is supportive, specific, and motivating; penalizes dismissive or vague tone. Stage-specific flags 0/1 (binary) Binary checks (e.g., timeline guidance, expectation management, resource estimation, risk mitigation) that fire only on applicable prompts. Table 2: Expert-judge metrics used for absolute evidence/compliance and mentorship-quality checks in the single-turn evaluation. Figure 2: LLM-judge pairwise preferences across stages ( n{=}15 prompts/stage; ties \leq 8\% excluded). METIS wins 71\% vs Claude Sonnet 4.5 and 54\% vs GPT-5 overall; error bars show Wilson 95% CIs. Metric Scale