Project Hermes: A Model-Agnostic Validation Layer for Wearable Health Prediction Systems Richik Chakraborty Abstract The deployment of wearable-based health prediction systems has accelerated rapidly, yet these systems face a fundamental challenge: they generate alerts under substantial uncertainty without principled mechanisms for user-specific validation. While large language models (LLMs) have been increasingly applied to healthcare tasks, existing work focuses predominantly on diagnosis generation and risk prediction rather than post-prediction validation of detected signals. We introduce Project Hermes, a model-agnostic validation layer that treats signal confirmation as a sequential decision problem. Hermes operates downstream of arbitrary upstream predictors, using LLM-generated contextual queries to elicit targeted user feedback and performing Bayesian confidence updates to distinguish true positives from false alarms. In a 60-day longitudinal case study of migraine prediction, Hermes achieved a 34% reduction in false positive rate (from 61.7% to 12.5%) while maintaining 89% sensitivity, with mean lead time of 4.2 hours before symptom onset. Critically, Hermes does not perform diagnosis or make novel predictions; it validates whether signals detected by upstream models are clinically meaningful for specific individuals at specific times. This work establishes validation as a first-class computational problem distinct from prediction, with implications for trustworthy deployment of consumer health AI systems. 1 Introduction: The Validation Gap in Wearable Health AI The past decade has witnessed exponential growth in consumer wearable adoption, with global shipments exceeding 500 million devices annually. This proliferation has enabled continuous, unobtrusive monitoring of physiological signals previously accessible only in clinical settings. Correspondingly, machine learning models trained on these signals now claim to predict stress episodes ( 7 ) , migraine attacks ( 11 ) , sleep disorders ( 9 ) , and cardiovascular events ( 24 ) hours to days before symptom onset. These systems extract features from heart rate variability (HRV), sleep architecture, activity patterns, and ambient context, achieving impressive sensitivity in controlled studies ( 28 ) . However, a critical gap separates research prototypes from trustworthy deployment: persistent false positives that erode user trust and clinical utility. Modern wearable prediction systems typically achieve sensitivities of 80-90% but suffer from specificity as low as 50-60% ( 5 ) , meaning that for every true alert, users receive one or more false alarms. This imbalance reflects a fundamental tension in health monitoring: population-level models trained on aggregate data must make individual-level predictions about heterogeneous users in diverse contexts. Most deployed systems respond to this challenge with static decision thresholds, offering users binary alerts (“Risk elevated”) without mechanisms for refinement or personalization ( 18 ) . Users face an unsatisfying choice: accept frequent false alarms or manually adjust sensitivity settings without understanding the underlying uncertainty. As these systems scale from research pilots to millions of consumer devices, the gap between population model performance and individual relevance becomes not merely inconvenient but potentially dangerous, either through missed events (threshold too high) or alert fatigue (threshold too low). We argue that this gap stems from treating prediction and validation as a single problem. When a model outputs “migraine risk: 0.75,” it makes a claim under uncertainty derived from population statistics and historical patterns. But the user’s implicit question—“Is a migraine actually happening to me right now, given my current symptoms, context, and recent history?”—requires validation, not merely prediction. Validation demands incorporating individual-specific information that prediction models, by construction, cannot access: subjective symptom reports, contextual explanations for physiological changes (e.g., “I just exercised”), and recent behavioral factors. Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in medical question-answering ( 32 ) , clinical note summarization ( 39 ) , and diagnostic dialogue ( 36 ) . Systems like Med-PaLM 2 achieve performance comparable to human physicians on medical licensing examinations, while GPT-4 has been shown to generate clinically appropriate responses to patient queries ( 22 ) . However, existing LLM applications in healthcare operate almost exclusively in generative mode, producing diagnoses, treatment recommendations, or risk assessments de novo. These systems face well-documented challenges including hallucination ( 1 ) , inappropriate confidence ( 43 ) , and potential for diagnostic overreach in domains where they lack true understanding ( 41 ) . We introduce Project Hermes, which repurposes LLMs for a fundamentally different task: not prediction or diagnosis, but validation of signals detected by upstream systems. Hermes treats validation as a sequential decision problem under uncertainty. Given a signal from an arbitrary upstream predictor (which may be a black-box model, a clinician-defined threshold, or a complex ensemble), Hermes generates targeted questions to gather additional evidence, updates confidence via Bayesian inference, and decides whether to alert the user, remain silent, or gather more information. This approach offers several architectural advantages. First, model-agnosticism: Hermes operates downstream of any predictor without requiring access to model internals, training data, or prediction mechanisms. Second, epistemic humility: rather than claiming to know the correct diagnosis, Hermes merely refines confidence in hypotheses already raised by upstream systems. Third, regulatory decoupling: validation layers may face different oversight than diagnostic systems, particularly when they augment rather than replace existing workflows. Fourth, compositional deployment: multiple prediction models can share a single validation layer, enabling consistent user experiences across heterogeneous backends. We demonstrate feasibility through a 60-day longitudinal case study of migraine prediction, a domain characterized by high individual variability, early physiological signals, and user-reportable ground truth. Our pilot participant wore a consumer fitness tracker continuously while logging migraine occurrences. A simple upstream predictor flagged 47 potential events based on HRV and sleep disruption, achieving 92% sensitivity but only 39% specificity (29 false positives). Hermes processed these signals through targeted questioning (mean 3.2 questions per event), reducing false positives to 2 (specificity 88%) while maintaining 89% sensitivity. Mean lead time was 4.2 hours before full symptom onset, providing actionable warning for prophylactic intervention. The contributions of this work are threefold. First, we establish validation as a first-class computational problem distinct from prediction, with different objectives, inputs, and epistemic commitments. Second, we formalize validation as sequential Bayesian updating where LLM-generated questions serve as information-gathering actions, providing a principled framework for human-in-the-loop health AI. Third, we demonstrate that model-agnostic validation can substantially improve signal quality without requiring access to or modification of upstream prediction systems, opening pathways for trustworthy deployment of consumer health AI. 2 Related Work 2.1 Wearable-Based Health Prediction The application of machine learning to wearable physiological data has generated a substantial literature across multiple health domains. Early work focused on activity recognition and sleep staging using accelerometry ( 9 ) , but recent systems leverage richer signal modalities including photoplethysmography-derived heart rate variability, skin temperature, and galvanic skin response. For migraine specifically, several studies have demonstrated that physiological precursors emerge hours to days before subjectively-reported onset. Houle et al. ( 11 ) trained random forest models on HRV features from 4,400+ patient-days, achieving 0.72 AUC for 24-hour migraine prediction. Taherdoost et al. ( 34 ) conducted a systematic review of HRV-migraine relationships, identifying consistent patterns of parasympathetic withdrawal (reduced RMSSD, increased LF/HF ratio) in pre-ictal states. Sleep disruption has also been implicated as both trigger and prodrome ( 40 ) , with architecture changes (reduced REM percentage, increased awakenings) preceding attacks in diary-based studies. Beyond migraine, stress detection from multimodal wearables has attracted considerable research attention. Can et al. ( 7 ) surveyed 50+ stress detection systems, noting that most achieve 75-85% accuracy in laboratory conditions but degrade substantially in free-living deployments due to context variability and sensor noise. Smets et al. ( 33 ) demonstrated that large-scale (N=1,002) wearable data could identify digital phenotypes for daily stress, with personalized models outperforming population-level approaches by 8-12% in cross-validated accuracy. Cardiovascular applications have perhaps the strongest clinical evidence: the Apple Heart Study ( 24 ) enrolled 419,297 participants and achieved 84% positive predictive value for atrial fibrillation alerts when validated against ECG patch monitoring, though notably this included physician review of algorithm outputs rather than purely automated alerting. A common pattern emerges across these domains: sensitivity often exceeds 80-90% in controlled evaluations, but specificity remains problematic, typically 50-70%. Bent et al. ( 5 ) systematically investigated accuracy limitations in consumer optical heart rate sensors, finding that motion artifact, skin tone, and device placement introduce substantial noise that prediction models must overcome. More fundamentally, these systems face the challenge of distinguishing physiologically-driven signal changes (e.g., stress response) from benign contextual factors (e.g., exercise, caffeine, ambient temperature) that produce similar wearable signatures. Existing approaches to this challenge include adaptive thresholding ( 5 ) , which adjusts decision boundaries based on recent user history, and ensemble methods that combine multiple signal modalities to improve specificity. However, these remain prediction-centric solutions: they attempt to make the upstream model better at discriminating signal from noise. None incorporate post-prediction validation mechanisms that could leverage information unavailable to wearable sensors, such as subjective symptom reports or contextual knowledge. 2.2 Large Language Models in Healthcare The application of large language models to medical tasks has accelerated dramatically following the release of GPT-3 and subsequent models. Med-PaLM ( 32 ) demonstrated that instruction-tuned LLMs could achieve 67.6% accuracy on MedQA (medical licensing exam questions), approaching the 69.2% passing threshold, with Med-PaLM 2 subsequently exceeding expert physician performance at 86.5%. These systems leverage medical literature, clinical guidelines, and case-based reasoning, though their training data and internal representations remain largely opaque. Nori et al. ( 22 ) evaluated GPT-4 on medical problem-solving tasks, finding strong performance on diagnostic reasoning but notable failures on questions requiring numerical calculation, temporal reasoning about disease progression, or integration of multimodal data like lab values and imaging. Clinical documentation applications have shown particular promise. Van Veen et al. ( 39 ) demonstrated that adapted LLMs could outperform medical experts in clinical text summarization, reducing reading time by 40% while maintaining or improving information retention. Tu et al. ( 36 ) introduced AMIE, a diagnostic dialogue system that engaged in multi-turn conversation with patients to gather history and propose differential diagnoses. In simulated consultations with actors, AMIE matched or exceeded primary care physicians on diagnostic accuracy and communication quality, though the authors emphasized substantial barriers to real-world deployment. Patient-facing applications have also emerged. Ayers et al. ( 3 ) compared ChatGPT and physician responses to patient questions posted on Reddit’s r/AskDocs, finding that ChatGPT responses were rated higher on both quality (3.6x more likely to be rated good/very good) and empathy (9.8x more likely to be rated empathetic/very empathetic). However, this work has attracted methodological criticism regarding selection bias and lack of medical outcome validation. Despite these successes, LLM applications in healthcare face serious challenges. Hallucination—generating confident but incorrect medical claims—poses particular risks in clinical domains ( 1 ) . Wang et al. ( 43 ) conducted a systematic review of LLM limitations in clinical reasoning, identifying failure modes including: reasoning from outdated medical knowledge, inappropriate confidence in uncertain situations, inability to properly handle numerical or temporal reasoning, and tendency to prioritize common diagnoses over rare but serious conditions. Wachter and Brynjolfsson ( 41 ) argue that generative AI in healthcare faces a “trust paradox”: the more accurate these systems become, the more dangerous their errors, because users may over-rely on them in cases where they fail. Critically, nearly all existing LLM health applications operate in generative mode: the model produces diagnoses, treatment plans, or medical explanations de novo. This places enormous epistemic burden on the LLM to correctly reason about complex medical scenarios with incomplete information. Validation, by contrast, starts from hypotheses generated externally (by sensors, simple algorithms, or humans) and merely refines confidence—a more constrained and potentially more reliable task. 2.3 Active Learning and Human-in-the-Loop Systems Active learning provides theoretical foundations for intelligent query selection under uncertainty. The canonical framework ( 31 ) assumes a learner with a hypothesis space \mathcal{H} , a pool of unlabeled data, and the ability to query labels for selected examples. The learner selects queries to maximize information gain, expected model change, or reduction in generalization error. In medical contexts, active learning has been applied to diagnostic query optimization, where the goal is to identify the minimum set of tests (bloodwork, imaging, patient history questions) needed to confidently diagnose a condition. Ko and Fox ( 15 ) used Gaussian process regression with active learning for disease progression modeling in dementia, demonstrating that intelligently-selected patient assessments reduced prediction uncertainty more efficiently than fixed schedules. Medical image analysis has been a particularly active area for human-in-the-loop learning. Yang et al. ( 45 ) introduced “suggestive annotation” for biomedical image segmentation, where a deep network identifies ambiguous regions and requests human labels only for those areas, reducing annotation burden by 70-80%. Zhang et al. ( 46 ) applied Bayesian decision theory to optimize clinical trial designs, using sequential enrollment decisions to minimize patient exposure while maintaining statistical power. More broadly, human-in-the-loop machine learning has emerged as a paradigm for building trustworthy AI systems. Mosqueira-Rey et al. ( 21 ) survey the field, identifying three primary patterns: (1) humans provide training data through interactive labeling, (2) humans validate model outputs before deployment, and (3) humans and models collaborate in mixed-initiative systems. Zhang et al. ( 47 ) specifically examine evaluation methodologies for such systems, noting that standard accuracy metrics often fail to capture key qualities like user trust, mental workload, and long-term engagement. However, few systems apply active learning or human-in-the-loop principles specifically to post-prediction validation in health monitoring. Most work focuses on improving model training through human feedback (reinforcement learning from human feedback, interactive machine learning) rather than using human input to validate predictions after models have been deployed. The gap is particularly acute for consumer health applications, where users lack medical expertise but possess unique knowledge about their own symptoms, context, and history—knowledge that sensors cannot capture but which is essential for interpreting physiological signals. 2.4 Positioning Hermes Project Hermes occupies a distinct niche in this landscape. Unlike wearable prediction systems, it does not attempt to detect health events from physiological signals; it assumes an upstream system has already flagged a potential event. Unlike diagnostic LLM systems, it does not generate medical hypotheses de novo; it validates hypotheses proposed by external predictors. Unlike active learning systems focused on model training, it operates at inference time to refine individual predictions. And unlike human-in-the-loop systems that treat humans as oracles providing ground truth labels, Hermes recognizes that user responses are themselves uncertain and must be weighted according to their diagnostic informativeness. The closest prior work is in clinical decision support systems that implement sequential questioning protocols (e.g., computerized diagnostic algorithms that gather history through structured forms). However, these systems typically use fixed decision trees rather than Bayesian updating, lack natural language interfaces, and operate in clinical settings rather than consumer wearable contexts. Hermes combines Bayesian active learning, LLM-based question generation, and model-agnostic architecture to create a validation layer suitable for deployment alongside black-box consumer health prediction systems—a domain where traditional clinical decision support tools cannot operate. 3 Conceptual Framing: Validation as a First-Class Problem The distinction between prediction and validation is not merely terminological but reflects fundamentally different computational problems with distinct objectives, constraints, and epistemic commitments. This section formalizes these differences and establishes why model-agnostic validation layers constitute a necessary component of trustworthy health AI systems. 3.1 Prediction versus Validation: A Formal Distinction Consider a wearable-based health monitoring system attempting to detect a condition C (e.g., migraine, stress episode, cardiac arrhythmia) in a user. Let \mathbf{x}_{t} denote physiological signals from wearable sensors at time t , and y_{t}\in\{0,1\} the true presence or absence of condition C at time t . A prediction system learns a function f:\mathbb{R}^{d}\rightarrow[0,1] that maps sensor signals to risk estimates: \hat{p}_{t}=f(\mathbf{x}_{t},\mathbf{x}_{t-1},\ldots,\mathbf{x}_{t-k}) (1) where \hat{p}_{t} estimates P(y_{t}=1\mid\mathbf{x}) . The predictor is optimized on population data \mathcal{D}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N} to maximize some objective like AUC, F1 score, or expected utility. Critically, the predictor has access only to information available to sensors: heart rate, activity, sleep patterns, etc. It cannot access subjective symptoms, contextual explanations (“I just ran up stairs”), or user knowledge about their own health history and patterns. A validation system , by contrast, operates conditional on a prediction already having been made. It receives as input: • The sensor signals \mathbf{x}_{t} • The predictor’s output \hat{p}_{t} • A hypothesis H about the presence of condition C The validator’s task is to refine confidence in H by gathering additional evidence \mathbf{q}=(q_{1},\ldots,q_{n}) through user queries and incorporating their responses \mathbf{r}=(r_{1},\ldots,r_{n}) . The validation process computes: P(H\mid\mathbf{x}_{t},\hat{p}_{t},\mathbf{q},\mathbf{r}) (2) Table 1 summarizes the architectural and epistemic distinctions between these two functions. The predictor generalizes from population patterns; the validator individualizes to specific contexts. The predictor optimizes for sensitivity/specificity tradeoffs; the validator explicitly manages uncertainty and decision costs. Most importantly, the predictor commits to a risk estimate based on incomplete information, while the validator acknowledges this incompleteness and seeks to address it through interaction. Table 1: Prediction versus Validation: Conceptual Distinctions Dimension Prediction Models Hermes (Validation) Input space Sensor signals \mathbf{x}_{t} Signals + prediction + user context Output Risk score \hat{p}_{t} Updated confidence P(H\mid\cdot) Objective Maximize population AUC Minimize decision loss for individual Ground truth Historical labels y_{i} Current user state (unobserved) Information source Wearable sensors only Sensors + user feedback Role of LLM Not applicable Question generation interface Optimization Offline training on \mathcal{D} Online inference per instance Deployment mode Replace/augment clinician Augment user decision-making This distinction has practical consequences. Prediction systems face the classic bias-variance tradeoff and must balance sensitivity against specificity using a single threshold. Validation systems, by contrast, implement a three-way decision: alert (high confidence), suppress (low confidence), or gather more information (medium confidence). This enables asymmetric error costs—for example, allowing more false negatives for non-urgent conditions while maintaining low false positive rates to preserve user trust. 3.2 Why Model-Agnosticism Matters Hermes treats the upstream predictor as a black box, accessing only its output \hat{p}_{t} rather than internal representations, feature importance, or model architecture. This design choice offers several advantages that are critical for real-world deployment. Compatibility with proprietary systems: Many commercial wearable health applications use proprietary prediction algorithms whose details are not disclosed. Fitbit’s “Stress Management Score,” Apple Watch’s “Cardio Fitness,” and Whoop’s “Strain” metrics all involve complex, undocumented processing of sensor data. A validation layer that requires access to model internals cannot be deployed alongside these systems. Model-agnosticism enables validation to wrap arbitrary upstream predictors, whether open-source research prototypes or commercial black boxes. Regulatory decoupling: In many jurisdictions, medical AI systems face regulatory scrutiny proportional to their risk classification and claimed capabilities. A diagnostic system that claims to detect disease faces higher bars (FDA Class II or III in the US) than a wellness monitoring tool. Crucially, a validation layer that refines confidence in signals detected by already-regulated systems may face different regulatory pathways than the upstream predictor itself. By maintaining strict separation between prediction (upstream) and validation (Hermes), we enable modular regulatory approaches where each component can be evaluated independently. No retraining required: Traditional personalization approaches for wearable health systems often involve fine-tuning models on user-specific data or learning user-specific thresholds. This requires storing user data, implementing continuous learning pipelines, and managing model versioning. Hermes, by contrast, performs validation using only the current instance (signal + user responses), without modifying upstream models. This reduces infrastructure complexity and enables deployment without user-specific model training. Compositional deployment: A single user might have multiple health monitoring systems active simultaneously: migraine prediction, stress detection, sleep quality assessment, etc. Each may use different sensors, models, and decision criteria. Rather than building validation mechanisms separately into each predictor, a unified validation layer can process signals from multiple sources, managing query budgets and user burden holistically. This compositional architecture reduces redundancy and provides consistent user experiences across heterogeneous backend systems. Interpretability by construction: Black-box predictors face ongoing criticism for lack of interpretability. While techniques like SHAP values and attention visualization attempt to explain model decisions, they provide post-hoc rationalization rather than mechanistic understanding. Validation, by contrast, makes confidence refinement explicit and observable: users see exactly which questions are asked, how their responses update confidence, and why the system decides to alert or suppress. This transparency may build trust more effectively than opaque “explanation” of black-box models. The cost of model-agnosticism is that Hermes cannot leverage internal model representations that might indicate uncertainty or provide additional context. For instance, if an ensemble predictor has high internal disagreement (multiple models give conflicting outputs), this would be valuable information for validation. However, we argue that in deployment contexts involving proprietary or legacy systems, model-agnostic validation is often the only feasible approach. 3.3 Scope and Limitations by Design Hermes intentionally constrains its scope to avoid epistemic overreach and regulatory complications. These constraints are features, not limitations: No de novo hypothesis generation: Hermes does not propose new conditions or diagnoses unprompted by upstream signals. If the user reports symptoms not associated with any triggered hypothesis, Hermes does not attempt to explain them. This prevents the system from operating as an unregulated diagnostic tool. No treatment recommendations: Hermes outputs confidence estimates and, in high-confidence cases, alerts that a condition may be present. It never recommends treatments, medications, or clinical interventions. At most, it might suggest general categories of response (“Consider prophylactic measures”) that users would already know from disease management plans. No operation without upstream prediction: Hermes cannot function as a standalone health monitoring system. It requires an external predictor to flag potential events. This ensures that Hermes adds safety (reducing false positives) rather than introducing new risks. No claims beyond signal validation: All system outputs are framed as refinements of confidence about signals detected by other systems, not as independent medical claims. For example: “Based on your responses, the likelihood of the detected signal representing a true migraine has increased to 85%” rather than “You have a migraine.” User control and transparency: Users can inspect confidence levels before responding to questions, adjust thresholds for alerting, opt out of validation for specific conditions, and ignore questions without penalty. The system maintains transparency about its limitations and uncertainty. These constraints position Hermes in a defensible regulatory and ethical space: it augments existing monitoring systems by reducing false positives, but does not claim diagnostic authority or medical expertise. In doing so, it provides a template for LLM deployment in health contexts that avoids the risks of generative diagnostic systems while still offering meaningful value. 4 System Architecture: Project Hermes Hermes implements validation as a modular pipeline with six components that transform uncertain signals into calibrated confidence estimates. Figure 1 illustrates the information flow through the system. Each component is designed for independent modification or replacement, enabling adaptation to different conditions, user populations, and deployment contexts without architectural changes. Figure 1: Hermes System Architecture. The system processes signals from an upstream predictor through six components: signal triggering evaluates whether validation should initiate, the illness codex provides structured medical knowledge, LLM question generation creates natural language queries, user responses provide evidence, Bayesian updating computes posterior confidence, and decision logic determines whether to alert, suppress, or ask additional questions. 4.1 Upstream Predictor (External) The upstream predictor is any system that outputs risk estimates or binary signals indicating potential health events. Formally, at time t , the predictor produces either a continuous risk score s_{t}\in[0,1] or a binary flag s_{t}\in\{0,1\} based on sensor inputs \mathbf{x}_{t-w:t} over some window w . Hermes treats this output as an uncertain prior, not ground truth. In our migraine case study, we implemented a simple multivariate threshold predictor: s_{t}=\mathbb{1}\left[\text{HRV}_{t}<\mu_{\text{HRV}}-1.5\sigma_{\text{HRV}}\lor\text{SleepQual}_{t}<60\lor\text{Stress}_{t}\geq 7\right] (3) where \mu_{\text{HRV}} and \sigma_{\text{HRV}} are computed from the user’s 30-day rolling baseline. This intentionally simple predictor achieves high sensitivity (92%) but poor specificity (39%), creating substantial opportunity for validation to add value. However, Hermes’s architecture accommodates arbitrary predictor designs. Examples of compatible upstream systems include: • Machine learning models: Random forests, gradient boosted trees, or neural networks trained on historical wearable data with labeled outcomes • Clinical decision rules: Expert-defined criteria like “resting heart rate ¿ 100 bpm AND sleep efficiency ¡ 70% AND reports stress ¿ 6” • Proprietary algorithms: Commercial wearable analytics (Fitbit Stress Management, Apple CardioFitness) accessed via API • Ensemble systems: Multiple models whose outputs are combined via voting or stacking The only requirement is that the predictor can be polled for current risk estimates or flags potential events via callback. Hermes does not access predictor internals, training data, or feature representations. This opacity is intentional: it ensures Hermes can be deployed alongside systems where such access is unavailable or prohibited by intellectual property constraints. 4.2 Signal Triggering Layer The triggering layer decides when validation should initiate. Triggering too aggressively burdens users with unnecessary questions; triggering too conservatively misses opportunities to refine confidence before false alerts propagate. We implement three classes of triggers that can be combined: Threshold breaches: The simplest trigger fires when predictor output exceeds a fixed threshold \tau_{\text{trigger}} : \text{Trigger}_{\text{threshold}}=\mathbb{1}[s_{t}>\tau_{\text{trigger}}] (4) For binary predictors, \tau_{\text{trigger}}=0.5 triggers on any positive signal. For continuous predictors, this threshold balances user burden against coverage of potential events. Volatility patterns: Sudden changes in physiological signals often precede health events but may also reflect benign context shifts (exercise, caffeine, stress). We detect volatility using Bollinger Bands adapted to physiological time series: \text{Trigger}_{\text{volatility}}=\mathbb{1}[|s_{t}-\mu_{t-k:t-1}|>2\sigma_{t-k:t-1}] (5) where \mu and \sigma are computed over a sliding window of recent predictions. This captures anomalous behavior relative to the user’s recent baseline, triggering validation even when absolute risk scores are moderate. Multi-factor conjunctions: Some conditions exhibit characteristic multi-factor patterns that increase confidence when multiple weak signals co-occur. For migraine, we trigger when: \text{Trigger}_{\text{multi}}=\mathbb{1}[\text{HRV anomaly}\land\text{sleep disruption}\land\text{elevated stress}] (6) even if no single factor exceeds its individual threshold. This implements a form of weak supervision where pattern co-occurrence indicates higher prior probability. In practice, we use a composite trigger that fires when any condition is met, with hysteresis to prevent rapid re-triggering. Once validation begins for a signal, that signal is marked as processed for 24 hours to avoid redundant questioning. 4.3 Illness Codex The Illness Codex is a structured knowledge base encoding clinical expertise about symptom-condition relationships. Unlike LLM-generated knowledge, which may hallucinate or confabulate relationships, the codex is manually curated by medical experts based on diagnostic criteria, clinical guidelines, and peer-reviewed literature. For each condition C in the codex, we define: 1. Question templates \mathcal{Q}_{C}=\{q_{1},\ldots,q_{n}\} : Specific symptoms or contextual factors to query 2. Likelihood ratios \{\text{LR}^{+}_{i},\text{LR}^{-}_{i}\} : Diagnostic value of positive/negative responses 3. Question metadata: Priority, timing constraints, dependencies between questions The likelihood ratios encode sensitivity and specificity for each symptom: \displaystyle\text{LR}^{+}_{i} \displaystyle=\frac{P(\text{yes to }q_{i}\mid C\text{ present})}{P(\text{yes to }q_{i}\mid C\text{ absent})}=\frac{\text{sensitivity}}{1-\text{specificity}} (7) \displaystyle\text{LR}^{-}_{i} \displaystyle=\frac{P(\text{no to }q_{i}\mid C\text{ present})}{P(\text{no to }q_{i}\mid C\text{ absent})}=\frac{1-\text{sensitivity}}{\text{specificity}} (8) For migraine, our codex includes question templates like: condition: "migraine" questions: - id: "aura_visual" template: "Have you noticed any visual changes such as blind spots, shimmering lights, or zigzag lines in the past [TIMEFRAME]?" LR_plus: 5.8 LR_minus: 0.5 priority: high - id: "photophobia" template: "Is light bothering you more than usual?" LR_plus: 3.2 LR_minus: 0.3 priority: medium - id: "unilateral_pain" template: "If you have any head discomfort, is it primarily on one side?" LR_plus: 2.1 LR_minus: 0.6 priority: low depends_on: ["pain_present"] Likelihood ratios were derived from systematic reviews of migraine diagnostic criteria. For visual aura, \text{LR}^{+}=5.8 indicates that patients with migraine are 5.8× more likely to report visual changes than those without migraine during a prodromal period. The ICHD-3 (International Classification of Headache Disorders, 3rd edition) criteria informed these estimates, though we note substantial inter-individual variability. The codex also encodes question dependencies to avoid illogical sequences (e.g., asking about pain location before confirming pain exists) and timing constraints (e.g., visual aura questions are most informative within 2 hours of signal detection, as aura typically precedes headache by 15-60 minutes). Importantly, the codex is declarative rather than procedural: it specifies what questions exist and their diagnostic value, but not the order in which they should be asked. Question selection is determined dynamically by the Bayesian updating module to maximize expected information gain given current uncertainty. 4.4 LLM Question Generation Module The LLM’s sole role is linguistic: converting codex templates into natural language queries adapted to context, time of day, and conversation flow. This constrained usage minimizes risks associated with LLM deployment in health contexts. The generation process follows a template: SYSTEM PROMPT: You are a health monitoring assistant helping validate a potential health signal. Your role is ONLY to: 1. Convert the provided question template into natural language 2. Adapt phrasing based on time/context 3. NEVER make medical claims or diagnoses 4. NEVER suggest treatments 5. Keep questions clear, empathetic, and brief USER PROMPT: Template: [CODEX_TEMPLATE] Context: [TIME_OF_DAY, RECENT_RESPONSES, USER_PREFERENCES] Generate a natural language question. For the visual aura template at 8:30 AM when the user recently woke: • Template: “Have you noticed any visual changes such as blind spots, shimmering lights, or zigzag lines in the past [TIMEFRAME]?” • Generated: “Good morning. Since waking up, have you experienced any unusual visual changes—like blind spots, flickering lights, or zigzag patterns in your vision?” The LLM adapts phrasing while preserving diagnostic intent. We use GPT-4 with temperature 0.3 to balance naturalness against consistency. Importantly, generation is deterministic given the same inputs: the LLM does not introduce variability in question semantics, only in phrasing. We implement several safeguards against LLM misbehavior: 1. Output validation: Generated questions must contain all key symptom terms from the template (e.g., “blind spots,” “shimmering lights”) 2. Prohibited phrase detection: Reject outputs containing diagnostic claims (“You have,” “This indicates”), treatment suggestions (“Take,” “Try”), or excessive certainty (“definitely,” “certainly”) 3. Fallback to template: If generation fails validation, use the template directly without LLM adaptation 4. Human review: During deployment, a sample of generated questions is reviewed weekly for drift or inappropriate content This architecture separates medical knowledge (codex) from linguistic interface (LLM), ensuring that question content is determined by clinical expertise rather than statistical patterns in training data. 4.5 Bayesian Confidence Updating User responses trigger Bayesian updates to confidence in the hypothesis H that condition C is present. Let \mathbf{q}=(q_{1},\ldots,q_{n}) denote questions asked and \mathbf{r}=(r_{1},\ldots,r_{n})\in\{0,1\}^{n} the binary responses (yes/no). The posterior probability is: P(H\mid s,\mathbf{r})=\frac{P(s\mid H)\cdot\prod_{i=1}^{n}P(r_{i}\mid H)\cdot P(H)}{P(s,\mathbf{r})} (9) where: • P(H) is the base rate prior for condition C in the user population • P(s\mid H) captures predictor reliability: how likely is signal s when H is true? • P(r_{i}\mid H) encodes symptom informativeness from the codex likelihood ratios We compute the update sequentially after each response using log-odds: \text{logit}(P(H\mid s,r_{1},\ldots,r_{i}))=\text{logit}(P(H\mid s,r_{1},\ldots,r_{i-1}))+\log(\text{LR}_{i}) (10) where \text{LR}_{i}=\text{LR}^{+}_{i} if r_{i}=1 (yes) and \text{LR}_{i}=\text{LR}^{-}_{i} if r_{i}=0 (no). The signal likelihood P(s\mid H) is calibrated from historical predictor performance. For our threshold-based migraine predictor, we observed that 92% of true migraines triggered the signal (sensitivity), while 61% of non-migraine periods also triggered it (1 - specificity). This gives: \displaystyle P(s=1\mid H=1) \displaystyle=0.92 (11) \displaystyle P(s=1\mid H=0) \displaystyle=0.61 (12) The base rate P(H) is personalized. Our participant averaged 6 migraines per month over 60 days, giving P(H)\approx 0.2 over 24-hour windows. This prior is continually updated from user history. For computational efficiency, we maintain a particle filter representation when the hypothesis space extends beyond binary presence/absence. For conditions with subtypes (e.g., migraine with vs. without aura), we track a probability distribution over all hypotheses and select questions to maximize expected reduction in Shannon entropy: q^{*}=\arg\max_{q\in\mathcal{Q}}\mathbb{E}_{r}\left[H(P(H))-H(P(H\mid r))\right] (13) This greedy selection prioritizes questions with high discriminative power given current uncertainty. 4.6 Decision Logic Based on posterior confidence p=P(H\mid s,\mathbf{r}) , Hermes selects one of three actions: \text{action}=\begin{cases}\text{Alert}&\text{if }p>\tau_{\text{alert}}\\ \text{Suppress}&\text{if }p<\tau_{\text{suppress}}\\ \text{Ask}&\text{if }\tau_{\text{suppress}}\leq p\leq\tau_{\text{alert}}\land n10 is considered strong evidence) motivates the need for additional information gathering. 5.4 Query Selection: Maximizing Expected Information Gain Given current belief state P(H_{i}\mid s,\mathbf{r}_{1:n}) after n responses, we must decide which question q\in\mathcal{Q} to ask next, where \mathcal{Q} is the set of available questions from the codex. We use a greedy information gain criterion. For each candidate question q , we compute the expected reduction in entropy: \text{IG}(q)=H\left(P(H\mid s,\mathbf{r}_{1:n})\right)-\mathbb{E}_{r}\left[H\left(P(H\mid s,\mathbf{r}_{1:n},r)\right)\right] (23) where the expectation is over possible responses r\in\{0,1\} : \mathbb{E}_{r}\left[H(P(H\mid\cdot))\right]=\sum_{r\in\{0,1\}}P(r\mid s,\mathbf{r}_{1:n})\cdot H(P(H\mid s,\mathbf{r}_{1:n},r)) (24) The response probability is computed by marginalizing over hypotheses: P(r\mid s,\mathbf{r}_{1:n})=\sum_{i=0}^{K}P(r\mid H_{i},q)\cdot P(H_{i}\mid s,\mathbf{r}_{1:n}) (25) The question with maximum expected information gain is selected: q^{*}=\arg\max_{q\in\mathcal{Q}}\text{IG}(q) (26) This greedy approach is suboptimal compared to full tree search (which would plan multi-step ahead), but computational tractability and responsiveness constraints make greedy selection practical. In experiments with synthetic data, greedy selection achieved 94% of the information gain of 3-step lookahead while requiring 1-2 seconds of computation versus 30+ seconds. 5.5 Bayesian Update Rule Upon receiving response r to question q , we update beliefs using Bayes’ rule: P(H_{i}\mid s,\mathbf{r}_{1:n},r)=\frac{P(r\mid H_{i},q)\cdot P(H_{i}\mid s,\mathbf{r}_{1:n})}{\sum_{j=0}^{K}P(r\mid H_{j},q)\cdot P(H_{j}\mid s,\mathbf{r}_{1:n})} (27) For computational efficiency in the binary case, we work in log-odds space: \text{logit}(P(H_{1}\mid\cdot))=\log\left(\frac{P(H_{1}\mid\cdot)}{P(H_{0}\mid\cdot)}\right) (28) The update becomes additive: \text{logit}(P(H_{1}\mid s,\mathbf{r}_{1:n},r))=\text{logit}(P(H_{1}\mid s,\mathbf{r}_{1:n}))+\log(\text{LR}_{q}(r)) (29) where \text{LR}_{q}(r) is the likelihood ratio for response r to question q : \text{LR}_{q}(r)=\begin{cases}\text{LR}^{+}_{q}=\frac{P(\text{yes}\mid H_{1},q)}{P(\text{yes}\mid H_{0},q)}&\text{if }r=1\\ \text{LR}^{-}_{q}=\frac{P(\text{no}\mid H_{1},q)}{P(\text{no}\mid H_{0},q)}&\text{if }r=0\end{cases} (30) This log-odds formulation makes the contribution of each piece of evidence explicit and enables efficient sequential updating. 5.6 Optimal Stopping and Decision Boundaries Validation is an optimal stopping problem: at each step, decide whether to (a) take a terminal action (alert or suppress), or (b) gather more information. We formalize this using decision-theoretic principles. Define a cost function \mathcal{L}(a,h) for taking action a when true state is h : \mathcal{L}(a,h)=\begin{cases}0&\text{if }a=\text{alert}\land h=H_{1}\text{ (true positive)}\\ C_{\text{FP}}&\text{if }a=\text{alert}\land h=H_{0}\text{ (false positive)}\\ C_{\text{FN}}&\text{if }a=\text{suppress}\land h=H_{1}\text{ (false negative)}\\ 0&\text{if }a=\text{suppress}\land h=H_{0}\text{ (true negative)}\\ C_{\text{query}}&\text{if }a=\text{ask}\text{ (any state)}\end{cases} (31) The expected cost of action a given current belief state is: \mathbb{E}[\mathcal{L}(a)]=\sum_{i=0}^{K}\mathcal{L}(a,H_{i})\cdot P(H_{i}\mid s,\mathbf{r}_{1:n}) (32) At each decision point, we select the action minimizing expected cost: a^{*}=\arg\min_{a\in\{\text{alert, suppress, ask}\}}\mathbb{E}[\mathcal{L}(a)] (33) For migraine, we set cost parameters based on user interviews: • C_{\text{FP}}=5 (false alert causes inconvenience, anxiety, unnecessary medication) • C_{\text{FN}}=3 (missed early warning reduces opportunity for prophylaxis) • C_{\text{query}}=0.5 (each question imposes minor burden) These asymmetric costs reflect that false positives are more harmful than false negatives for non-emergency conditions, as they erode trust and may lead to alert fatigue. The query cost ensures that asking questions is preferred only when the expected reduction in decision loss exceeds the burden imposed. Solving the optimal stopping problem yields action thresholds. Let p=P(H_{1}\mid\cdot) . The expected costs are: \displaystyle\mathbb{E}[\mathcal{L}(\text{alert})] \displaystyle=C_{\text{FP}}\cdot(1-p) (34) \displaystyle\mathbb{E}[\mathcal{L}(\text{suppress})] \displaystyle=C_{\text{FN}}\cdot p (35) \displaystyle\mathbb{E}[\mathcal{L}(\text{ask})] \displaystyle=C_{\text{query}}+\mathbb{E}_{q,r}[\min_{a^{\prime}}\mathbb{E}[\mathcal{L}(a^{\prime})]] (36) The third term requires estimating future decision quality, which we approximate using value iteration. Empirically, this yields thresholds: a^{*}=\begin{cases}\text{alert}&\text{if }p>\frac{C_{\text{FP}}}{C_{\text{FP}}+C_{\text{FN}}}\approx 0.63\\ \text{suppress}&\text{if }p<\frac{C_{\text{FN}}}{C_{\text{FP}}+C_{\text{FN}}}\approx 0.37\\ \text{ask}&\text{otherwise, if }n