Solomonoff-Inspired Hypothesis Ranking with LLMs for Prediction Under Uncertainty

Josh Barber Queensland University of Technology (QUT) Brisbane, Australia [email protected] Rourke Young Queensland University of Technology (QUT) Brisbane, Australia [email protected] Cameron Coombe Queensland University of Technology (QUT), CSIRO Brisbane, Australia came · 2025

Read Solomonoff-Inspired Hypothesis Ranking with LLMs for Prediction Under Uncertainty by Josh Barber Queensland University of Technology (QUT) Brisbane, Australia [email protected] Rourke Young Queensland University of Technology (QUT) Brisbane, Australia [email protected] Cameron Coombe Queensland University of Technology (QUT), CSIRO Brisbane, Australia came (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17145.txt.

Introduction

Josh Barber and coauthors rank hypotheses for prediction under uncertainty with a Solomonoff-inspired score, which favors hypotheses that are both simple and predictive. Sparse-data tasks force a balance that plain accuracy contests handle poorly.

Large language models propose the candidate hypotheses, and the method weights them by simplicity and fit into a mixture forecast per cell on Mini-ARC tasks. Against Bayesian Model Averaging, Solomonoff scoring spreads weight more evenly across rivals, while the Bayesian route piles weight on one likely but possibly flawed pick. The outputs stay conservative and uncertainty-aware even when hypotheses are noisy, a case for algorithmic priors in multi-hypothesis reasoning.

Catalog record

CREATORJosh Barber Queensland University of Technology (QUT) Brisbane, Australia [email protected] Rourke Young Queensland University of Technology (QUT) Brisbane, Australia [email protected] Cameron Coombe Queensland University of Technology (QUT), CSIRO Brisbane, Australia came
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17145

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Solomonoff-Inspired Hypothesis Ranking with LLMs for Prediction Under Uncertainty Josh Barber Queensland University of Technology (QUT) Brisbane, Australia [email protected] Rourke Young Queensland University of Technology (QUT) Brisbane, Australia [email protected] Cameron Coombe Queensland University of Technology (QUT), CSIRO Brisbane, Australia [email protected] Will Browne Queensland University of Technology (QUT) Brisbane, Australia [email protected] Abstract Reasoning under uncertainty is a key challenge in AI, especially for real-world tasks, where problems with sparse data demands systematic generalisation. Existing approaches struggle to balance accuracy and simplicity when evaluating multiple candidate solutions. We propose a Solomonoff-inspired method that weights LLM-generated hypotheses by simplicity and predictive fit. Applied to benchmark (Mini-ARC) tasks, our method produces Solomonoff-weighted mixtures for per-cell predictions, yielding conservative, uncertainty-aware outputs even when hypotheses are noisy or partially incorrect. Compared to Bayesian Model Averaging (BMA), Solomonoff scoring spreads probability more evenly across competing hypotheses, while BMA concentrates weight on the most likely but potentially flawed candidates. Across tasks, this highlights the value of algorithmic information-theoretic priors for interpretable, reliable multi-hypothesis reasoning under uncertainty. 1 Introduction Humans rarely bet everything on a single guess. Whether crossing a busy street or interpreting vague instructions, we naturally keep multiple possibilities in mind before committing to an action. This ability to reason under uncertainty by simultaneously considering and weighing alternative outcomes is a hallmark of human cognition [ 1 , 2 ] . It allows us to make decisions that are robust even in unfamiliar or ambiguous situations, which will be a necessary component of Artificial Intelligence (AI) embedded in real-world robotics. However, many modern AI systems struggle with uncertainty estimation. Large language models (LLMs) such as ChatGPT often produce outputs with high confidence even when incorrect , due to miscalibrated uncertainty [ 3 ] . More generally, artificial neural networks frequently fail to provide reliable confidence estimates, leading to over- or under-confident predictions in complex domains, i.e., domains that expose multiple hypotheses [ 4 ] . These limitations motivate the development of methods that explicitly evaluate and integrate multiple hypotheses. An ideal testbed for exploring such mechanisms needs to be open-ended, contain multiple hypotheses, and not be exhaustive by design. The Abstraction and Reasoning Corpus (ARC) [ 5 ] , also referred to as ARC-AGI-1, is a collection of grid-based visual reasoning tasks in which an agent must infer the transformation between an input and corresponding output grids, designed to evaluate systematic generalisation rather than pattern memorisation. Mini-ARC [ 6 ] , a simplified version of ARC, reduces problem complexity while retaining the core challenge: given several input–output examples, an agent must infer the hidden transformation and apply it to a novel input. An example Mini-ARC task is shown in Figure 1 , where objects stretch upward in even columns and downward in odd columns. Mini-ARC thus provides a controlled environment to investigate multi-hypothesis reasoning and predictive uncertainty. Uncertainty in AI has been addressed with probabilistic and Bayesian models [ 7 ] and neural network methods like ensembles and Monte Carlo dropout [ 8 , 9 ] , though these struggle in complex, combinatorial domains common to robotics. Tasks such as motion planning or perception often involve multiple plausible outcomes that must be weighed before action. In ARC and Mini-ARC, similar uncertainty arises when evaluating competing transformations [ 5 , 6 , 10 ] , with some systems focusing on a single hypothesis while others explore multiple alternatives [ 11 ] . Although multi-hypothesis reasoning can improve generalisation [ 12 ] , integrating and weighting competing hypotheses remains challenging. Frameworks combining principled hypothesis evaluation with interpretable scoring, such as the Solomonoff-inspired approach proposed here, could extend to domains requiring multi-hypothesis reasoning, such as path planning under uncertainty [ 13 ] . Instead of selecting one hypothesis, we use a Solomonoff-inspired approach (see Section 2.2 ) that builds a probability distribution over candidates, weighting them by simplicity and consistency. This allows reasoning under ambiguity, integrating alternatives, and making more robust predictions than single-hypothesis methods. The contributions of this work are: • Solomonoff-inspired scoring: Assigns probabilities to LLM-generated hypotheses using simplicity and predictive accuracy , unlike prior single-hypothesis methods, enabling robust evaluation of alternatives. • Weighted mixture-based prediction: Aggregates multiple hypotheses in a Solomonoff-weighted matrix with cell-level uncertainty, allowing informed predictions under ambiguity. • Empirical insights: Compared to Bayesian Model Averaging (BMA), the Solomonoff method was better calibrated in its confidence (uncertain when appropriate), and showed less noise across hypotheses. The words “problem,” “task,” and “puzzle” all refer to a single Mini-ARC puzzle consisting of several input-output grid examples and a test pair. These terms are used inter-changeably. 2 Related Work Bayesian methods are widely used to model predictive uncertainty in deep learning [ 14 , 7 , 15 ] . By placing distributions over parameters, Bayesian neural networks capture both model uncertainty (epistemic), arising from limited data or incomplete knowledge, and data uncertainty (aleatoric), arising from inherent noise in observations. Extensions such as ensembles and Monte Carlo dropout apply these principles to deep architectures [ 8 , 9 ] . However, Bayesian methods are computationally expensive, sensitive to priors, and less effective for high-dimensional, combinatorial inputs [ 16 , 17 ] . In ARC-style tasks, the main challenge is structural uncertainty (uncertainty over transformations or object manipulations) rather than parameter uncertainty, which standard Bayesian networks handle at the parameter level. Structural uncertainty instead arises from not knowing the rules or operations generating the observed patterns. Multi-hypothesis and LLM-based approaches target this structural uncertainty more directly. Wang et al. [ 18 ] show that maintaining multiple candidate hypotheses, ranked by simplicity and data consistency, improves robustness and generalisation compared to committing early to a single explanation. This highlights the value of reasoning over multiple alternatives, especially with sparse training examples as in Mini-ARC. Our method evaluates LLM-generated hypotheses with a Solomonoff-inspired scoring scheme, weighting them by simplicity and predictive accuracy. As a probabilistic baseline, we compare against Bayesian Model Averaging (BMA) [ 19 , 20 ] , which combines competing models probabilistically, weighting better-performing predictions higher. As a well-established method, it provides a reliable benchmark. Unlike parameter-based Bayesian methods, our approach targets structural uncertainty by ranking and merging alternative hypotheses, making it suitable for combinatorial reasoning with limited data. 2.1 Problem Setting Mini-ARC [ 6 ] is a simplified version of the Abstraction and Reasoning Corpus (ARC) [ 5 ] , designed to test abstract reasoning and systematic generalisation in small, grid-based visual tasks. Each problem consists of input–output pairs , where the input is a small grid (typically 5\times 5 ) of integer-valued colours (0–9, with 0 as background), and the output shows the result of a hidden transformation. The agent must infer the transformation and predict outputs for novel inputs (see Figure 1 ). Compared to ARC, Mini-ARC reduces combinatorial complexity with smaller grids (up to 30x30 vs 5x5), fewer input–output pairs (2-10 vs 3-5), and minimalistic settings, making tasks more tractable, interpretable, and suitable for evaluating abstract reasoning, systematic generalisation, and multi-hypothesis inference [ 6 ] . Figure 1: Mini-ARC task where odd columns shift down and even columns up; the agent infers this rule from input–output examples (see Table 1 ). The nth output is withheld for illustration but included in the dataset. Mini-ARC tasks emphasise

Read the complete text