Scorio.jl: A Julia package for ranking stochastic responses
Mohsen Hariri, Michael Hinczewski, Vipin Chaudhary · 2026
Read Scorio.jl: A Julia package for ranking stochastic responses by Mohsen Hariri, Michael Hinczewski, Vipin Chaudhary (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14103.txt.
Introduction
Hariri, Hinczewski, and Chaudhary present Scorio.jl, a Julia package for ranking systems from repeated task responses. One tensor-based interface hosts many ranking styles: direct scores, pairwise matches, psychometric models, voting, graph methods, and listwise rules. The same benchmark can thus be read under several ranking assumptions. The authors set the package beside existing Julia tools. Pilot tests cover rank recovery on synthetic data, stability with few trials, and runtime scaling.
Catalog record
| CREATOR | Mohsen Hariri, Michael Hinczewski, Vipin Chaudhary |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16260 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14103
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scorio.jl: A Julia package for ranking stochastic responses Mohsen Hariri, Michael Hinczewski, Vipin Chaudhary Abstract Scorio.jl is a Julia package for evaluating and ranking systems from repeated responses to shared tasks. It provides a common tensor-based interface for direct score-based, pairwise, psychometric, voting, graph, and listwise methods, so the same benchmark can be analyzed under multiple ranking assumptions. We describe the package design, position it relative to existing Julia tools, and report pilot experiments on synthetic rank recovery, stability under limited trials, and runtime scaling. keywords Julia, Ranking, Stochastic evaluation, Numerical analysis, Scientific computing, Large Language Models 1 Summary When a benchmark involves stochastic outputs—language models sampled multiple times per prompt [ 37 , 16 ] , randomized software tests, or repeated simulation runs—the natural data object is a three-dimensional tensor R\in\{0,\ldots,C\}^{L\times M\times N} recording L systems on M tasks over N trials. Different ranking methods consume different views of this tensor: averages, pairwise win counts, per-task orderings, or latent trait estimates. Moving between these views typically means moving between separate tools. Scorio.jl keeps the tensor as the single input and handles the view conversions internally. For binary-response benchmarks, a user can pass the same R to avg , bradley_terry , rasch , borda , or pagerank and get back comparable rank vectors without touching the surrounding pipeline; weighted score-based methods such as bayes also support graded outcomes. The package therefore serves two roles: day-to-day leaderboard construction and methodological studies of how rankings shift when the ranking rule changes. Julia’s multiple dispatch and dense-array primitives make it practical to keep these heterogeneous methods behind one API [ 4 ] . 2 Statement of need Ranking under repeated stochastic observation is common across several fields. In educational testing, examinees answer the same items multiple times or under adaptive schedules [ 6 , 23 ] . In software benchmarking, nondeterminism from hardware scheduling or random initialization means that a single run is rarely conclusive. In language-model evaluation, metrics like Pass@k aggregate multiple samples per task, and limited sample budgets can produce unstable rankings [ 8 , 17 ] . In all these cases, the analyst faces questions that go beyond “which system scored highest on average”: How many trials are needed before the ranking stabilizes? Do pairwise skill models and direct score averages agree? Does the top system change when ties are handled differently? Can prior information from earlier runs be folded in? Answering these questions today requires stitching together separate tools, each with its own input format. Scorio.jl removes that friction. It treats the repeated response tensor as the canonical data structure and lets the user swap ranking methods with a single function call. This is useful whenever the ranking rule itself is part of the analysis rather than a fixed post-processing step. 3 State of the field Several Julia packages handle parts of this problem. StatsBase provides general-purpose ranking and tied-ranking utilities but does not operate on repeated benchmark tensors [ 21 ] . RankAggregation.jl aggregates multiple score columns into a consensus ordering [ 20 ] . Ranking.jl focuses on binary comparison records and includes draft Elo, Bradley–Terry, and TrueSkill implementations [ 42 ] . TrueSkillThroughTime.jl specializes in temporally evolving skill estimation [ 24 ] . These are useful packages, but each addresses a narrower slice of the problem. Scorio.jl differs in two ways. First, its primitive input is a repeated response tensor over systems, tasks, and trials under stochastic sampling. Second, it brings together evaluation-based, pairwise, probabilistic, psychometric, voting, graph, and listwise rankers behind one interface. That combination is the package’s main software contribution: it turns ranking-method comparison from a custom integration problem into an ordinary part of benchmark analysis. 4 Software design 4.1 Response tensors and a shared API For evaluating a single system, Scorio.jl uses outcome matrices E\in\{0,\ldots,C\}^{M\times N} , where M is the number of tasks, N the number of trials, and C the maximum category label ( C+1 outcome categories in total). For ranking multiple systems, it uses response tensors R\in\{0,\ldots,C\}^{L\times M\times N} , where L is the number of systems. Two-dimensional inputs are promoted automatically to a single-trial tensor, so one-shot benchmarks use the same code path. When outcomes are graded, score-based methods can consume them directly via rubric weights. Different method families consume different views of R . Pointwise methods work with trial means \bar{r}_{\ell m}=\frac{1}{N}\sum_{n=1}^{N}R_{\ell mn} or rubric-weighted scores. Pairwise methods work with induced decisive-win counts W_{\ell,\ell^{\prime}}=\lvert\{(m,n):R_{\ell mn}>R_{\ell^{\prime}mn}\}\rvert , with ties tracked separately when needed. Voting and listwise methods use the per-task ordering induced by (R_{1mn},\ldots,R_{Lmn}) . Graph methods build a weighted comparison graph. IRT methods estimate latent abilities and task difficulties from binary outcomes. All these conversions happen inside the package; the user passes tensors in the same (L,M,N) format. 4.2 Score construction, tie handling, and priors Score construction is separated from rank assignment. Most methods return continuous scores, and a shared tie-aware ranking layer converts those scores to ranks. This avoids duplicating tie logic and makes the rank semantics (competition, dense, ordinal, fractional) explicit. The package also exposes explicit priors for MAP-style methods, including Gaussian, Laplace, Cauchy, uniform, custom, and empirical priors. This is useful when previous benchmark runs or baseline evaluations should inform the current ranking, and it encourages users to state their assumptions rather than burying them in custom code. 4.3 Illustrative Julia workflow A typical workflow starts from an integer tensor R with shape (L, M, N) . The same object can be passed to several rankers without any reshaping. Listing shows the shared API. {lstlisting} [language=julia,basicstyle= ,breaklines=true,breakatwhitespace=false,columns=fullflexible,keepspaces=true,frame=single,showstringspaces=false,caption=Illustrative Scorio.jl workflow using the shared tensor interface.,label=lst:workflow] using Scorio # R has size (L, M, N) for systems, tasks, and trials r_bayes, s_bayes = bayes(R; return_scores=true) r_bt, s_bt = bradley_terry(R; return_scores=true) r_rasch, theta = rasch_mml(R; return_scores=true) r_pr, s_pr = pagerank(R; return_scores=true) # A graded evaluation for one system across tasks and trials E = [0 1 2 2 1; 1 1 0 2 2] w = [0.0, 0.5, 1.0] mu, sigma = bayes(E, w) # Convert a score vector into several tie-aware rank views rank_views = rank_scores(s_bayes) 4.4 Method families Table 1 lists the ranking families currently available in Scorio.jl together with the data view each family uses. The evaluation-based family includes avg , bayes , and several Pass@k variants [ 8 , 17 ] . Sequential rating models (Elo, Glicko, TrueSkill) process induced pairwise matches [ 12 , 15 , 18 ] . Paired-comparison models such as Bradley–Terry, Davidson, and Rao–Kupper estimate strength parameters from aggregated win/tie counts [ 7 , 10 , 34 ] , alongside Thompson sampling and a Metropolis MCMC ranker [ 38 , 27 ] . Item-response theory models (Rasch, 2PL, 3PL, MML, and a dynamic variant) estimate latent ability jointly with task difficulty [ 35 , 5 , 28 , 6 , 40 ] . Voting rules cover Borda, Copeland, Schulze, ranked pairs, Kemeny–Young, Nanson, Baldwin, and majority judgment [ 11 , 9 , 26 , 36 , 39 , 22 , 43 , 29 , 2 , 3 ] . Graph and spectral methods include PageRank, Rank Centrality, AlphaRank, a Nash meta-game solver, SerialRank, and HodgeRank [ 32 , 41 , 30 , 31 , 1 , 14 , 19 ] .