Conservative Software Reliability Assessments Using Collections of Bayesian Inference Problems

Kizito Salako · 2025

Read Conservative Software Reliability Assessments Using Collections of Bayesian Inference Problems by Kizito Salako (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07038.txt.

Introduction

Kizito Salako and Rabiu Tsoho Muhammad work on cautious reliability assessment for safety-critical software, where an estimate must not overstate how dependable the software is. Their Bayesian approach considers not one inference problem but a whole collection of them. It reports the worst case across the collection: the predicted probability that the software runs without failure in the future. Software failures are modeled as a Bernoulli process.

The authors determine these worst-case posterior predictive probabilities explicitly and deduce asymptotic features of those probabilities and their priors. They then show how the results feed into useful assessments of high-stakes systems. The work extends robust Bayesian inference and earlier cautious Bayesian methods, giving assessors a sound way to keep reliability claims honest under prior uncertainty.

Catalog record

CREATORKizito Salako
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Conservative Software Reliability Assessments Using Collections of Bayesian Inference Problems Kizito Salako Abstract When using Bayesian inference to support conservative software reliability assessments, it is useful to consider a collection of Bayesian inference problems, with the aim of determining the worst-case value (from this collection) for a posterior predictive probability that characterizes how reliable the software is. Using a Bernoulli process to model the occurrence of software failures, we explicitly determine (from collections of Bayesian inference problems) worst-case posterior predictive probabilities of the software operating without failure in the future. We deduce asymptotic properties of these conservative posterior probabilities and their priors, and illustrate how to use these results in assessments of safety-critical software. This work extends robust Bayesian inference results and so-called conservative Bayesian inference methods. Keywords: robust Bayesian inference, conservative Bayesian inference, conservative software reliability assessment 1 Introduction Bayesian inference can support conservative assessments of software reliability. When assessing safety-critical software — whose primary function is ensuring safety (e.g., nuclear-power-plant safety protection systems or protection functions in autonomous vehicles) — an assessor will typically justify their beliefs (about how reliable the software is) with evidence. Evidence can include extensive successful operation of the software in other environments, use of code-blocks within the software that have undergone detailed analyses for correctness and validity, and the “quality” of the software development process (including the expertise of the development team); for various examples of evidence, see 23 , 25 , 22 , 28 , 9 , 29 , 15 . In principle, the assessor’s beliefs are about statistical model parameters that characterize software reliability, such as the probability of the software failing — these parameters’ true values are unknown, so the assessor expresses their uncertainty about the values (i.e. expresses their beliefs) as a prior distribution over possible parameter values. To be cautious, the assessor might only partly specify this prior, by specifying only probabilities of those parameter value ranges they have “strong” evidence for. In effect, they are specifying a credal set — a (convex) collection of prior distributions that are consistent with these partial specifications . Upon observing the failure behavior of the software during software testing, the assessor updates their beliefs and makes conservative claims about whether the software is sufficiently reliable; claims supported by posterior probabilities of interest. These posterior probabilities form part of the justification for certifying, or rejecting, the software: examples include the assessor’s confidence in an upper (credible) bound on the probability of the software failing at random, or the (predictive) probability that the software will not fail during future executions. In all such cases, the assessor is seeking the most undesirable value for the posterior probability of interest — the assessor is being as critical as they can be of the software, given the software’s observed performance and the available evidence. What is this worst-case posterior probability value, and which prior distribution (consistent with the assessor’s beliefs) gives this value? This question can be formalized as a constrained optimization over a collection of prior distributions. Let (\Omega,\Sigma) be a measurable space and X:\Omega\rightarrow I be a random variable mapping onto the compact interval I\subset\mathbb{R} . Let \mathcal{D} be the set of all probability measures defined on \Sigma . Consider a pair of continuous, real-valued, non-negative functions f,\,g , with shared domain I . Since I is compact, f(X) and g(X) are bounded, so f(X),\,g(X)\in\mathbb{L}^{1}(\Omega,\,\mathbb{P}) for all \mathbb{P}\in\mathcal{D} . Partition I into contiguous sub-intervals I_{1} , \ldots , I_{n} . The preimages A_{i}:=X^{-1}(I_{i})\in\Sigma partition \Omega . Assume \inf_{I_{j}}g>0 for some interval I_{j} . Let 0<p_{1},\ldots,p_{n}<1 satisfy \sum_{i=1}^{n}p_{i}=1 . Our assessor wants the answer to the following optimization problem. \displaystyle\underset{\mathcal{D}}{\inf}\frac{\mathbb{E}[f(X)]}{\mathbb{E}[g(X)]} s.t. \displaystyle\mathbb{P}(A_{1})=p_{1},\ldots,\mathbb{P}(A_{n})=p_{n} (1) Problem ( 1 ) is solved when one can state the infimum and a prior distribution of X that achieves it. Typically, for parameter value x , g(x) is the likelihood of the software’s observed failure behavior during operation, while f(x)/g(x) is a probability of a yet-to-be-observed (possibly unobservable ) event related to the software’s reliability. Prior distributions in \mathcal{D} that satisfy ( 1 )’s constraints are “feasible” and form a credal set. Problem ( 1 ) is not new. Generalizations have been extensively studied in the robust Bayesian inference literature (see 4 , 26 , 18 , 3 ), and certain nonlinear fractional program forms of ( 1 ) can be solved using Dinkelbach-type fixed-point iteration (see 13 , 34 ), while several special cases related to ( 1 ) — so-called conservative Bayesian inference ( CBI ) problems — have been solved to support conservatism in reliability assessments (see 8 , 35 , 39 , 21 , 31 , 30 , 32 ). Although CBI problems can often be solved numerically, explicit solutions tend to be only for special cases, and the solution methods employed can appear distinct and ad hoc . This paper: i) unifies these solution methods, exemplified by an explicit general solution to a problem that arises often in software reliability assessments; ii) analyses this solution’s properties; iii) highlights the practical implications of these results. Worst-case software reliability estimates are stated in Section 2 , while Section 3 discusses their properties, applications, limitations, and future work. 2 Results Proposition 1 simplifies problem ( 1 ) ( cf. Theorem 1 of 26 , Theorem 3 of 6 ). Proposition 1 . For any \mathbb{P}\in\mathcal{D} , \dfrac{\mathbb{E}[f(X)]}{\mathbb{E}[g(X)]}=\frac{\sum\limits_{i=1}^{n}f(x_{i})\mathbb{P}(A_{i})}{\sum\limits_{i=1}^{n}g(x_{i})\mathbb{P}(A_{i})} for some x_{i}\in I_{i} ( i=1...n ). Proof. See Appendix A, Supplementary Material. ∎ By proposition 1 , problem ( 1 ) is equivalent to the nonlinear fractional program \displaystyle\underset{x_{1}\in I_{1},...,x_{n}\in I_{n}}{\inf}\frac{\sum_{i=1}^{n}f(x_{i})p_{i}}{\sum_{i=1}^{n}g(x_{i})p_{i}}\,. (2) To illustrate solving ( 2 ), consider the following software reliability assessment scenario. An assessor is interested in the probability that a piece of on-demand software will operate correctly on the next m demands, after observing the software correctly handle k demands while also failing on r demands. A Bernoulli ( i.i.d. ) process is judged to adequately model the software’s successes and failures on a random sequence of demands during its operation. The Bernoulli process parameter X is the unknown probability of failure on the next demand ( pfd ), and \mathcal{D} is the set of all prior distributions over the unit interval (i.e. all possible prior distributions of X ). Specifically, for 0=y_{0}<y_{1}<\ldots<y_{n}=1 , the assessor expresses prior beliefs about how likely X is to lie in each of n intervals I_{1}:=[y_{0},y_{1}] , I_{2}:=(y_{1},y_{2}] , …, I_{n}:=(y_{n-1},y_{n}] , by singling out those \mathbb{P}\in\mathcal{D} that obey \mathbb{P}(X^{-1}(I_{i}))=p_{i} for i=1,\ldots,n . With f(x)=x^{r}(1-x)^{m+k} and g(x)=x^{r}(1-x)^{k} , problem ( 2 ) becomes \displaystyle\underset{x_{1}\in I_{1},\ldots,x_{n}\in I_{n}}{\inf}\frac{\sum_{i=1}^{n}x_{i}^{r}(1-x_{i})^{m+k}p_{i}}{\sum_{i=1}^{n}x_{i}^{r}(1-x_{i})^{k}p_{i}}\,. (3) Theorem 1 . A unique fixed point solves ( 3 ) — i.e. a triplet, \phi^{\ast} , y_{\ast} , y_{**} , that satisfies \displaystyle(1-y_{**})^{m}\left(\frac{r-y_{**}(m+k+r)}{r-y_{**}(k+r)}\right)=\phi^{\ast}=(1-y_{\ast})^{m}\left(\frac{r-y_{\ast}(m+k+r)}{r-y_{\ast}(k+r)}\right), \displaystyle\phi^{\ast}=\min\{\phi^{\ast}_{1},\phi^{\ast}_{2}\}\,, where \displaystyle\phi^{\ast}_{1} \displaystyle=\left\{\begin{array}[]{ll}\dfrac{\sum_{i=2}^{j_{1}}f(y_{i-1})p_{i}+\sum_{i=j_{1}+1}^{j_{2}-1}f(y_{i})p_{i}+f({y_{\ast}})p_{j_{2}}+\sum_{i=j_{2}+1}^{n}f(y_{i-1})p_{i}}{\sum_{i=2}^{j_{1}}g(y_{i-1})p_{i}+\sum_{i=j_{1}+1}^{j_{2}-1}g(y_{i})p_{i}+g({y_{\ast}})p_{j_{2}}+\sum_{i=j_{2}+1}^{n}g(y_{i-1})p_{i}}\,,\hfill j_{1}<j_{2}\\ \dfrac{\sum_{i=2}^{n}f(y_{i-1})p_{i}}{\sum_{i=2}^{n}g(y_{i-1})p_{i}}\,,\hfill j_{1}=j_{2}\end{array}\right. \displaystyle\phi^{\ast}_{2} \displaystyle=\left\{\begin{array}[]{ll}\dfrac{\sum_{i=2}^{j_{1}-1}f(y_{i-1})p_{i}+\sum_{i=j_{1}}^{j_{2}-1}f(y_{i})p_{i}+f({y_{\ast}})p_{j_{2}}+\sum_{i=j_{2}+1}^{n}f(y_{i-1})p_{i}}{\sum_{i=2}^{j_{1}-1}g(y_{i-1})p_{i}+\sum_{i=j_{1}}^{j_{2}-1}g(y_{i})p_{i}+g({y_{\ast}})p_{j_{2}}+\sum_{i=j_{2}+1}^{n}g(y_{i-1})p_{i}}\,,\hfill j_{1}<j_{2}\\ \dfrac{\sum_{i=2}^{j_{2}-1}f(y_{i-1})p_{i}+f(y_{\ast})p_{j_{2}}+\sum_{i=j_{2}+1}^{n}f(y_{i-1})p_{i}}{\sum_{i=2}^{j_{2}-1}g(y_{i-1})p_{i}+g(y_{\ast})p_{j_{2}}+\sum_{i=j_{2}+1}^{n}g(y_{i-1})p_{i}}\,,\hfill j_{1}=j_{2}\end{array}\right. for some 1\leqslant j_{1}\leqslant j_{2}\leqslant n such that \ y_{**}<\frac{r}{r+m+k}<\frac{r}{r+k}<y_{\ast} , \ y_{**}\in\bar{I}_{j_{1}} , and \ y_{\ast}\in\bar{I}_{j_{2}} (n.b. \bar{I}_{i} is the closure of I_{i} ). The infimum \phi^{*} is attained by the prior distribution of X \displaystyle\mathbb{P}(X=x)=\left\{\begin{array}[]{cl}p_{1},&\text{if }x=\mathbbm{1}_{n=1},\\ p_{i},&\left\{\begin{array}[]{l}\text{if }x=y_{i-1}\text{ and }\{2\leqslant i<j_{1}\}\cup\{j_{2}<i\leqslant n\},\\ \text{if }x=y_{i}\text{ and }\{j_{1}<i<j_{2}\},\end{array}\right.\\ p_{j_{1}},&\text{if }x=y_{j_{1}-1}\mathbbm{1}_{{\phi}^{*}=\phi_{1}^{*}}+\min\{y_{j_{1}},\,y_{*}\}\mathbbm{1}_{{\phi^{*}}=\phi_{2}^{*}\neq\phi_{1}^{*}},\\ p_{j_{2}},&\text{if }x=\left(y_{j_{1}-1}\mathbbm{1}_{j_{1}=j_{2}}+y_{*}\mathbbm{1}_{j_{1}<j_{2}}\right)\mathbbm{1}_{{\phi}^{*}=\phi_{1}^{*}}+y_{*}\mathbbm{1}_{{\phi^{*}}=\phi_{2}^{*}\neq\phi_{1}^{*}},\\ 0,&\text{otherwise.}\end{array}\right. Proof. See Appendix B, Supplementary Material. ∎ Corollary 1 . Consider the limiting case of ( 3 ) with no software failures ( r=0 ),

Read the complete text