Nothing new under the sun: The NIST primer on “Probability and Likelihood Ratios” gives a misleading negative portrayal of the likelihood-ratio framework Authors: Geoffrey Stewart Morrison Published: 2026 ABSTRACT AbstractIn March 2025, the National Institute of Standards and Technology (NIST) published a series of primers designed to provide judges and attorneys with a neutral source of information on foundational forensic-science principles. Unfortunately, the primer on “Probability and Likelihood Ratios” gives a misleading negative portrayal of the likelihood-ratio framework. This perspectives paper refutes that misleading portrayal.Keywords: Interpretation, Likelihood ratio, NIST, Primer, Probability Quotation Introduction There is widespread agreement among experts in forensic inference and statistics that the likelihood-ratio framework is the logically correct framework for interpretation of forensic evidence (see position statements in Table 1). Use of the likelihood-ratio framework is also advocated by key organizations (see Table 2). ISO 21043-4:2025 Forensic Sciences – Part 4: Interpretation [1] does not require the use of the likelihood-ratio framework, but it does include requirements and recommendations applicable if the likelihood-ratio framework is used (Morrison et al. [2]). Table 1Examples of position statements supporting the use of the likelihood-ratio framework.Table 1Position statementNumber of authors‌/signatories‌/supporters[3] Aitken et al. (2011) Expressing evaluative opinions: A position statement31[4] Morrison et al. (2017) A comment on the PCAST report: Skip the “match”/“non-match” stage19[5] Morrison et al. (2021) Consensus on validation of forensic voice comparison20[6] Expert Working Group on Human Factors in Forensic DNA Interpretation (2024) Forensic DNA interpretation and human factors: Improving practice through a systems approach25[7] Morrison et al. (2025) A response to EA-4/23 INF:2025 “The Assessment and Accreditation of Opinions and Interpretations using ISO/IEC 17025:2017”a57a[8] describes changes made to EA-4/23 INF in response to [7]. Examples of position statements supporting the use of the likelihood-ratio framework. [8] describes changes made to EA-4/23 INF in response to [7]. Table 2Publications by key organizations supporting the use of the likelihood-ratio framework.Table 2OrganizationPublication(s)DNA Commission of the International Society of Forensic Genetics (ISFG)[9] Gill et al. (2006) Recommendations on the interpretation of mixtures[10] Gill et al. (2012) Recommendations on the evaluation of STR typing results that may include drop-out and/or drop-in using probabilistic methodsAssociation of Forensic Science Providers of the United Kingdom and of the Republic of Ireland (AFSP)[11] AFSP (2009) Standards for the formulation of evaluative forensic science expert opinionRoyal Statistical Society (RSS)[12] Aitken et al. (2010) Fundamentals of probability and statistical evidence in criminal proceedings: Guidance for judges, lawyers, forensic scientists and expert witnessesEuropean Network of Forensic Science Institutes (ENFSI)[13] Willis et al. (2015) ENFSI guideline for evaluative reporting in forensic scienceNational Institute of Forensic Science of the Australia New Zealand Policing Advisory Agency (NIFS ANZPAA)[14] Ballantyne et al. (2017) An introductory guide to evaluative reportingAmerican Statistical Association (ASA)[15] Kafadar et al. (2019) American Statistical Association position on statistical statements for forensic evidenceRoyal Society (RS) and Royal Society of Edinburgh (RSE)[16] Nic Daéid et al. (2020) The use of statistics in legal proceedings: A primer for courtsForensic Science Regulator for England & Wales (FSR)[17] FSR (2021) Codes of practice and conduct: Development of evaluative opinions[18] FSR (2026) Forensic science activities: Interpretation and communication Publications by key organizations supporting the use of the likelihood-ratio framework. Introductions to the likelihood-ratio framework include: Balding & Steele [19], Hicks et al. [20], Robertson et al. [21], Bright & Coble [22], Aitken et al. [23]. Morrison [24] and Koehler et al. [25] each list the likelihood-ratio framework as a key element in an ongoing paradigm shift in evaluation of forensic evidence / in a scientific revolution affecting forensic science. Morrison [24] describes the paradigm shift as moving away from methods based on human perception and subjective judgment, and replacing them with methods based on relevant data, quantitative measurements, and statistical models. The new paradigm (which Morrison [24] labels “the forensic-data-science paradigm”) uses methods that are: •logically correct (i.e., they calculate likelihood ratios), logically correct (i.e., they calculate likelihood ratios), •transparent and reproducible, transparent and reproducible, •resistant to cognitive bias, resistant to cognitive bias, •calibrated under casework conditions, and calibrated under casework conditions, and •validated under casework conditions. validated under casework conditions. Morrison [26], which is aimed at a legal-professional audience, discusses the items in this list as criteria for assessing scientific validity. As described in Kuhn [27], paradigm shifts / scientific revolutions often face concerted and long-lasting opposition. The paradigm shift in evaluation of forensic evidence is no exception. Resistance to the adoption of the likelihood-ratio framework is reported in Evett [28] (published in 1991) and in Swofford et al. [29] (published 30 years later in 2021). Over many years, multiple publications have argued against the use of likelihood ratios, and multiple other publications have refuted those arguments (see Table 3). Table 3Examples of publications that have argued against the use of likelihood ratios and of publications that have refuted those arguments.Table 3Arguments against the use of likelihood ratiosRefutations of arguments against the use of likelihood ratios[30] French & Harrison (2007)[31] French et al. (2010)[32] Rose & Morrison (2009)[33] Morrison (2009)[34] Morrison (2014)R v T [2010] EWCA Crim 2439[35] Berger et al. (2011)[36] Redmayne et al. (2011)[37] Robertson et al. (2011)[38] Morrison (2012)[39] Biedermann et al. (2012)[40] Nordgaard & Rasmussen (2012)[41] Sjerps & Berger (2012)[42] Kaye (2012)[43] Faigman et al. (2011)[44] Jamieson (2012)[45] Berger et al. (2011)[46] Berger et al. (2012)[47] Bodziak (2012a)[48] Bodziak (2012b)[49] Thompson (2012)[50] Risinger (2013)[51] Lennard (2013)[52] Morrison & Stoel (2014)[53] National Institute of Standards and Technology (NIST) (2017-10-12)a[54] Lund & Iyer (2017)b[55] Morrison (2017-10-16)[56] Aitken & Nordgaard (2018)[57] Gittelson et al. (2018)[58] Aitken et al. (2018)caThe original title of NIST’s press release was “NIST urges caution in use of courtroom evidence presentation method”. In response to criticism, NIST later changed the title to “NIST experts urge caution in use of courtroom evidence presentation method: Use of ‘Likelihood Ratio’ not consistently supported by scientific reasoning approach, authors state”, and added a disclaimer “to clarify that NIST as an institution [was] not making a policy recommendation on this subject”. The original press release with the original title and without the disclaimer had already been circulated and reposted elsewhere, e.g., at https://phys.org/news/2017-10-nist-urges-caution-courtroom-evidence.html (retrieved 2025-11-08).bSpitzner (2023) [59] states that “Lund and Iyer’s foundational arguments have been effectively refuted by others”.cAppendix B of Lund & Iyer (2022-05-10) [60] responds to [55]–[58]. Examples of publications that have argued against the use of likelihood ratios and of publications that have refuted those arguments. The original title of NIST’s press release was “NIST urges caution in use of courtroom evidence presentation method”. In response to criticism, NIST later changed the title to “NIST experts urge caution in use of courtroom evidence presentation method: Use of ‘Likelihood Ratio’ not consistently supported by scientific reasoning approach, authors state”, and added a disclaimer “to clarify that NIST as an institution [was] not making a policy recommendation on this subject”. The original press release with the original title and without the disclaimer had already been circulated and reposted elsewhere, e.g., at https://phys.org/news/2017-10-nist-urges-caution-courtroom-evidence.html (retrieved 2025-11-08). Spitzner (2023) [59] states that “Lund and Iyer’s foundational arguments have been effectively refuted by others”. Appendix B of Lund & Iyer (2022-05-10) [60] responds to [55]–[58]. In March 2025, the United States Department of Commerce’s National Institute of Standards and Technology (NIST) published a series of informational primers on forensic science (Jones et al. [61]). According to the abstract for the series, the audience for the primers is “Officers of the Court including judges and attorneys”, and the purpose of the primers is to provide “neutral sources to inform their understanding of foundational scientific and forensic science principles.” Unfortunately, the primer on “Probability and Likelihood Ratios” (Jones et al. [61] II.E. pp 34–35) presents a misleading negative portrayal of the likelihood-ratio framework. This appears to be yet another example of resistance to the paradigm shift. Because NIST is a highly respected organization, NIST publications are imbued with authority. The NIST primer is therefore likely to influence decisions made by courts and by managers of forensic-service providers. For this reason, we considered it important to refute the primer’s misleading negative portrayal of the likelihood-ratio framework. We titled the present paper “Nothing new under the sun” because the sorts of anti-likelihood-ratio arguments presented in the NIST primer are not novel. These sorts of arguments have been made before and have been refuted before (see Table 3). Agreement among experts The NIST primer gives the false impression that, among experts in forensic inference and statistics, there is an evenly matched debate between those who support the use of likelihood ratios and those who oppose the use of likelihood ratios. This is exemplified in the following two sentences from the first paragraph of the primer: “Those who support the use of probability and likelihood ratios (LRs) in forensic science opinions see them as an application of logical reasoning with uncertainty. Opposition to using probability and LRs in forensic science opinions focuses on the subjectiveness (variability) in the LRs that are provided, the use of LRs that do not align with performance data and misunderstanding of the expression.” Nowhere does the primer reveal that (as indicated in §1 above) the vast majority of experts in forensic inference and statistics support the use of the likelihood-ratio framework (see Table 1 and Table 2). Nor does it reveal that many publications have refuted anti-likelihood-ratio arguments (see Table 3). By giving the false impression that there is little agreement among experts in forensic inference and statistics on this matter, the primer misleads readers unfamiliar with the field who might otherwise have been influenced by the fact that there is overwhelming agreement. This is particularly pertinent in the context of legal admissibility in US federal and state courts because both Daubert1 and Frye2 include “general acceptance” within a relevant scientific community as a criterion for judges to consider when making decisions on admissibility. Subjectivity Emphasis on subjectivity The NIST primer repeatedly states that probabilities and likelihood ratios are subjective, e.g.: “Although probability is often thought to be an objective measure because it is expressed as a number, it is in fact subjective.” “A likelihood ratio (LR), a ratio of two mutually exclusive probabilities, is also subjective.” The primer even emphasizes subjectivity by having a section titled: “What Adds Subjectivity?” And its list of limitations of likelihood ratios includes: “Is subjective because of personal or organizational choices” The everyday meaning of “subjective” is negative In everyday use, “subjective” is usually understood to mean something like: •“based on or influenced by personal feelings, tastes, or opinions”,3 or “based on or influenced by personal feelings, tastes, or opinions”,3 or •“influenced by or based on personal beliefs or feelings, rather than based on facts”.4 “influenced by or based on personal beliefs or feelings, rather than based on facts”.4 The connotations of such everyday definitions of “subjective” are negative. Readers who are unfamiliar with forensic inference and statistics are likely to read “subjective” with its everyday meaning and to be misled by the primer into forming a negative view of likelihood ratios. The technical meaning of “subjective” is neutral When used in a technical sense, “subjective” does not have the negative connotations of its everyday meaning. In the Oxford English Dictionary, definition 3 of the adjective “subjective” includes “having its source in the mind”,5 and it is contrasted with definition I.3b of the adjective “objective”, which includes “external to or independent of the mind”.6 In §4.6 below, we will discuss a specific technical meaning of “subjective” used in the field of probability and statistics. No method for assessing strength of evidence is 100% objective No method for calculating a likelihood ratio is 100% objective. If a forensic practitioner calculates a likelihood ratio based on relevant data, quantitative measurements, and statistical models, before calculating the likelihood ratio, they must choose: •what properties of the items of interest to measure, what properties of the items of interest to measure, •what statistical models to use, and what statistical models to use, and •what data to use for training and validating those models. what data to use for training and validating those models. A forensic practitioner has to decide what properties, models, and data are appropriate given the conditions of the case. Making those choices requires subjective judgments on the part of the forensic practitioner.7 It is also true, however, that no non-likelihood-ratio method for assessing strength of evidence is 100% objective. All methods will, at some level, require subjective judgments to be made. By emphasizing subjectivity involved in likelihood-ratio methods, but not acknowledging subjectivity involved on non-likelihood-ratio methods, the primer gives that misleading impression that there may be some alternative to likelihood ratios that is objective and that that alternative would therefore be superior to likelihood ratios. Subjectivity is not a problem, but transparency is essential Contrary to what the NIST primer implies, subjectivity per se is not a problem; however, it is essential that the forensic practitioner be transparent with respect to the subjective judgments they make. This includes being transparent about which methods and data they use to calculate likelihood ratios, and being transparent about the fact that their choices with respect to which methods and data to use are made on the basis of subjective judgments. Such transparency allows an expert appointed by an opposing party in a case to critique what the forensic practitioner did, and for a judge at an admissibility hearing or a trier of fact a trial to decide whether what the forensic practitioner did is reasonable. Resistance to cognitive bias For methods based on relevant data, quantitative measurements, and statistical models, subjective judgments about what properties to measure, what statistical models to use, and what data to use for training and validating those models are made by the forensic practitioner at the beginning of the process of calculating a likelihood ratio. Once these decisions have been made, the remainder of the process is objective. Because the subjective judgments are made at the beginning of the process, the process is much less susceptible to cognitive bias than a process in which perceptual observations (instead of quantitative measurements) are made and/or in which the assessment of the strength of evidence associated with the observations is performed using subjective judgment (instead of statistical models). A process in which the output of the assessment of strength of evidence is directly the result of subjective judgment is highly susceptible to cognitive bias. Transparency A process in which the output of the assessment of strength of evidence is directly the result of subjective judgment is also not transparent. One cannot know what happened in the forensic practitioner’s mind. Even the forensic practitioner themself does not know (Edmond et al. [62]). A judge or trier of fact cannot, therefore, assess whether what the forensic practitioner did was reasonable. In contrast, in a process in which subjective judgments are made before calculating a likelihood ratio, those subjective judgments are simpler and easier to describe. They are easier to make transparent, and their reasonableness is easier to debate and consider (we reference an example in §3.10 below). Reproducibility Because a process based on perceptual observations and/or subjective judgment is not transparent, what the original forensic practitioner did cannot be reproduced by another forensic practitioner. In contrast, if the process is based on relevant data, quantitative measurements, and statistical models, the methods and procedures can be described in detail, and, in principle, the data and software used to calculate the likelihood ratio can be shared. This allows another suitably qualified forensic practitioner to reproduce what the original forensic practitioner did, and to determine whether they obtain the same (or similar) results. Different decisions lead to different results If, based on their subjective judgments, different forensic practitioners make different decisions about what properties of items to measure, what statistical models to use, and/or what data to use for training those models, then one would expect the likelihood-ratio values that they calculate to be different. The NIST primer appears to consider this a problem. For example, it states: “Opposition to using probability and LRs in forensic science opinions focuses on the subjectiveness (variability) in the LRs that are provided” “Probability … is subjective because it relies on differing information available and assumptions made to arrive at the probability number. … In theory, an infinite number of combinations of assumptions can be made that would result in an infinite number of probability values.” “Probabilities and LRs are personal and will vary depending on the data …, assumptions made, and calculation model used.” But if the assumptions made, properties measured, data used, and/or statistical models used differ, then of course one would expect the resulting likelihood-ratio values to differ. The primer’s negativity toward the use of probabilities and likelihood ratios suggests its authors may favor use of non-statistical methods for assessing strength of evidence; however, if one were using a non-statistical method to assess strength of evidence, and the information/data available and/or the assumptions changed, then one would also expect the output of the non-statistical method to change. As an anonymous reviewer quoted in Morrison & Stoel [52] stated: “All decisions, whether statistically informed [or not], are conditional on a set of assumptions and background information. To criticise likelihood ratios for having them is absurd. Equally, if your opinion does not change when a radical deviation from the assumptions is observed (or made), then what is the value of that opinion? Either the assumption wasn’t necessary, or the opinion is worthless.” Tracing causes of different results If forensic practitioners’ decisions about what properties to measure, what statistical models to use, and what data to use to train those statistical models are made transparent, then the causes of differences in likelihood-ratio values can be traced to differences in subjective judgments or differences in assumptions made at the beginning of the process. Thompson [63] provides a detailed example from an actual DNA case.8 In principle, after considering input from experts instructed by opposing parties, the judge at an admissibility hearing or trier of fact at trial could decide that a forensic practitioner’s subjective-judgment-based decisions were not reasonable, and therefore decide to not admit the practitioner’s testimony or decide to give little weight to the practitioner’s testimony. In contrast, if two forensic practitioners using methods based on human perception and/or subjective judgment (rather than methods based on relevant data, quantitative measurements, and statistical models) reported different strength-of-evidence conclusions, there would be no way to trace the causes of those differences. Morrison [68] discusses an example from an actual forensic-voice-comparison case. If a trier of fact were to decide that two different forensic practitioners who calculated different likelihood-ratio values both made reasonable subjective judgments, their choice of which likelihood-ratio value to give greater weight to could be made by considering the results of validations of the two methods. We discuss validation in §5 below. Frequentist and Bayesian approaches Unnecessary and unhelpful The NIST primer includes a section titled “LR – Frequentist and Bayesian Approaches”. The differences between frequentist and Bayesian approaches are technical, and, in our opinion, discussing them was unnecessary and unhelpful in the context of a two-page primer aimed at judges and attorneys. The discussion of these approaches in the primer is superficial and misleading, and we believe it is more likely to confuse rather than enlighten a reader who is not already familiar with probability and statistics. We suspect that discussion of these approaches was included in the primer in order to emphasize an area of disagreement between experts in forensic inference and statistics, and in order to emphasize subjectivity. Because we believe that discussion of differences between frequentist and Bayesian approaches is unnecessary and unhelpful in a discussion aimed at a legal-professional audience, we would prefer not to discuss this topic unless it were absolutely necessary. Because of the prominence of this topic in the primer, however, we feel obliged to respond. Our apologies in advance if some readers find the following to be too technical. Both frequentists and Bayesians calculate probabilities of evidence given hypotheses, not probabilities of hypotheses given evidence One way in which the primer’s discussion of frequentist and Bayesian approaches is misleading is in the juxtaposition of the following sentences: “Frequentist statistics focus on the probability (Pr) of the evidence (E) given a hypothesis (H), and is expressed as Pr(E|H).” “The Bayesian approach helps an individual quantify their personal probability assessment of a hypothesis given the evidence, Pr(H|E).” In fact, under both frequentist and Bayesian approaches, forensic practitioners calculate likelihood ratios or assign Bayes factors9 that quantify p(E|H1)/p(E|H2) (probabilities of evidence given hypotheses). If they are working in the likelihood-ratio framework, under neither approach do forensic practitioners calculate posterior odds, p(H1|E)/p(H2|E) (probabilities of hypotheses given evidence). Both frequentists and Bayesians use random sampling Another way in which the primer is misleading is by including the following sentence only in the paragraph on frequentist approaches: “Inferences are based on random sampling.” In fact, random sampling is also used in Bayesian approaches. Bayesians do not replace missing information with their belief Yet another way in which the primer is misleading is the following sentence in the paragraph on Bayesian approaches: “Inferences can be made even if the information is incomplete using probability distribution models to describe a belief.” Contrary to what a reader might reasonably infer from this sentence, Bayesian approaches do not replace missing information with a forensic practitioner’s belief. Frequentist concept of probability Yet another way in which the primer is misleading is by omission, particularly by only including the following definition of the frequentist concept of probability: “A frequentist approach relates probability to the frequency of observing an event in a large number of experiments where the event may or may not occur.” Although this definition is correct, it omits a much more relevant definition of the frequentist concept of probability: an estimate of the proportion of members of a population who exhibit a particular property.10 The proportion in the population is estimated using the proportion of members of a sample who exhibit that property. The sample consists of members selected (ideally at random) from the population. The number of members in the sample is usually small relative to the total number of members of the population. This concept of probability, and extensions thereof, is widely applicable in forensic inference, e.g., in estimating the probability that a person selected at random from the relevant population would exhibit a particular allele at a particular locus in their DNA profile. Note that, contrary to the primer’s claims that “There is no single, correct probability for the possibility that an event will occur” and that “There is no ‘true value’ or ‘correct value’” for a likelihood ratio, under the frequentist conceptualization there is a true value for the proportion for a population, a value that is external to the mind of any person. In practice, one is seldom able to obtain the true value, which is why samples and statistical models are used to estimate the true value.11 Bayesian concept of probability In probability and statistics, the term “subjective” has a specific technical meaning. It designates the Bayesian conceptualization of probability as an expression of the degree of certainty that a person has as to whether an event has occurred (or will occur). In this subjectivist-Bayesian conceptualization, probability is a state of mind and it does not have a true value that is external to the mind of any person. Since, in the subjectivist-Bayesian conceptualization, probability is a state of mind, which can differ from person to person, it is described as “personal”. Some publications that promote subjectivist-Bayesian approaches emphasize that probabilities are subjective and personal (e.g., Taroni et al. [69], Taroni et al. [70], Berger & Slooten [71], Biedermann et al. [72], Biedermann et al. [73], Slooten & Berger [74]). Morrison [75] criticized such emphasis as unhelpful to the goal of promoting, to forensic practitioners and legal professionals who are not experts in probability and statistics, the use of likelihood ratios which are based on relevant data, quantitative measurements, and statistical models. Nordgaard [76] also suggested that insisting that forensic practitioners adopt a subjectivist-Bayesian conceptualization of probability would be counterproductive. Other than within the section on frequentist and Bayesian approaches, the primer assumes a subjectivist-Bayesian conceptualization of probability. For example, it states: “An individual’s uncertainty is expressed in the probability value” “Probability measures the strength of an individual’s belief” “Values between zero and one represent the uncertainty in the individual’s belief of a particular event occurring.” “Probabilities and LRs are personal and will vary depending on the data (empirical or personal belief), assumptions made, and calculation model used.” “Combinations of different assumptions or different levels of belief can result in the same probability or LR … Conversely, two people with the exact same information may not provide the same probability or LR.” Assignment of Bayes-factor values is not arbitrary Contrary to what a reader might reasonably infer from the primer (including from the quotations just given at the end of §4.6 above), forensic practitioners cannot legitimately assign any arbitrary value they desire to a Bayes factor. Bayes factors are calculated using prior probabilities, sample data, and statistical models. To satisfy requirements for transparency, when calculating a Bayes factor, a forensic practitioner must specify: •the statistical model that they will use, the statistical model that they will use, •the sample data that they will use, and the sample data that they will use, and •the prior probabilities that they will use. the prior probabilities that they will use. With respect to inference in general, not specifically forensic inference, Jaynes [77] p. 373 states (emphasis in original): “problems of inference are ill-posed until we recognize three essential things. (A) The prior probabilities represent our prior information, and are to be determined, not by introspection, but by logical analysis of that information. (B) Since the final conclusions depend necessarily on both the prior information and the data, it follows that, in formulating a problem, one must specify the prior information to be used just as fully as one specifies the data. (C) Our goal is that inferences are to be completely ‘objective’ in the sense that two persons with the same prior information must assign the same prior probabilities. If one fails to specify the prior information, a problem of inference is just as ill-posed as if one had failed to specify the data.” Reasonable informative prior probabilities could be based on existing data. Relatively uninformative priors, or reference priors, could also be considered reasonable. Debates about the best way to calculate likelihood ratios / Bayes factors There are debates among experts in forensic inference and statistics regarding the best way to calculate likelihood ratios / Bayes factors. This is a healthy part of science. One debate related to how to account for sampling variability (and potentially other sources of variability) when calculating likelihood ratios / assigning Bayes factors. The debate was captured in a series of papers: Taroni et al., [69], Nordgaard [76], Sjerps et al. [78], Taroni et al. [70], Morrison & Enzinger [79], Curran [80], Ommen et al. [81], Berger & Slooten [71], Biedermann et al. [72], van den Hout & Alberink [82], Taylor et al. [83], Dawid [84], Martire et al. [85], Biedermann et al. [73], Slooten & Berger [74], Morrison [75]. Much of the disagreement among participants in the debate was due to some participants adopting a frequentist conceptualization of probability and others adopting a subjectivist-Bayesian conceptualization of probability. This is a source of disagreement in the field of probability and statistics in general. It is not specific to forensic inference and statistics. Morrison [75] observed that, irrespective of frequentist or Bayesian approach, a common theme in the debate was a desire to reduce the probability of overstating the strength of evidence when the amount of case-relevant sample data available is small. In forensic casework, the amount of case-relevant sample data available is often small. Morrison [75] proposed that, irrespective of frequentist or Bayesian approach, reducing the probability of overstating the strength of evidence could be achieved by adopting methods that shrink the likelihood ratio / Bayes factor toward the neutral value of 1. Morrison & Poh [86] expanded on this proposal and provided examples of concrete methods, both frequentist and Bayesian. Although experts in forensic inference and statistics may disagree about the best way to calculate a likelihood ratio / assign a Bayes factor, the vast majority agree that calculating a likelihood ratio / assigning a Bayes factor is the correct way to assess strength of evidence. Validation Validation is essential If a judge or trier of fact has decided that a forensic practitioner has made reasonable subjective judgments at the beginning of the process of calculating a likelihood ratio (see §3 above), they should then consider whether the system that the forensic practitioner used to calculate the likelihood ratio has a sufficient level of performance.12 The level of performance of a system is assessed by testing, or validating, the system. Validation is required or recommended by international standards ISO/IEC 17025 [87] and ISO 21043-4 [1], and Daubert recommends that, when considering admissibility, the judge consider validation results: “Ordinarily, a key question to be answered in determining whether a theory or technique is scientific knowledge that will assist the trier of fact will be whether it can be (and has been) tested. … (‘[T]he statements constituting a scientific explanation must be capable of empirical test’)” “Additionally, in the case of a particular scientific technique, the court ordinarily should consider the known or potential rate of error” In its list of strengths of likelihood ratios, the NIST primer includes the ability to perform case-relevant validation of likelihood-ratio systems: “Can be evaluated for usefulness to a specific case when context information and performance data demonstrating how often this level of LR has occurred in both H1 and H2 true cases is provided” Validation procedures Assuming a source-level comparison, validation of a system for calculating likelihood ratios is achieved by entering a large number of pairs of items into the system, and, for each pair, obtaining the likelihood-ratio value that the method outputs. For some pairs, the two items must come from the same source, for other pairs, the two items must come from different sources. The better the performance of the system, the larger, on average, the likelihood ratios will be when the inputs are same-source pairs, and the smaller, on average, the likelihood ratios will be when the inputs are different-source pairs. For the validation results to be meaningful in the context of a case, the validation data must be representative of the relevant population for the case and must reflect the conditions of the questioned-source item and the known-source item in the case. If there is a mismatch between the conditions of the questioned-source item and the known-source item, one item in each pair in the validation data must reflect the conditions of the questioned-source item and the other item must reflect the conditions of the known-source item. Decisions as to whether the validation data are sufficiently representative of the relevant population for the case and sufficiently reflective of the conditions of the case require the forensic practitioner to make subjective judgments, and the practitioner must be transparent about those subjective judgments so that a judge or trier of fact can decide whether they are reasonable (see §3 above). If the judge during an admissibility hearing accepts that the validation data are sufficiently representative of the relevant population for the case and sufficiently reflective of the conditions of the case, they can then consider the validation results to decide whether the performance of the system that calculated the likelihood ratio is good enough to admit testimony that presents the likelihood ratio. If the trier of fact at trial accepts that the validation data are sufficiently representative of the relevant population for the case and sufficiently reflective of the conditions of the case, they can then consider the validation results to decide how much weight (if any) to give to the likelihood ratio presented in the forensic practitioner’s testimony. Data for validating a system must be separate from data used to train the system, including being separate from data used to train the calibration model (see §6.3 below). If the same data are used for training and for validation, the validation results will be overly optimistic compared to if the system is validated using data it has not previously seen. The observations made on the questioned-source and the known-source items in the case are data that the system has not previously seen. For details of recommended validation procedures, and of a recommended metric and a recommended graphic for representing validation results, see Morrison et al. [5] Consensus on validation of forensic voice comparison.13,14 Representations of validation results As stated in §5.2 above, the better the performance of a forensic-evaluation system, the larger, on average, the likelihood ratios will be when the inputs are same-source pairs, and the smaller, on average, the likelihood ratios will be when the inputs are different-source pairs. In Morrison et al. [5], the recommended performance metric is the log-likelihood-ratio cost (Cllr, Brümmer & du Preez [92]). Cllr values are greater than 0, and a well-calibrated system (see §6.2 below) will have a Cllr value of less than or equal to 1. The better the performance of the system, the lower the Cllr value will be. Some example Cllr values are provided at the end of this subsection. In Morrison et al. [5], the recommended graphic is a Tippett plot (Meuwly [93]).15 Example Tippett plots for same-source and different-source likelihood ratios output by perfectly calibrated forensic-evaluation systems (see §6.2 below) are shown in the right panels of Fig. 1. In each panel: •For the (blue) curve rising to the right, the value on the y axis represents the proportion of same-source likelihood-ratio values that are less than or equal to the likelihood-ratio value indicated on the x axis. For the (blue) curve rising to the right, the value on the y axis represents the proportion of same-source likelihood-ratio values that are less than or equal to the likelihood-ratio value indicated on the x axis. •For the (red) curve rising to the left, the value on the y axis represents the proportion of different-source likelihood-ratio values that are greater than or equal to the likelihood-ratio value indicated on the x axis. For the (red) curve rising to the left, the value on the y axis represents the proportion of different-source likelihood-ratio values that are greater than or equal to the likelihood-ratio value indicated on the x axis. Left panels: Examples of probability-density distributions of same-source and different-source likelihood ratios output by perfectly calibrated forensic-evaluation systems. Rightmost curve (blue): probability-density distribution of same-source likelihood ratios. Leftmost curve (red): probability-density distribution of different-source likelihood ratios. Right panels: Corresponding Tippett plots (cumulative-probability distributions). The performance of systems whose results are shown in lower panels is better than the performance of systems whose results are shown in higher panels. From top row to bottom row, the corresponding Cllr values are 0.85, 0.52, 0.24, and 0.09. The better the performance of the forensic-evaluation system, the further to the right and the shallower the same-source curve will be, and the further to the left and the shallower the different-source curve will be. As stated above, the better the performance of the system, the lower the Cllr value will be. From top row to bottom row, the Cllr values corresponding to the plots in Fig. 1 are 0.85, 0.52, 0.24, and 0.09. The system used does not have to be the best possible system In its list of limitations of likelihood ratios, the NIST primer includes the following: “Other models may be as ‘good’ or even ‘better’” This is misleading because it also applies to validation of any non-likelihood-ratio method for evaluating strength of evidence. It is also misleading because it appears to imply that, because a system might not be the best possible system, it would not be acceptable to use that system. Use of the best possible system is not what the law requires (Faigman et al. [94]). It may be possible to know that a system outperforms another particular system or several other particular systems. Each system could be validated using the same set of validation data that represent the relevant population for the case and reflect the conditions for the case. All else being equal, a forensic practitioner should use the system that performs the best. It is impossible to know, however, whether there might be some untested system that performs better than the system used by the forensic practitioner. In principle, allowing for variations in methods and for variation in samples used to train systems, there are an infinite number of systems, and it would be impossible to validate all possible systems that could potentially perform better. All else being equal, given a choice of systems, a forensic practitioner should use a system that they expect to perform well under the conditions of the case, and should not use a system that they expect to perform more poorly. We recommend that a forensic practitioner make the decision as to which system to use at the beginning of working on a case and that they then validate that system under conditions that reflect those of the case. In the context of working on a case, we recommend that the forensic practitioner not validate multiple systems and then choose the system that performs best. Doing so might give the impression that they are hunting for the system that produces the results that would be the most favorable to the instructing party.16 With regard to admissibility, Daubert does not advise the judge to consider whether the system used to evaluate strength of evidence is the best possible system, it advises the judge to consider whether the performance of the system is sufficient. Faigman et al. [94] discusses, from a legal perspective, the question of how good is good enough. Limited availability of case-relevant data In its list of limitations of likelihood ratios, the NIST primer also includes the following: “Is difficult to have sufficient data from known samples to evaluate the reasonableness of the LR values produced” It is true that limited availability of case-relevant data is a problem in forensic science. It is a problem for training statistical models used to calculate likelihood ratios, and a problem for validating systems that output likelihood ratios. The primer is misleading, however, because it implies that this is a problem specific to likelihood ratios. Limited availability of case-relevant data would also be a problem for training non-likelihood-ratio statistical methods for evaluating strength of evidence. It would also be a problem for validating non-likelihood-ratio methods irrespective of whether they were statistical or non-statistical. Assessment of validation results by legal-decision makers In its list of limitations of likelihood ratios, the NIST primer also includes the following: “An officer of the court will need to determine if an LR provided by an FSP aligns with their own evaluation of the performance data and their personal beliefs”17 Assuming that “performance data” means validation results (as it appeared to do in the quotation at the end of §5.1 above), we note that: •it is not a limitation that a judge at an admissibility hearing has to evaluate validation results in order to decide whether the performance of the system that calculated a likelihood ratio is good enough to admit testimony that presents that likelihood ratio, and it is not a limitation that a judge at an admissibility hearing has to evaluate validation results in order to decide whether the performance of the system that calculated a likelihood ratio is good enough to admit testimony that presents that likelihood ratio, and •it is not a limitation that a trier of fact has to evaluate validation results in order to decide how much weight (if any) to give to the likelihood ratio presented by the forensic practitioner. it is not a limitation that a trier of fact has to evaluate validation results in order to decide how much weight (if any) to give to the likelihood ratio presented by the forensic practitioner. These are regular aspects of judicial practice. The primer is misleading because they do not constitute a “limitation”. The primer is also misleading because judges and triers of fact having to evaluate validation results is not specific to likelihood-systems. It also applies to non-likelihood-ratio methods for evaluating strength of evidence. Lund & Iyer [95] presents models for how triers of fact might use validation results to assign weight to forensic practitioners’ strength-of-evidence statements, including both likelihood-ratio and non-likelihood-ratio expressions of strength of evidence. Whether the validation results support the presented likelihood ratio The NIST primer also states that: “Opposition to using probability and LRs in forensic science opinions focuses on … the use of LRs that do not align with performance data” Morrison [5] includes a hypothetical example in which validation results included likelihood-ratio values that ranged from 1/1,000 to 100, and the likelihood-ratio value calculated for the comparison of the questioned-source and known-source items was 10,000. This is an example in which the calculated likelihood-ratio is clearly not supported by the validation results (which is what we interpret “LRs that do not align with performance data” to mean). Such a result would likely be due to a mistake, which could, for example, be that the validation data are not sufficiently representative of the relevant population and/or not sufficiently reflective of the conditions of the case. Such a result should sound an alarm that something had gone wrong and that the calculated likelihood ratio should not be presented to the court. Contrary to what the primer implies, a competent forensic practitioner would not present the court with a likelihood ratio that did “not align with performance data”. Calibration What is a well calibrated set of scales? If a set of scales is perfectly calibrated, when an object is placed on the scale, the value of the readout equals the mass of the object placed on the scale. For a real-world set of well-calibrated scales, for objects whose masses are within the operating range of the set of scales, when an object is placed on the scale, the value of the readout approximately (within some acceptable tolerance) equals the mass of the object placed on the scale. If a set of scales is not well calibrated, the values of its readout are not accurate indicators of the masses of objects. What is a well calibrated likelihood-ratio system? For a perfectly-calibrated likelihood-ratio system, the likelihood ratio of the likelihood ratio that the system outputs equals the likelihood ratio that the system outputs: We illustrate what this means using graphical examples. Fig. 2 shows the distributions of same-source and different-source likelihood ratios for a perfectly calibrated system: •The rightmost curve (the blue curve) represents the distribution of same-source likelihood ratios, i.e., likelihood ratios that were output by the system in response to pairs of inputs in which, for each pair, both members of the pair came from the same source. The rightmost curve (the blue curve) represents the distribution of same-source likelihood ratios, i.e., likelihood ratios that were output by the system in response to pairs of inputs in which, for each pair, both members of the pair came from the same source. •The leftmost curve (the red curve) represents the distribution of different-source likelihood ratios, i.e., likelihood ratios that were output by the system in response to pairs of inputs in which, for each pair, the two members of the pair came from different sources. The leftmost curve (the red curve) represents the distribution of different-source likelihood ratios, i.e., likelihood ratios that were output by the system in response to pairs of inputs in which, for each pair, the two members of the pair came from different sources. Example of the probability-density distributions of same-source and different-source likelihood ratios output by a perfectly calibrated forensic-evaluation system. Rightmost curve (blue): distribution of same-source likelihood ratios. Leftmost curve (red): distribution of different-source likelihood ratios. Solid vertical line, circles, and dashed horizontal lines (green): calculation of a likelihood-ratio value of 10. The x axis represents likelihood-ratio values and the y axis represents relative likelihood. The greater the likelihood of obtaining a likelihood ratio at a value indicated on the x axis, the higher the curve will be and the higher the corresponding value on the y axis will be. In the example shown in Fig. 2, same-source likelihood-ratio values of around 100 are most likely, and different-source likelihood-ratio values of around 1/100 are most likely. Fig. 2 includes an example using a likelihood-ratio value on the x axis of 10 (i.e., the observations are 10 times more likely if the same-source hypothesis were true than if the different-source hypothesis were true). The relative likelihood for obtaining a likelihood-ratio value of 10 if the same-source hypothesis were true is 10 (p(10|Hsame)∝10), and the relative likelihood for obtaining a likelihood-ratio value of 10 if the different-source hypothesis were true is 1 (p(10|Hdifferent)∝1).18 Dividing the first relative likelihood by the second relative likelihood produces a likelihood ratio of 10: i.e., the likelihood ratio of the likelihood ratio that was output by the system equals the likelihood ratio that was output by the system. Because the system is perfectly calibrated, for any likelihood-ratio value on the x axis of Fig. 2, the likelihood ratio of the likelihood ratio that was output by the system equals the likelihood ratio that was output by the system. Using the same perfectly calibrated system as in Fig. 2, Fig. 3 shows an example using a likelihood-ratio value on the x axis of 100 (i.e., the observations are 100 times more likely if the same-source hypothesis were true than if the different-source hypothesis were true):19 Example of the probability-density distributions of same-source and different-source likelihood ratios output by a perfectly calibrated forensic-evaluation system. Rightmost curve (blue): distribution of same-source likelihood ratios. Leftmost curve (red): distribution of different-source likelihood ratios. Solid vertical line, circles, and dashed horizontal lines (green): calculation of a likelihood-ratio value of 100. Fig. 4 shows an example using a likelihood-ratio value on the x axis of ⅓ (i.e., the observations are 3 times more likely if the different-source hypothesis were true than if the same-source hypothesis were true): Example of the probability-density distributions of same-source and different-source likelihood ratios output by a perfectly calibrated forensic-evaluation system. Rightmost curve (blue): distribution of same-source likelihood ratios. Leftmost curve (red): distribution of different-source likelihood ratios. Solid vertical line, circles, and dashed horizontal lines (green): calculation of a likelihood-ratio value of ⅓. The left panels of Fig. 1 show the distributions of other examples of perfectly calibrated likelihood-ratio systems. For any likelihood-ratio value on the x axis of any of these panels, the likelihood ratio of the likelihood ratio that was output by the system equals the likelihood ratio that was output by the system. For a real-world well-calibrated likelihood-ratio system, the likelihood ratio of the likelihood ratio that the system outputs approximately equals the likelihood ratio that the system outputs. If a likelihood-ratio system is well calibrated, the likelihood-ratio values that it outputs are not arbitrary numbers, but are numbers that have intrinsic meaning. In order to understand that meaning, one must understand the hypotheses addressed, and know which properties of the items of interest were measured to obtain the observations. If a system that is purportedly a likelihood-ratio system is not well calibrated, the numbers that it outputs cannot be interpreted as meaningful quantifiers of strength of evidence in the context of the case. Publications describing what calibration is, calibration methods, and/or the importance of calibrating likelihood-ratio systems include: Brümmer & du Preez [92], González-Rodríguez et al. [96], Ramos & González-Rodríguez [97], Morrison [98], Brümmer et al. [99], Meuwly et al. [100], Vergeer et al. [101], Morrison [5], Morrison [102], Morrison [103]. How to calibrate a likelihood-ratio system A likelihood-ratio system can be calibrated by including an explicit calibration model as the last stage of the system. The penultimate stage of the system outputs uncalibrated likelihood ratios. The process of obtaining data for training the calibration model is similar to the process of validation (see §5.2 above). A large number of pairs of items are input to the system, and, for each pair, one obtains the uncalibrated likelihood-ratio value that the penultimate stage of the system outputs. For some pairs, the two items in the pair must come from the same source, for other pairs, the two items in the pair must come from different sources. The calibration data must be representative of the relevant population for the case and must reflect the conditions of the questioned-source item and the known-source item in the case. If there is a mismatch between the conditions of the questioned-source item and the known-source item, one item in each pair in the calibration data must reflect the conditions of the questioned-source item and the other item must reflect the conditions of the known-source item. If the calibration data are not sufficiently representative of the relevant population for the case or not sufficiently reflective of the conditions of the items in the case, the system will be miscalibrated. Decisions as to whether the calibration data are sufficiently representative of the relevant population for the case and sufficiently reflective of the conditions of the case require subjective judgments on the part of the forensic practitioner, and the practitioner must be transparent about these subjective judgments so that a judge at an admissibility hearing or trier of fact at trial can decide whether they are reasonable (see §3 above). Once the calibration model is trained, the model is applied to calibrate the uncalibrated likelihood ratios output by the penultimate stage of the system. The data used for validating the system must not be the same as the data used for calibrating the system. A popular calibration method uses logistic regression. Morrison [103] introduced and demonstrated bi-Gaussianized calibration, which results in likelihood-ratio values that are closer to perfect calibration than if logistic regression is used. Fig. 5 shows the distributions of same-source likelihood ratios and different-source likelihood ratios output by a forensic-voice-comparison system in response to validation data. Results are shown after logistic-regression calibration and after bi-Gaussianized calibration. The distributions for the corresponding perfectly calibrated system are also shown. Fig. 6 shows the corresponding Tippett plots. In Fig. 5 and Fig. 6, it can be seen that the likelihood-ratio results after bi-Gaussianized calibration are closer to perfect calibration than are the results after logistic-regression calibration. Examples of distributions of likelihood ratios after logistic-regression calibration and after bi-Gaussianized calibration. The distributions for the corresponding perfectly calibrated system are also shown. For all three sets of distributions, the corresponding Cllr value is 0.17. Examples of Tippett plots of validation results after logistic-regression calibration and after bi-Gaussianized calibration. The Tippett plot for the corresponding perfectly calibrated system is also show. For all three Tippett plots, the corresponding Cllr value is 0.17. Calibrated likelihood-ratio values are not arbitrary numbers In its list of strengths of likelihood ratios, the NIST primer includes the ability to calibrate systems that calculate likelihood ratios: “Can be evaluated and calibrated, for consistency (over or understatement) with known samples Can be limited (e.g., a lower and upper limit) to that supported by available data” However, the primer also states: “one person’s probability of 80% may not mean the same as another person’s probability of 80%” The context implies that this would apply to likelihood ratios as well as to probabilities, e.g.: “one person’s [likelihood ratio of 4] may not mean the same as another person’s [likelihood ratio of 4]”. Contrary to what the primer implies, calibrated likelihood-ratio values are not arbitrary numbers. If two forensic practitioners each calculate a calibrated likelihood-ratio value of 4, one’s calibrated likelihood ratio of 4 cannot have a different meaning from the other’s calibrated likelihood ratio of 4. Because they are both calibrated, they both mean the same thing: they both mean that the probability of obtaining the observations made on the questioned-source and known-source items is 4 times greater if the same-source hypothesis were true than if the different-source hypothesis were true. Communication The NIST primer states that: “Opposition to using probability and LRs in forensic science opinions focuses on … misunderstanding of the expression.” It is true that people who are laypersons with respect to forensic inference and statistics have been found to have difficulty understanding the meaning of likelihood ratios. For reviews of the literature on this topic see: Martire [104], Thompson [105], Eldridge [106], Martire & Edmond [107], Morrison et al. [108]. The solution to this problem, however, is not to abandon the use of likelihood ratios in favor of some non-likelihood-ratio approach that is not logically correct. As outlined in Morrison et al. [108], the solution to this problem should be sought by conducting better-designed and more systematic research aimed specifically at addressing the question of how best to present likelihood ratios so as to maximize their understandability for legal-decision makers. Rather than helping judges and attorneys to better understand the meaning of likelihood ratios, the primer’s presentation of misleading information about the likelihood-ratio framework is detrimental to this goal. Practices inconsistent with the forensic-data-science paradigm In its list of strengths of likelihood ratios, the NIST primer includes the ability to assign likelihood ratios without using relevant data: “Can be assigned with or without empirical data; therefore, there is no limitation to when an LR can be provided” This positive statement on assignment of likelihood ratios without the use of empirical data seems to conflict with the primer’s overall negative portrayal of subjectivity. There are publications which advocate assignment of likelihood ratios based only on subjective judgment (e.g., Willis et al. [13], Marquis et al. [109]), but this practice has been criticized in other publications (e.g., Risinger [50], Morrison et al. [110]). Risinger [50] objects to: “the abuse of the notion of subjective probability to allow people to render opinions, and develop likelihood ratios, too, on what is sometimes no better than a guess”; and “the generation of likelihood ratios with numbers from nowhere that tends to cover up the weakness of the ingredients”.20 Morrison et al. [110] states: “we believe that ‘Much resistance to the adoption of the likelihood ratio framework is not to the idea of assessing the relative probabilities (or likelihoods) of the evidence under prosecution and defence hypotheses per se, but to what is perceived as unwarranted subjective assignment of those probabilities’” Although assigning the numerator for a likelihood ratio using only subjective judgment, assigning the denominator for a likelihood ratio using only subjective judgment, and dividing the former by the latter is consistent with the logic of the likelihood-ratio framework, this practice is inconsistent with the forensic-data-science paradigm: it is highly susceptible to cognitive bias, it is not transparent (and therefore it is not reproducible), and it is practically difficult to calibrate and validate. Some forensic practitioners: •calculate likelihood ratios using data that do not represent the relevant population; and/or calculate likelihood ratios using data that do not represent the relevant population; and/or •calculate likelihood ratios using data that do not reflect the conditions of the case at hand; and/or calculate likelihood ratios using data that do not reflect the conditions of the case at hand; and/or •do not calibrate their likelihood-ratio systems; and/or do not calibrate their likelihood-ratio systems; and/or •do not validate their likelihood-ratio systems, either not at all, or not using appropriate data and procedures. do not validate their likelihood-ratio systems, either not at all, or not using appropriate data and procedures. Examples are described in: Morrison & Thompson [111], Morrison [68], Morrison [111], Morrison [112]. Such practices are inconsistent with the forensic-data-science paradigm. Criticism of implementations of the likelihood-ratio framework that are inconsistent with the forensic-data-science paradigm is appropriate, but such criticism does not impeach the likelihood-ratio framework itself. The fact that some, many, or even most forensic practitioners implement versions of the likelihood-ratio framework that are inconsistent with the forensic-data-science paradigm is not a reason to reject use of the likelihood-ratio framework. The appropriate course of action is to replace such implementations with implementations that are consistent with the forensic-data-science paradigm. Conclusion A paradigm shift in evaluation of forensic evidence is in progress. The new paradigm (the forensic-data-science paradigm) uses methods based on relevant data, quantitative measurements, and statistical models. Its methods are transparent and reproducible, are resistant to cognitive bias, are logically correct (i.e., they calculate likelihood ratios), and are calibrated and validated under casework conditions. With respect to the likelihood-ratio framework, the paradigm shift has progressed to the point where its use is advocated by the vast majority of experts in forensic inference and statistics, and also by key organizations around the world. Paradigm shifts, however, usually face concerted and long-lasting opposition. An instance of such opposition is the NIST primer on “Probability and Likelihood Ratios”. That primer gives a misleading negative portrayal of the likelihood-ratio framework. The present perspectives paper has refuted that portrayal. Many forensic practitioners still do not use the likelihood-ratio framework, and among those who do, many, or even most, still do not implement the likelihood-ratio framework in a manner that is consistent with the forensic-data-science paradigm. As described in Kuhn [27], it can take a long time before a paradigm shift is complete and all scientists in the field are then practicing under the new paradigm. Despite the progress made over approximately the last half century, we expect that it will still take a long time before the paradigm shift is complete. In the meantime, we expect resistance to the paradigm shift to continue, and refutation of anti-likelihood-ratio arguments to continue to be necessary. Disclaimer All opinions expressed in the present paper are those of the author, and, unless explicitly stated otherwise, should not be construed as representing the policies or positions of any organizations with which the author is associated. Declaration of competing interest The author declares that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgments Thanks to David H Kaye, Philip Weber, Jessica Dixon, and two anonymous reviewers for comments on earlier drafts of this paper. Any shortcomings in the final version of the paper are entirely the author’s responsibility. https://www.google.com/search?q=subjective+definition, retrieved 2026-01-01. Cambridge Academic Content Dictionaryhttps://dictionary.cambridge.org/dictionary/english/subjective, retrieved 2026-01-01. Daubert v. Merrell Dow Pharm. Inc., 509 US 579 (1993). Frye v. United States, 293 F. 1013 (D.C.Cir.1923). https://doi.org/10.1093/OED/1098712908, retrieved 2026-01-12. https://doi.org/10.1093/OED/3908623815, retrieved 2026-01-12. In some contexts, those subjective judgments are made by individual forensic practitioners on a case-by-case basis. In other contexts, those subjective judgments are made beforehand, and the resulting choices are embedded in standard operating procedures that individual forensic practitioners follow when working on cases. Even in the latter context, practitioners may still have to make subjective judgments to decide which standard operating procedure is appropriate to use in which case. Kalafut et al. [64] and Perlin et al. [65] respond to Thompson [63]. Thompson [66] and Thompson [67] reply to those responses. This demonstrates that, in a process in which subjective judgments are made before calculating a likelihood ratio, those subjective judgments can be debated. A “Bayes factor” is the Bayesian analog of a frequentist (or likelihoodist) “likelihood ratio”, but, in the literature on forensic inference and statistics, the term “likelihood ratio” is often used for both. Advocates of subjectivist-Bayesian approaches prefer to use “assign” rather than “calculate”. Bayes factors are, however, the output of explicit calculations. The definition given in the primer could be interpreted to include this definition, but it is highly unlikely that a reader who is unfamiliar with probability and statistics would interpret it that way. True values and estimates are distinguished using the terms “population parameters” and “sample statistics” respectively. We define a “system” as a specific implementation of a specific set of methods and procedures that are used to assess strength of evidence, e.g., a specific set of methods and procedures that are used to calculate a likelihood ratio. These include methods and procedures for analysis, i.e., making observations (measurements) of the properties of the items of interest, and methods and procedures for interpretation, i.e., calculating likelihood ratios which quantify the probability of obtaining the observations if the same-source hypothesis were true divided by the probability of obtaining the observations if the different-source hypothesis were true. For a system that calculates likelihood ratios using relevant data, quantitative measurements, and statistical models, the specific implementation would include software that implements specific statistical models. Although, in Morrison et al. [3], the scope of application of its recommendations is stated to be forensic voice comparison, only minor wording changes would be required for the recommendations to be applicable across many branches of forensic science. For broadly compatible, but somewhat different, perspectives on validating likelihood-ratio systems, see Meuwly et al. [88] and Ramos et al. [89]. A key difference from Morrison et al. [3], is that Meuwly et al. [88] and Ramos et al. [89] include additional metrics and graphics. For published responses to Meuwly et al. [88], see Alberink et al. [90] and Aitken [91]. The format for Tippett plots recommended in Morrison et al. [3] differs from the original Tippett-plot format in Meuwly [93]. In Meuwly [93], both same-source and different-source curves rose to the left and the value on the y axis represented the proportion of likelihood-ratio values that were greater than or equal to the likelihood-ratio value indicated on the x axis. Also, if one uses a dataset to test multiple systems and then picks the best-performing system on that dataset, one should not then present those test results as indicative of the performance of that system. Because they have been used to choose the “best” system, those results tend to be biased. If one wants a fair indicator of the performance of the chosen system, one should test that system using a new, independent, dataset. In this quotation, we do not know how to interpret the words regarding the officer of the court’s personal beliefs. We use “∝” (proportional to) rather than “=” (equals) because the values given are relative likelihood values rather than absolute likelihood values. For calculating likelihood ratios, the relative values of the likelihoods are what matter, absolute likelihood values are not needed. The relative scaling for Fig. 2 was set so that the math for the calculation of the likelihood ratio would be easy. The relative likelihood values on the y axes of Fig. 3 and Fig. 4 have been linearly rescaled relative to the y axis on Fig. 2. The values were linearly rescaled to make the math for the calculation of the likelihood ratios easier. The linear rescaling has no effect on the calculated likelihood-ratio value. However, Risinger [51] also states: “it is my impression that forensic scientists in the [likelihood-ratio] pool often seek and value real data on issues before them more than similarly situated [non-likelihood-ratio] practitioners”. References