Design and Analysis Strategies for Pooling in High Throughput Screening: Application to the Search for a New Anti-Microbial Byran J. Smucker, Benjamin Brennan, Emily Rego, Meng Wu, Zhihong Lin, Brian M. M. Ahmer, Blake R. Peterson Abstract A major public health issue is the growing resistance of bacteria to antibiotics. An important part of the needed response is the discovery and development of new antimicrobial strategies. These require the screening of potential new drugs, typically accomplished using high-throughput screening (HTS). Traditionally, HTS is performed by examining one compound per well, but a more efficient strategy pools multiple compounds per well. In this work, we study several recently proposed pooling construction methods, as well as a variety of pooled high-throughput screening analysis methods, in order to provide guidance to practitioners on which methods to use. This is done in the context of an application of the methods to the search for new drugs to combat bacterial infection. We discuss both an extensive pilot study as well as a small screening campaign, and highlight both the successes and challenges of the pooling approach. KEYWORDS: experimental design, supersaturated experiment, regularized regression, antibiotic-resistant bacteria 1 Introduction Pathogens are becoming increasingly resistant to antibiotics, and this fact has alarming implications for public health including increased mortality and associated healthcare costs ( 23 ; 13 ) . The United Nations estimates that deaths due to antimicrobial resistance will be 10 million per year by 2050 ( 3 ) . For a specific example, the typhoidal strains of Salmonella enterica cause typhoid fever resulting in more than 100,000 deaths per year ( 14 ; 1 ) , and non-typhoidal strains are a leading cause of death from food-borne illness in the United States ( 18 ) and a contributor to the mortality in developing nations due to diarrhea ( 10 ; 17 ) . Because of the surge in antibiotic-resistant typhoidal and non-typhoidal Salmonella enterica strains, the CDC and the WHO have called for the development of new drugs to address this need ( 23 ) . In response to this problem, a new anti-microbial strategy has been suggested ( 19 ) with potential in many bacteria including Salmonella enterica . An important part of developing that strategy is drug discovery, and the initial phase of this search is accomplished via high-throughput screening (HTS). High-throughput screening is used widely in drug discovery, chemical biology, and many other scientific and industrial domains. HTS involves the placement of specified compounds into wells in 384- or even 1536-well plates. The plates are then assayed and checked for a desired indication. Normally, a single compound is placed in each well, but various authors have suggested and shown that pooling multiple compounds in each well can improve throughput and/or statistical efficiency. Pooling has been controversial and contested, with both successes ( 26 ; 21 ; 15 ; 4 , e.g.,) and cautions ( 2 ; 5 ) in the literature. Recently, however, there have been reports of several successful pooled screening procedures ( 7 ; 12 ; 20 ) . Older methodologies, using approaches like orthogonal pooling ( 9 ) or poolHiTS ( 8 ) , neither constructed their pools nor analyzed them using statistical methods, while the new approaches use statistical design ideas for pool construction ( 7 ; 20 ) and statistical regularization for analysis ( 7 ; 12 ; 20 ) . In this work, we add to this emerging pooling literature in several ways. First, we describe in some detail a particular application related to a search for antimicrobials discussed at the outset (Section 2 ). In Section 3 , we describe a number of existing pool construction methods and pooled HTS analysis methods, and make a set of extensive comparisons between them in Section 4 . Most of the methods we consider are from the literature, but we propose a new Lasso thresholding method that exploits our knowledge of effect directions. We also discuss a secondary analysis method to address a problem in these types of screens: they often produce too many false positives, which consume a large amount of resources. The secondary criterion severely reduces the number of compounds to be validated, while still detecting large effects. In Section 5 we then present an extensive description of the pilot study (Section 5.1 ) used to establish pooling as a viable approach in this setting, as well as results from an initial screen (Section 5.2 ) which identified several promising compounds while reducing the number of considered false positives. We finish with a Discussion in Section 6 . 2 The Screening Problem Schwieters et al. ( 19 ) report that the enzyme mannitol-1-phosphate 5-dehydrogenase (MtlD) offers antimicrobial potential in many bacteria. This enzyme converts the compound mannitol-1-phosphate to fructose-6-phosphate. When mtlD mutant bacteria are provided mannitol, mannitol-1-phosphate accumulates, intoxicating the bacterium leading to reduced growth and attenuated virulence in animal models. Thus, the goal is to identify a compound that inhibits MtlD in the presence of mannitol. Using a wild-type bacterium, inhibition of MtlD in the presence of mannitol will result in lack of growth. However, numerous compounds will inhibit the growth of a wild-type bacterium for reasons unrelated to MtlD inhibition. To eliminate these from consideration, a parallel screen is used in which the bacterium cannot be harmed by a MtlD inhibitor (because it is a mtlA mutant and cannot form mannitol-1-phosphate). Thus, the screening problem is to find a drug which inhibits growth of the wild-type (WT) bacteria but not of the mtlA mutant (MUT) when assayed against both. We call such a drug a true hit . A drug which inhibits both WT and MUT is called a pseudo-hit . Our goal is to identify true hits. Previous single-replicate, one-compound-one-well (OCOW) screening of 10,000 compounds for this system yielded 140 that appeared to inhibit WT, and 40 of those did not appear to inhibit MUT. However, upon retest in duplicate, none of the 140 were validated as true hits. Contemplating a similar screen scaled up to hundreds of thousands of compounds, the investigators realized that even a false positive rate of 1% would be extremely expensive. This led them to consider pooling as a more efficient alternative, since it is a way to observe each compound more than once while using more compounds than wells. This yields the potential of larger true positive rates with the same or smaller false positive rates. The researchers agreed that eight compounds per pool was feasible. From their previous testing, they expect the set of true hits to be extremely sparse within the space of compounds they anticipate searching. Statistically, this does have an advantage: Interactions between compounds are not expected, unless two or more true hits end up in the same pool. Because of the level of sparsity, this is unlikely. Despite its promise, pooling also presents challenges. Logistically, the compounds, stored in source plates, must be transferred to target plates in pools dictated by the experimental design. This is accomplished via automated liquid handling machines, but constructing the pooled plates is still more time-consuming than OCOW. It also requires careful preparation and communication between the statistician and the personnel implementing the design. We revisit this screening problem in Section 5 . 3 Design and Analysis Methodologies In addition to demonstrating an application of pooling in HTS and the associated data and statistical challenges, two additional objectives of this work are to (1) compare pooled HTS design strategies; and (2) compare pooled HTS analysis strategies. A basic question we seek to learn about: If one wishes to undertake a pooled high-throughput screen, what design and analysis method will be most effective? In this section, we describe the designs and analysis methods we consider. Throughout, we use the designs and analysis methods to estimate the main effects model, which we assume to be true: \displaystyle\mathbf{y}=\beta_{0}\mathbf{1}+X\bm{\beta}+\bm{\epsilon}, (1) where \mathbf{y} is n\times 1 , X\in\{-1,1\}^{n\times k} , \bm{\beta}=(\beta_{1},\ldots,\beta_{k})^{T} and \bm{\epsilon}\sim N(\mathbf{0},\sigma^{2}I) with \mathbf{0} an n -vector and I the n\times n identity matrix. 3.1 Designs Following the model above, we assume n wells to study k compounds, with n\mu+r\sigma , where \mu is the average response for a well with no active compounds, r is a user-specified value, and WLOG we assume positive effects are of interest. In our case, we considered p_{s}\in\{0.75,1\} and r\in\{2,3\} . Practically, for the k=640 CRowS design for which each compound appears in 4 wells, this means that we considered four secondary criteria: “3 Wells > 2 SD (above \mu )”, “4 Wells > 2 SD”, “3 Wells > 3 SD”, and “4 Wells > 3 SD”. We did a simulation of this secondary analysis strategy, using CRowS with P\equiv\lambda -specific Gauss-Lasso using \tau_{\lambda}=\max(\hat{\beta}_{\lambda}) . The simulation settings were the same as with the previous simulations in Section 4 ; that is, with k\in\{500,640,960,1280\} and \beta\in\{1,2,3,4\} . We show TPR and FPR for the secondary criteria compared with the results of P without a secondary criterion in Figure 3 . As expected, both the TPR and FPR plummet as the secondary criteria get more strict. If we focus on k=640 , which is the screen size we ended up using in the real experiments, we see that only the most strict “4 Wells > 3 SD” secondary criterion have unacceptably low power for the largest effect. However, this highlights that these secondary criteria are really only viable if the experimenter is willing forgo the detection of small or even medium-sized effects. But the advantage is also important: the number of false positives is much more manageable. With a FPR of 0.0015 as in the analysis with no secondary criterion, investigators would be chasing 15 false positives for a 10,000 compound screen and 1,500 for a million-compound screen. But with the “3 Wells > 3 SD” secondary criterion with an FPR of around 0.00006 , one would expect less than 1 and 60 false positives, respectively. Figure 3 : Comparison of FPR and TPR in CRowS designs for differing secondary criterion, using the \lambda -specific Gauss-Lasso with \tau_{\lambda}=\text{max}(\hat{\beta}_{\lambda}) primary thresholding criterion. 5 Designing and Analyzing the Anti-Microbial Screens In this section, we describe the process by which our screens were designed, executed, validated, and used to identify promising compounds for the MtlD system. First, we conducted several proof-of-concept screens to ensure that our methods could detect known hits, for different levels of screening boldness. Then, we report results of a small screening campaign the goal of which was to identify new compounds that inhibit MtlD. Recalling Section 2 , each plate must be measured using a MUT and WT assay in order to find a drug that inhibits the WT but not the MUT. We used the basic methodology described in ( 20 ) to construct CRowS designs, then used the Lasso to obtain lists of potential hits. Details of our analysis are provided in the results below, but ultimately we used the \lambda -specific Gauss-Lasso along with the secondary criteria of Section 4.4 to construct hit lists. Note that we report on actual experiments conducted before we did the extensive testing described in Sections 3 and 4 . Thus, our recommendations from the comparative work may not map precisely onto what we actually describe in this section. The purpose of reporting these results is to demonstrate a high-quality solution to the screening problem presented in Section 2 , while providing insight and guidance for other HTS applications. 5.1 Proof-of-Concept Screening Results A first step when considering pooling for a particular assay and for the type of compounds to be screened is to ensure that the pooling method can recognize known hits. This is important because it is possible for pooled HTS to fail. If there are large and/or numerous interactions, the ability to identify main effects may be degraded ( 20 , see results referenced in the Discussion of) . There have also been reports of compounds reacting together in wells in therapeutically uninteresting ways ( 5 , e.g.,) . Thus, for the proof-of-concept screens described in this section, we spiked the screens with a known pseudohit (Carbenicillin, a drug known to hit both the WT and MUT assays, denoted CntP1A03 ), as well as a known true hit (Chloramphenicol, a drug known to hit WT but not MUT, denoted CntP1A04 ). We used compounds from the MedChem Express Diversity library (5,000 Scaffold Library; Cat. No.: HY-L902) We also wished to investigate the extent to which our screening method could be successfully stretched. This led us to executing the following pooled HTS experiments, using CRowS designs: • (n=320,k=500,c=8) , with a_{\text{min}}=5 and a_{\text{max}}=6 . That is, each compound appears in either 5 or 6 pools. • (n=320,k=640,c=8) , with a=4 . • (n=320,k=960,c=8) , with a_{\text{min}}=2 and a_{\text{max}}=3 . • (n=320,k=1280,c=8) , with a=2 . The results of these experiments were promising. Examining the Lasso profile plots, along with the data broken out by the spiked compounds of interest, showed that pooling was working in both the k=500 and k=640 settings. Figures 4 and 5 show these plots for (n=320,k=640,c=8) . In Figure 4 observe that CntP1A03 is annotated as one of the most prominent among the profiles in both the WT and MUT assay, while CntP1A04 only appears prominent in the WT. This suggests, correctly, that CntP1A03 is a pseudohit and that CntP1A04 scores as a true inhibitor. This is visually confirmed in Figure 5 , where CntP1A03 clearly inhibits both WT and MUT, but CntP1A04 inhibits only WT. In order to reduce the subjectivity in these judgments, we used the \gamma -specific Gauss-Lasso described in Section 4.3 , with r=0.9 . Using this procedure, we found three compounds that hit WT but not MUT, including ControlP1A04 , as expected. The two false positives are consistent with a false positive rate of about 0.3%. We found similar results for (n=320,k=500,c=8) , with the profile plots, data, and \gamma -specific Gauss-Lasso all identifying ControlP1A04 as an inhibitor, and the latter procedure identifying in addition four presumed false positives. The results from the more aggressive (n=320,k=960,c=8) and (n=320,k=1280,c=8) experiments were also promising, but more ambiguous. Visually, the profile plots yielded no hint of the expected pseudohit and true hit, and the \gamma -specific Gauss-Lasso failed to identify the compounds. However, plotting the compounds separately from the rest of the data for the experiments still suggested that CntP1A03 inhibited both, while CntP1A04 inhibited just the WT. Still, with just 2 or 3 pools per compound, there wasn’t enough statistical information to clearly identify them. We have included the plots for the (n=320,k=1280,c=8) experiment in the Supplementary document. Based on these results, we moved forward to the screening stage with plans to use the (n=320,k=640,c=8) design. Figure 4 : Lasso profile plots for (n=320,k=640,c=8) proof-of-concept CRowS design. The plot annotates the top 10 compounds in terms of magnitude at the smallest \lambda . Figure 5 : Plots of the data for (n=320,k=640,c=8) proof-of-concept CRowS design. The control wells consist of Carbenicillin as a positive control and DMSO (no compounds) as a negative control. 5.2 Screening Results Following the proof-of-concept experiments, we conducted a small-scale screen consisting of 16 plates, called PLINGS. The compounds used in this screen were from ChemBridge Macrocycle Library (10,000 Macrocycles; 2018 version; N1558-1) As per the test screens, each PLING included 640 unique compounds, which means the total number of compounds represented in this initial screen is 10,240. For this system, the expectation is that true hits may be as rare as 1 in 100,000. Thus, there is no certainty that in this first screen we will find a true hit. Still, we present the results that we have obtained to this point. In the Supplementary Document, we have included the boxplots from the PLINGS. Predictably, with real data there are ambiguities. For instance, it is clear that in some plates there are a relatively small number of wells that demonstrate inhibition, and in others there are indications of potential activation. Also, some plates have more apparently inhibited wells than would be expected in such a sparse system. Finally, the MUT vs. WT were expected to have similar distributions, but in reality the MUT assay is consistently centered lower than the WT assay (This is certainly due to the WT utilizing mannitol in the medium while the MUT cannot. This could be remedied in future assays by reducing the concentration of mannitol). Despite these data challenges, each plate was analyzed visually via profile plots. Though we don’t include the profile plots from all of the PLINGs, in Figure 6 we provide four: (a) In PLING2 there is a promising compound S01F010 that appears to inhibit in WT but not in MUT; (b) in PLING3 there are no clear inhibitors for either assay; (c) in PLING7, there appear to be three pseudohits, S02H008 , S01D009 , and S01D022 , because they hit on both assays; and (d) in PLING11 there are several possible true hits, S02N020 , S02E007 , and S01M005 , which show up on WT but not MUT. a b c d Figure 6 : Four profile plots. The preceding analysis is relatively subjective, so we use the \lambda -specific Gauss-Lasso with r=0.9 to provide an objective hit list. To review, this Gauss-Lasso procedure sets any Lasso estimate to 0 if the estimate is greater than -0.9\times\max{\hat{\beta}_{\lambda}} . That is, the only estimates that survive are the ones that are relatively large and negative. This is done for both the WT and MUT assays, for each of the PLINGs separately. Then, for each PLING, we take an initial hitlist to be those that hit on WT but not on MUT. Using the 90% \lambda -specific Gauss-Lasso yields 1.1% of the studied compounds as potential hits. Though this seems to be a reasonable hit rate—even if most of them are false positives—but even in this small initial screen there are more than 100 hits identified, and if scaled to hundreds of thousands of compounds such a hit rate would place a tremendous burden on secondary screens. In order to further screen the initial hitlists, we search among the potential hit compounds for those whose pools consistently exhibited substantial inhibition, using the method described in Section 4.4 . For the (n=320,k=640,c=8) design, each compound appears in four wells, so we looked for promising compounds with three or four pools more than 2 or 3 standard deviations away from the median of the pools not containing the compound under consideration. To estimate the standard deviation, we used the robust estimator of 1.48\times MAD , where MAD is the median absolute deviation of the remaining points. This was done in case other hit compounds exist in the remaining pools. Table 3 identifies 8 unique compounds, arbitrarily named, that meet the 2 SD secondary criteria, which is that at least three of its four pools inhibit by 2 or more SDs. It also includes, naturally, the two compounds that meet the 3 SD secondary criteria. The secondary criteria eliminates the issue of compounds chosen by the method due to one or two outliers. Of these 8, we show in Figures 7 and 8 two compounds for which all four wells are 2 SD hits, as well as the two additional compounds which have three of four wells beyond 3 standard deviations. This methodology allows the researchers to visually focus on the very most promising compounds. PLING Compound Number Beyond 2 SDs Number Beyond 3 SDs 1 S01I013 3 0 5 S01E012 3 3 5 S01L010 3 2 6 S02B003 3 3 7 S01G022 4 0 7 S02O006 3 0 8 S01K008 3 2 11 S02N020 4 2 Table 3 : For the initial screen, the eight compounds that met the 2 SD secondary criterion, including the two compounds that met the 3 SD secondary criterion. In this screen, each compound appears in four pools. a Promising compound PLING7-S01G022 with all four of its pools beyond two standard deviations from the median for WT assay, while not inhibiting the MUT assay. The solid line for the WT pools is an estimate of the median; the dashed line is an estimate of 2 SD below the median. b Promising compound PLING11-S02N020 with all four of its pools beyond two standard deviations from the median for WT assay, while not obviously inhibiting the MUT assay. The solid line for the WT pools is an estimate of the median; the dashed line is an estimate of 2 SD below the median. Figure 7 : Two promising 2-SD compounds. a Promising compound PLING5-S01E012 with three of its four pools beyond three standard deviations from the median, though is appears that two of its pools also inhibit the MUT assay. The solid line for WT is an estimate of the median; the dashed line is an estimate of 3 SD below the median. b Promising compound PLING6-S02B003 with three of its four pools beyond three standard deviations from the median of the WT assay. The solid line is an estimate of the median for WT; the dashed line is an estimate of 3 SD below the median. Figure 8 : Two promising 3-SD compounds. So far, none of the promising compounds have been validated as true hits, but this is not unexpected in a system anticipated to be as sparse as this. 6 Discussion In this work, one major contribution is to provide practitioners of pooling in HTS guidance regarding pool construction methods and screen analysis methods. Along the way, we introduce two improvements to previously-described analysis methods that align with our goal of reducing false positives. The second contribution of our work is to apply these methods to a real problem in search of a solution. This problem is the resistance to antibiotics that has progressed and a possible solution is the inhibition of an enzyme that neutralizes the toxic accumulation of the compound mannitol-1-phosphate in many bacteria. We show how the designs and analysis methods we studied can be used in practice, with both an extensive pilot study—in which controls are included—and a small-scale screening campaign. Such an extensive set of experiments provides insights into the challenges and compromises that must be faced in real world data. Regarding the comparisons of design methods, we compared several recent pool construction strategies and concluded that the CRowS approach of ( 20 ) offers a good combination of effectiveness and flexibility. We note that when the design parameters allow balance in terms of pool sizes and compound replication, it appears that all three methods we compared perform similarly. Of course, we are limited in the generality of our conclusion by the specificity of our simulation scenarios. But we would tentatively state that for such balanced cases, there is likely little difference between CRowS, MAPS, and the randomly constructed pools. In fact, we conjecture that for those balanced cases, “random” pools are sufficiently constrained to yield CRowS designs, though we have not proved this. We also compared a number of methods to analyze pooled HTS data and concluded that for the present application—in which it is critical to reduce the number of false positives to near zero—a new \lambda -specific Gauss-Lasso method provides the best balance of TPR and FPR. This method takes advantage of the fact that we know that any true effect will be inhibitory. We also described an additional, secondary analysis method that further reduces the number of false positives. For the analysis method comparisons, the preferred method strongly depends on the balance the experimenter wishes to strike between true and false positive rate. The \lambda -specific Gauss-Lasso, with its relatively low FPR but also lower TPR, aligns with the need of our application. But if it is important to identify more and smaller effects, the Elastic Net or the Gauss-Lasso with \tau=0.5\times\text{max}(|\hat{\beta}_{\lambda=0}|) . A critical assumption in pooling is that statistical interactions between compounds are not numerous nor large. ( 20 ) briefly investigated this and showed that interactions can have a large impact on whether hits are detected. There is also concern regarding promiscuous aggregation ( 5 ) of compounds, in which molecules combine in unpredictable and unhelpful ways and prevent clean deconvolution of individual compound effects. It is beyond the scope of this work to investigate these things further, but the present work has demonstrated at least that strong hits are detectable in real-world, sparse settings such as the one investigated herein. Supplementary Material Supplementary Material consists of: A Supplementary document referred to in the main document. [Available at the end of this document] B Pooled and control data from the pilot experiments of Section 5.1 , along with pooled and control data from the screening experiments of 5.2 . [Unavailable in arXiv version] Funding BRP, MW, and ZL thank the OSU Comprehensive Cancer Center (2P30 CA016058) for financial support. MW acknowledges the NIH support of R50 CA243786. ER was supported by PHS grant NIH T32 AI165391. Conflicts of Interest The authors declare no conflicts of interest. Data Availability Statement The pooled data from Section 5 is available in the Supplementary Material. Raw data, control data, and/or codes for data processing and simulations are available upon request of the authors. Supplementary Document CRowS vs. Orthogonal Pooling Here we report results of CRowS vs. Orthogonal Pooling (Figure S1 , where CRowS is analyzed using the Gauss-Lasso with \tau_{\lambda}=\text{max}(\hat{\beta}_{\lambda}) while the Orthogonal Pooling design is analyzed using a traditional, non-statistical modeling method described below. Figure S1 : Comparison of FPR and TPR between CRowS and Orthogonal Pooling designs using the \lambda -specific Gauss-Lasso with \tau_{\lambda}=\text{max}(\hat{\beta}_{\lambda}) primary thresholding criterion for CRowS. For orthogonal pooling designs, the data was generated in a manner identical to our primary simulations, with effect sizes \beta=(1,2,3,4) and error \sigma^{2}=1 . The simulations were carried out with 10,000 iterations, and, at each iteration, a compound was designated a hit if the values of both observed wells ( y_{i},y_{j} ) in which the compound was present were greater than the 95^{th} percentile ( y_{0.95} ) of all observed well values. Proof-of-concept Screening Results for (n=320,k=1280,c=8) Here we provide a plot (Figure S2 ) of the results of the (n=320,k=1280,c=8) proof-of-concept screen. Though there is some visual evidence that the screen identified the pseudo-hit and true hit, we were unable to verify this using the Gauss-Lasso modeling. Figure S2 : Representation of the amount of cell growth for 32 WT Controls, the 320 WT pooled wells, the 320 MUT pooled wells, and the 32 MUT Controls. The pooled wells studied 1,280 compounds. The wells are colored by whether they included the pseudo-hit (Carbenicillin) or the true hit (Chloramphenicol). For the Controls, the Carbenicillin wells included only Carbenicillin; for the pooled wells, each well included 8 different compounds. Screen Results In Figure S3 we provide boxplots from the small screen described in Section 5.2 in the paper. Figure S3 : Raw data from the screen described in Section 5.2 of the document.