Population-level effects on crime of recovering firearms from armed prohibited persons: intention-to-treat analysis of a pragmatic cluster-randomised trial in California cities Authors: Garen J Wintemute, Daniel Tancredi, Veronica A Pear, Yueju Li, Christopher D McCort, Glenn Pierce, Anthony A Braga, Mona A Wright, Hannah Laqueur, Nicole Kravitz-Wirtz, David Studdert, Laurel Beckett Published: 2026 ABSTRACT AbstractBackgroundToo little is known about the effectiveness of efforts to prevent firearm violence. We evaluated California’s Armed and Prohibited Persons System (APPS), which identifies legal purchasers of firearms who have become prohibited persons and seeks to recover all firearms and ammunition to which they have access.Design and methodsThis cluster-randomised pragmatic trial was made possible by APPS’s expansion from a small pilot to a continuing statewide programme. We included 363 California cities, allocated to early (n=187) or later (n=176) intervention in blocks stratified by region within the state, and within region by population and violent crime rate. The study period began 1 February 2015; region-specific end dates ranged from 1 May 2015 to 1 February 2016. Analysis was on an intention-to-treat, difference-in-difference basis using generalised linear mixed models and generalised estimating equations with robust SEs. The population-level primary outcome measures were monthly city-level counts of firearm-related homicides, robberies and aggravated assaults. The primary model was adjusted for stratification variables; city-level population, population density, socioeconomic status and firearm purchasing; year; and month.FindingsAllocation groups were well balanced on baseline characteristics and implementation measures. In adjusted models, allocation to early intervention was not associated with statistically significant differences in any primary outcome measure; these findings were robust to multiple sensitivity analyses. There was some heterogeneity across regions.ConclusionsThe APPS intervention directly affects a very small percentage of the population, limiting its potential for population-level effects. Individual-level analyses may provide a better estimate of the intervention’s effectiveness.Trial registration numberNCT02318732. WHAT IS ALREADY KNOWN ON THIS TOPIC WHAT THIS STUDY ADDS In an intention-to-treat analysis, a programme to recover firearms from prohibited persons was not associated with beneficial effects at the population level. In an intention-to-treat analysis, a programme to recover firearms from prohibited persons was not associated with beneficial effects at the population level. HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY The intervention affects a small proportion of the population and, despite these findings, may have benefits for affected individuals. The intervention affects a small proportion of the population and, despite these findings, may have benefits for affected individuals. Introduction Firearm-related interpersonal violence is a major health and public safety problem in the USA, and rates are highest in urban populations. In mid-2022, an estimated 21% of American adults had experienced gun violence in the previous 5 years, either themselves or through a family member or close friend.1 In 2021, firearms were used in an estimated 20 966 homicides (81% of all homicides) and an estimated 326 894 non-fatal violent victimisations.2 3 One strategy for preventing firearm violence is to prohibit access to firearms by persons believed to be at high risk for committing violent acts. Background checks and denials of firearm purchases by prohibited persons, for example, can reduce the incidence of arrest for violent and firearm-related crime by at least 25% among individuals who are directly affected4 5 though they may not, by themselves, have similar benefits at the population level.6 7 Firearm access involves both acquisition and possession, however, and purchase restrictions have largely not been complemented by efforts to prevent retention of firearms by persons who acquire them legally and subsequently become prohibited persons. Transitions to prohibited-person status among armed persons are particularly concerning because they frequently arise from events associated with increased risk for future interpersonal violence or self-harm. Examples include convictions for violent crimes,810 involuntary hospitalisations for acute mental illness associated with dangerousness to self or others11 and domestic violence restraining orders (DVROs).12 Available evidence suggests that many legal firearm purchasers in the USA retain their firearms after becoming prohibited persons; California data from 2015 suggest that at least 100 000 have likely done so nationwide following prohibitions due to felony convictions,13 and no doubt others have remained armed following DVROs and other prohibiting events (see online supplemental file 1). California has implemented a systematic, large-scale effort to address this problem. The Armed and Prohibited Persons System (APPS), a programme of the California Department of Justice (CA DOJ), uses that agency’s archive of firearm sales and other transfers to identify firearm owners who have recently become prohibited persons under California or federal law. Specially trained APPS field teams then attempt to contact those individuals and recover all firearms and ammunition to which they have access. APPS was first implemented in November 2006 as a pilot programme that conducted fewer than 2000 investigations each year.14 15 Following a series of mass shootings in 2012, the California Legislature required APPS to be implemented statewide beginning in 2013 and greatly increased its funding. The enabling legislation was adopted as an urgency statute, a measure ‘necessary for immediate preservation of the public peace, health, or safety’16; the bill took effect immediately on its enactment.17 Working with CA DOJ, we designed and implemented a pragmatic18 cluster randomised trial that allocated more than 1000 cities and census-designated places (CDPs) to early or deferred implementation in a staged roll-out of the programme, the protocol for which was published in Injury Prevention.19 This report presents findings for a population-level intention-to-treat analysis. There are precedents for such focused interventions, which directly affect only a small proportion of a population, to have beneficial population-level effects on interpersonal violence. Boston’s CeaseFire was associated with significant reductions in youth homicide and firearm assaults.20 Operation Peacemaker was associated with decreases in firearm violence in Oakland.21 Our aim here is to determine whether allocation to early APPS intervention is associated with changes in monthly counts of firearm and non-firearm violent crimes reported to law enforcement agencies. Our subjects are 363 California cities (crime counts are not available for CDPs). Methods Overview The trial used a wait-list control design to evaluate an intervention that had been fully developed and was in operation before the trial began. Its design took advantage of the fact that, as APPS expanded, implementing the intervention across all of California simultaneously was impossible. The trial did not alter the content of the intervention and affected only the order in which cities received it. A detailed description of the intervention is in the online supplemental file 1. The 363 included cities contained an estimated 11 000 APPS-eligible individuals (prohibited persons who retained their firearms) at the time of randomisation. Cities were stratified by geographical region within the state, and within region by population and violent crime rate, and were then randomised 1:1 within strata to early (group 1) or deferred (group 2) intervention. The trial start date was 1 February 2015; region-specific intervention end dates were defined as the month prior to the onset of the intervention in group 2 cities in each region. The unit of analysis is the city month. The outcome measures (see the Outcomes section) are monthly city-level counts of reported crimes and the proportion of all violent crimes that involved firearms. The primary predictor is a time-dependent variable for the extent to which a city has experienced APPS implementation. Data sources Our primary data source was the APPS database, which contains identifiers for armed and prohibited persons, pointers to other CA DOJ records that provide grounds for the determinations that an individual owns a firearm and is a prohibited person, and information on the nature and duration of prohibitions. The linked APPS encounter database includes detailed reports of all field team encounters and attempted encounters with APPS-eligible individuals. APPS encounter records comprise detailed circumstantial data for all contacts and attempted contacts, a narrative describing encounters and their outcomes, and information on recovered firearms. California’s Dealer’s Record of Sale (DROS) database contains electronic records for handgun purchases from 1996 onward and long gun (rifle and shotgun) purchases from 2014 onward. DROS includes private party transfers, which must be processed by licensed retailers in California. We used DROS records to generate a proxy measure for city-level firearm purchasing (see online supplemental file 1). We obtained monthly crime counts from the California Criminal Justice Statistics Center’s archive of Return A and Supplementary Homicide Reports filings for the Uniform Crime Reporting (UCR) programme. We used a categorisation scheme developed by the California Department of Social Services to divide the state into regions: areas of relative geographical and labour market continuity defined to optimise socioeconomic homogeneity (online supplemental figure 1).22 The 2010 Census and the 2009–2011 American Community Survey provided community demographic and socioeconomic characteristics. California’s Behavioral Risk Factor Surveillance System (BRFSS) surveys included measurements of household firearm ownership in 2001–2004, 2009, 2017 and 2018. We used these data to generate a supplemental proxy measure of household firearm ownership at the county level (see online supplemental file 1). Participants APPS is a mandatory programme under state law; we did not seek consent of cities for participation. In consultation with CA DOJ, we excluded three groups of cities prior to randomisation (online supplemental figure 1). Two sets of exclusions were at the county level. CA DOJ indicated that it would not be fiscally appropriate for APPS to intervene early in the study period in 24 isolated, sparsely populated counties located primarily in the northernmost and eastern parts of the state. (21 of 22 counties in the CDSS North and Mountains region were excluded for this reason; Butte County was assigned to the Central Valley region, which it borders, for randomisation.) We also excluded five counties where DOJ had recently completed extensive enforcement actions. Finally, we excluded the city of Los Angeles, which had implemented its own intervention similar to APPS. The effect of these exclusions on the study population is shown in online supplemental figure 1. Measurement During the trial, APPS intervention in a city typically progressed to completion over a period extending from several days to a few weeks. We created a continuous progress-to-completion (PTC) variable ranging from 0 to 1.0, set to 0 on the study start date (no individuals intervened with) for all group 1 cities and to 1.0 for each city when all interventions delivered in that city prior to its region-specific end date had been delivered. PTC for each city was updated at the end of each month. Progress was often slow at the beginning and end of the period in which a city’s intervention occurred, as the first and last few individuals were contacted. We therefore considered cities to be preimplementation until the end of the month in which PTC reached 0.1 and post implementation at the end of the month in which PTC reached 0.9. Protocol deviation A protocol deviation occurred for a group 1 city if interventions did not begin in that city until after they had begun for at least one group 2 city in that same region. Similarly, a protocol deviation occurred for a group 2 city if interventions began in that city before interventions had begun in all group 1 cities in the same region. (See online supplemental figure 2 for additional information.) Monitoring Investigators monitored fidelity of implementation through regular meetings with CA DOJ senior staff (biweekly during the intervention period), real-time APPS records monitoring and field observations (ie, ride-alongs). Internet searches and the CA DOJ meetings were used to monitor for contamination (unplanned exposure of a city’s population to the existence of APPS generally or its presence in that city through media publicity), parallel interventions and requests from communities for intervention outside the allocation scheme. Outcomes The outcome measures are monthly city-level reported crime counts prepared for the FBI’s UCR programme.23 The primary outcome measures are firearm-related homicides, robberies and aggravated assaults. Secondary outcome measures include those same three crimes where firearms are not involved. They also include violent crimes in aggregate (the categories just mentioned and simple assault) and non-violent crimes (burglary, larceny-theft and motor vehicle theft). Rape was excluded, as a nationwide redefinition for UCR reporting in 2013 produced a major discontinuity in reported counts. Under UCR coding criteria, firearm-related aggravated assaults are not restricted to crimes in which a firearm is discharged. Crimes charged by local jurisdictions as assault and battery, disorderly conduct and brandishing a firearm, among others, are reported to UCR as aggravated assault.23 Outcome measures also included the proportion of violent crimes committed with firearms, a measure intended to capture possible weapon substitution. This was determined using a binomial generalised linear model with mixed effects in which the outcome events were defined by the UCR crime categories of ‘robbery with a gun’ and ‘assault with a gun’ and the Supplemental Homicide Reports category ‘murder with a gun’. Total events used the UCR categories ‘robbery total’, the sum of (‘assault with hands, feet, etc.’ + ‘assault with a knife’ + ‘assault other weapon’ + ‘assault with a gun’) and ‘murder’, respectively. Randomisation Randomisation was performed at the city level in a 1:1 ratio to early (group 1) or deferred (group 2) intervention. Cities were stratified for randomisation by CDSS region and then by whether they were above or below the regional medians for 2010 population and UCR Part I violent crime rate (average for 2009–2011). Randomisation was carried out using PROC SURVEYSELECT in SAS V.9.4 on 14 January 2015. Allocation concealment Treatment assignment was not masked to CA DOJ office staff and field teams or to the investigators; cities were not made aware of the study or of their allocation to treatment groups. Statistical methods Analyses were conducted in duplicate. We employed a superiority testing framework for all hypotheses and a difference-in-difference approach. Our primary null hypothesis was that any change in monthly crime counts in group 1 cities during the time when those cities received intervention and group 2 cities did not would not differ from changes in group 2 cities during that time period, when counts for both groups were compared with those for the preintervention period—1 January 2010 to 1 February 2015. This hypothesis was tested against the two-sided alternative that there would be such a difference. We fit unadjusted and adjusted models; unadjusted models included the prerandomisation stratification characteristics (CDSS region, log-transformed population and violent crime rate above or below regional median). All models specified the log of the predicted count as a linear function of the primary predictor (intervention status) and covariates, with random effects for city. Analyses were conducted using generalised linear mixed models. In cases of non-convergence, models using generalised estimating equations were employed. Robust SEs were used to generate CIs (see online supplemental file 1). Analysis population Eight small cities (with a combined 2010 population of 21 770) that had been included in the randomisation were excluded from the primary analysis. No residents of these cities were eligible for APPS on the study start date or during the study period. A Consolidated Standards of Reporting Trials (CONSORT) diagram (figure 1) summarises the flow of study cities from assessment for eligibility to inclusion in the analysis. Consolidated Standards of Reporting Trials flow diagram. *Here and following, the numbers in parentheses represent the number of cities included, the mean population of those cities, and the SD around that mean population. †Three cities (American Canyon, Laguna Woods, Buellton) do not have outcome data. Adjustment for covariates Anticipating residual heterogeneity following randomisation, we prespecified and investigated a wide range of covariates. Variables retained in the primary model included the prerandomisation stratification characteristics, log-transformed population in 2014, population density in 2010, year, month, DROS-derived estimated annual prevalence of firearm purchasing at the city level (a small-area proxy for firearm ownership) and a California-specific analogue to the Area Deprivation Index (ADI).24 The last two were included because the prevalence of firearm ownership and community characteristics such as those in the ADI have firmly established associations with rates of firearm violence.25 For the ADI analogue, we developed coefficients for individual ADI variables using principal components factor analysis on data for cities in California and categorised the ADI analogue in deciles. Sensitivity analyses Sensitivity analyses examined interactions between the prerandomisation stratification variables, the BRFSS-derived county-level estimate of household firearm ownership, exclusion of cities where no interventions occurred (though eligible individuals resided in those cities) or where intervention constituted a protocol deviation, alternative specifications of intervention start and end dates for individual cities, and exclusion of two outlier cities (see online supplemental file 1). Secondary analyses In secondary analyses, the null hypothesis was tested for primary and secondary outcomes at the regional level. Models with region × group interaction terms tested for heterogeneity among regions using p=0.20 as the threshold for significance, noting that this would allow us to gain a greater increase in power than the increase in type-1 error for the amount of heterogeneity we considered important to be able to detect.26 27 Statistical software Analyses were conducted using SAS V.9.4 (PROC GLIMMIX for generalised linear mixed models and PROC GENMOD for generalised estimating equations) and replicated in R V.4.2.0. Guidelines This study is reported in accordance with CONSORT guidelines for randomised trials.2830 Results Participant flow A total of 363 cities were randomised for inclusion: 187 to group 1 (early intervention) and 176 to group 2 (deferred intervention) (figure 1). Among group 1 cities, 94.7% (177) received intervention according to protocol; 69.3% (122) did so among group 2 cities, and the intervention was not discontinued prematurely in any city. Periods of observation were defined as the time in each region when group 1 cities were receiving intervention and group 2 cities were not. Periods of observation began 1 February 2015 in all regions but varied in duration from 2.9 months (Southern Farm region) to 15.9 months (Los Angeles County region) (online supplemental figure 2). Baseline data The two groups of cities were well balanced on a wide array of baseline characteristics (table 1), including population size and demographic composition, distribution across ADI deciles, location, violent and non-violent crime rates, prevalence of firearm ownership and purchasing, and proportion of the population eligible for the APPS intervention on the study start date. The proportions eligible were quite low in both groups, with a mean of just over 70 per 100 000 population. Table 1Baseline characteristics of study citiesCity characteristicGroup 1 cities(n=187)Group 2 cities(n=176)Demographics of APPS-eligible persons as of the study start date Age, years, mean (SD)47.0 (11.8)46.4 (11.6) Age 15–29, total n (%) of persons for city group228 (4.3)258 (4.6) Male, total n (%) of persons for city group4984 (93.4)5194 (92.7)Race/ethnicity, total n (%) of persons for city group White, non-Hispanic2421 (45.3)2508 (44.7) Black, non-Hispanic594 (11.1)694 (12.4) Latinx/Hispanic1505 (28.2)1485 (26.5) Asian American, non-Hispanic392 (7.3)425 (7.6) Other, non-Hispanic429 (8.0)501 (8.9)Other characteristics 2014 population, median (IQR)37 237.5 (52690)35 422 (74200) 2014 population, mean (SD)65 033.4 (124838.5)61 173.8 (67728.7) 2010 population density (persons per square mile), mean (SD)4864 (3818.1)4470 (3341.8)ADI, n (%) at decile 119 (10.2)13 (7.4) 214 (7.5)19 (10.8) 315 (8.0)17 (9.7) 418 (9.6)13 (7.4) 513 (7.0)23 (13.1) 618 (9.6)25 (14.2) 717 (9.1)21 (11.9) 829 (15.5)16 (9.1) 922 (11.8)18 (10.2) 1022 (11.8)11 (6.3)CDSS region, n (%) of cities Bay Area12 (6.4)11 (6.3) Central Valley*40 (21.4)37 (21.0) Southern Farm29 (15.5)28 (15.9) Los Angeles County44 (23.5)42 (23.9) Southern California62 (33.2)58 (33.0) DROS-based prevalence of firearm purchasing†: 2010–2014 yearly average transactions per 100 000 person years, mean (SD)1327 (890)1290 (593) BRFSS-based prevalence of household firearm ownership,† households per 100 000 persons, mean (SD)21 544 (6514)22 031 (6375) 2014 violent crime‡ rate (per 100 000, median (IQR)276.1 (282.4)260.1 (289.0) 2014 violent crime‡ rate (per 100 000), mean (SD)327.8 (245.2)327.9 (248.0) 2014 non-violent crime§ rate (per 100 000), median (IQR)2185.1 (1360.6)2110.7 (1170.5) 2014 non-violent crime§ rate (per 100 000), mean (SD)2659.2 (3742.8)2378.4 (1544.3) APPS-eligible persons per 100 000 adult population as of the study start date, mean (SD)72.8 (96.2)73.5 (34.2)*Butte County was added to the Central Valley region for purposes of analysis; all other counties in its CDSS region were excluded from the study.†See methods for details of calculation.‡Includes murder, rape, robbery and aggravated assault.§Includes burglary, larceny-theft and motor vehicle theft.CDSS California Department of Social Services; APPS Armed and Prohibited Persons SystemADI, Area Deprivation Index; BRFSS, Behavioral Risk Factor Surveillance System; DROS, Dealer’s Record of Sale. Baseline characteristics of study cities Butte County was added to the Central Valley region for purposes of analysis; all other counties in its CDSS region were excluded from the study. See methods for details of calculation. Includes murder, rape, robbery and aggravated assault. Includes burglary, larceny-theft and motor vehicle theft. CDSS California Department of Social Services; APPS Armed and Prohibited Persons SystemADI, Area Deprivation Index; BRFSS, Behavioral Risk Factor Surveillance System; DROS, Dealer’s Record of Sale. Numbers analysed Eleven cities were excluded from the analysis; 180 group 1 cities and 172 group 2 cities were included. Summary measures describing the delivery of the intervention in group 1 cities are in table 2. About 25% of persons eligible for intervention (0.16 persons per 1000 adults in participating cities) had contact with intervention teams. Table 2Measures of intervention as delivered in group 1 citiesCharacteristicGroup 1 cities, n=187Cities assigned to each Bureau of Firearms regional office on study start date, n (%) Fresno25 (13) Los Angeles75 (40) Riverside24 (13) Sacramento12 (6) San Diego11 (6) San Francisco40 (21)Cities intervened with by each Bureau of Firearms regional office, n (%) Fresno38 (20) Los Angeles44 (24) Riverside44 (24) Sacramento14 (7) San Diego11 (6) San Francisco36 (19) Time to completion (from 10% to 90%), months, mean (SD)3.2 (2.9) Time to completion (from 10% to 90%), months, median (IQR)3 (4) Number of persons experiencing the APPS intervention, mean (SD)6.7 (11) Number of persons experiencing the APPS intervention, median (IQR)2.6 (7) Number of persons experiencing the APPS intervention per 1000 adult population, mean (SD)0.16 (0.36) Number of persons experiencing the APPS intervention per 1000 adult population, median (IQR)9 (17) Proportion of APPS-eligible persons (as of the study start date) experiencing the APPS intervention, mean (SD)0.24 (.25) Proportion of APPS-eligible persons (as of the study start date) experiencing the APPS intervention, median (IQR)0.17 (0.28) Number of firearms recovered per 100 000 adult population, mean (SD)6.58 (17.4) Number of firearms recovered per 100 000 adult population, median (IQR)0 (5) Number of persons arrested during APPS interventions per 100 000 adult population, mean (SD)0.58 (1.42) Number of persons arrested during APPS interventions per 100 000 adult population, median (IQR)0 (0)Reasons for prohibition among persons intervened with (n (%) of persons*) Felony conviction263 (52) Brady (federal) prohibition†121 (24) Restraining order107 (21) Misdemeanour81 (16) Mental health75 (15) Condition of probation66 (13) Others2 (00) Juvenile prohibition1 (00)*Percentages do not add to 100 because the categories are not mutually exclusive; an individual may be prohibited for several reasons.†Prohibitions that are listed in federal statutes and therefore apply in California but are not also listed in California statutes.APPS, Armed and Prohibited Persons System. Measures of intervention as delivered in group 1 cities Percentages do not add to 100 because the categories are not mutually exclusive; an individual may be prohibited for several reasons. Prohibitions that are listed in federal statutes and therefore apply in California but are not also listed in California statutes. APPS, Armed and Prohibited Persons System. Outcomes and estimation In unadjusted models (online supplemental table 2), allocation to early intervention was not associated with differences in the incidence of most primary and secondary outcomes. Group 1 cities experienced relative increases in firearm-related aggravated assault (IRR 1.27, 95% CI 1.14, 1.42) and in the proportion of aggravated assaults involving firearms (OR 1.34, 95% CI 1.03, 1.75). Unadjusted region-specific models (online supplemental table 3) found no association between allocation to early intervention and most outcome measures; there were increases in firearm-related aggravated assault in the Los Angeles County and Southern California regions, in simple assault in the Southern California region, in all violent crimes in the Bay Area and Southern California regions, and in non-violent crimes in the Los Angeles County region. In adjusted models, however, no significant differences between group 1 and group 2 cities were observed for any outcome measure (table 3, figure 2, online supplemental figure 2); the estimate for firearm-related aggravated assault (IRR 1.29, 95% CI 0.99, 1.68) was similar to that from the unadjusted model. In region-specific models (online supplemental file 4, figure 2, and online supplemental figure 3), the only significant difference detected was a decrease in firearm robbery in the Central Valley region. Unadjusted and adjusted estimates for firearm-related aggravated assault (online supplemental tables 3,4, respectively) differed substantially for the Bay Area, Central Valley and Southern California regions. Models including a region × group interaction term suggested heterogeneity among regions only for firearm-related aggravated assault (p=0.15) and all non-violent crimes (p=0.11). Table 3Adjusted* incidence rate ratios for crimes by type and firearm involvement and proportion of violent crimes involving a firearm, comparing group 1 and group 2 citiesOutcomeIncidence rate ratio (95% CI)†Firearm-related violent crimes Homicide0.93 (0.65, 1.34) Robbery0.99 (0.86, 1.13) Aggravated assault1.29 (0.99, 1.68)Non-firearm violent crimes Homicide0.97 (0.60, 1.57) Robbery0.98 (0.83, 1.09) Aggravated assault0.98 (0.89, 1.07)‡ Simple assault0.95 (0.90, 1.01)‡ All violent crimes0.98 (0.95, 1.02) Non-violent crimes§1.02 (0.99, 1.06)Proportion of violent crimes involving a firearm OR (95% CI)† Homicide0.76 (0.29, 1.98) Robbery0.93 (0.70, 1.24) Aggravated assault1.16 (0.90, 1.50)*Models include CDSS region, violent crime rate (above/below regional median), non-violent crime rate (above/below regional median), population density, DROS-based prevalence of firearm purchasing, ADI decile, year, month and population (log-transformed). Cities were stratified on the first 3 variables for randomisation.†Group 2 cities are the referent group.‡Estimated using generalised estimating equations.§Includes burglary, larceny-theft and motor vehicle theft.ADI, Area Deprivation Index; CDSS California Department of Social Services; DROS, Dealer Record of Sale. Adjusted* incidence rate ratios for crimes by type and firearm involvement and proportion of violent crimes involving a firearm, comparing group 1 and group 2 cities Models include CDSS region, violent crime rate (above/below regional median), non-violent crime rate (above/below regional median), population density, DROS-based prevalence of firearm purchasing, ADI decile, year, month and population (log-transformed). Cities were stratified on the first 3 variables for randomisation. Group 2 cities are the referent group. Estimated using generalised estimating equations. Includes burglary, larceny-theft and motor vehicle theft. ADI, Area Deprivation Index; CDSS California Department of Social Services; DROS, Dealer Record of Sale. Combined and region-specific adjusted* incidence rate ratios for firearm-related violent crimes by type, comparing group 1 and group 2 cities. (a) Firearm homicide. (b) Firearm robbery. (c) Firearm aggravated assault. *Models include California Department of Social Services region (combined model only), violent crime rate (above/below regional median), non-violent crime rate (above/below regional median), population density, Dealer’s Record of Sale-based prevalence of firearm ownership, Area Deprivation Index decile, year, month and population (log-transformed). Cities were stratified according to the first three variables for randomisation. Incidence rate ratios were estimated using generalised estimating equations for firearm-related robbery in the Bay Area, Central Valley, Southern Farm and Southern California regions and for firearm-related aggravated assault in the Central Valley, Southern Farm and Los Angeles County regions. Sensitivity analyses Findings for sensitivity analyses were similar to those from the adjusted models; most yielded significant increases for firearm-related aggravated assault but not for other primary outcome measures. (Data not shown.) Additional analyses A region-specific and group-specific assessment of trends in firearm-related aggravated assault identified pre-existing trends that continued through the observation period and differed significantly between group 1 and group 2 cities in two of three regions (figure 2c, online supplemental table 4) where IRRs exceeded 1.0: the Los Angeles region (upward trends in group 1 and group 2 cities, but more steeply in group 1; group × time interaction p=0.02) and in the Bay Area region (downward trends in group 1 and group 2 cities, but more steeply in group 2; group × time interaction p=0.02) (online supplemental figure 4). No such interaction was seen in the Central Valley or Southern California regions, where IRRs were close to 1.0. Discussion APPS seeks to prevent firearm violence through systematic enforcement of laws regulating access to firearms. The subject population is precisely defined: persons who have legally acquired firearms in the past (and have acquisition records in the data used for the programme) and have subsequently become prohibited persons. This study reported population-level estimates from the intention-to-treat analysis of a cluster-randomised trial of APPS’s short-term effects on firearm-related violent crimes and crimes of other types. In adjusted models, allocation to early intervention was not associated with statistically significant differences for any outcome measure; a non-significant increase in firearm-related aggravated assault achieved statistical significance in unadjusted models. There was suggestion of regional variation, and some estimates for firearm-related aggravated assault varied substantially with adjustment. At least seven mechanisms, acting individually or together, could account for findings of no effect. One is inherent to the study design. Pragmatic intention-to-treat analyses such as this set the bar high; they assess effects of treatment allocation and not treatment as actually delivered. The other six are specific to APPS and to this study; items 2–4 in this list, taken together, are likely the most important. The first arose from the legislative mandate for APPS to be implemented statewide urgently; the study’s period of observation (the time between onset of intervention in group 1 cities and onset in group 2) was shorter than expected. This reduced power to detect differences between group 1 and group 2 cities. Second, only a small percentage of the population was eligible for the intervention. Even substantial reduction in risk in this small population might not produce a change in population measures of crime. Third and similarly, failing to intervene with all eligible individuals would increase the likelihood of a null finding, and our estimate is that contact was made with 25% of individuals who were APPS eligible at the start of the study. Note that of the remaining 75%, some fraction—and perhaps many of those prohibited as a result of DVROs—would have had their prohibitions expire before APPS intervened in their cities. There are several other possible reasons for the low contact proportion. It was important that no advance notice of contact be given, for example; such notice would have maximised opportunities for prohibited possessors to take action to prevent APPS personnel from recovering their firearms. The study design also played a role. Recognising the implementation difficulties, APPS only closes cases if all known firearms have been recovered or all possible leads have been definitively exhausted. Our study design terminated observation for group 1 cities in any region when group 2 interventions began, with the result that any late contacts were not included. These limitations notwithstanding, if APPS interventions have a durable effect on the individuals directly affected, population-level effects might well emerge over time as the intervened-with proportion of the population increases. Fourth, APPS does not intervene with some prohibited firearm owners at high risk of interpersonal violence: those who acquired their firearms illegally and whose firearm ownership is undocumented. Absent CA DOJ firearm acquisition records, these high-risk individuals cannot meet APPS’s eligibility criteria. They may constitute the majority of prohibited persons with firearms in California—nationwide, only 10.1% of persons incarcerated for firearm-related violence acquire their firearms from a retail source.31 Fifth and conversely, APPS does intervene with individuals who are likely not at high risk for the study outcomes. About 7% of persons intervened with during the study were prohibited only following hospitalisations for dangerousness arising from mental illness. Severe and acutely exacerbated mental illness is associated more strongly with risk for self-harm and suicide than with risk for commission of interpersonal violence.5 Finally, APPS relies on incomplete firearm ownership data and cannot identify firearm owners who legally purchased handguns only before 1996 or long guns only before 2014. If they subsequently become prohibited persons, they will not be detected by APPS’s screening processes. The increase in firearm-related aggravated assault associated with allocation to early intervention that we found in some models was unexpected and remains a subject for future exploration. As mentioned, several region-specific estimates varied substantially with adjustment for prespecified covariates. Separately, increases in firearm-related aggravated assault beginning in January 2014 (a year before the study began) in group 1 cities relative to group 2 cities occurred in the Los Angeles County and Bay Area regions, two of the three regions in which non-significant increases were associated with allocation to early intervention in adjusted models. (Striking decreases in firearm homicide and firearm-related aggravated assault in Oakland, a group 2 city in the Bay Area region, have been examined previously.32) Separately, it bears repeating that, under UCR coding criteria, firearm-related aggravated assault may include misdemeanours such as assault and battery and brandishing a firearm. Variation in coding and reporting procedures may also account in part for our findings. A modelling study has attributed strongly beneficial effect to APPS, estimating that the intervention prevented more than 100 firearm homicides per year from 2007 to 2016.33 Results from this real-world trial are quite different and, in our judgement, cast doubt on the validity of the estimates from that study. Separately, we note that APPS was a pilot programme during most of the intervention period included in the modelling study, conducting fewer than 2000 interventions per year.14 15 We do not consider it likely that such a small programme would have the effect estimated in the modelling study. To our knowledge, no other randomised trials of firearm recovery interventions exist to provide a basis for comparison with our findings. Ecological and observational studies of extreme risk protection order (ERPO) laws provide the closest parallel. ERPOs have been used hundreds of times in efforts to prevent mass shootings, for example, and where outcome data are available, none of those threatened shootings have occurred.34 35 But mass shootings account for a very small percentage of deaths from firearm violence and could be eliminated completely without producing a detectable decrease in the population-level firearm violence mortality rate. Individual-level studies suggest that ERPOs have benefits in preventing suicides.36 Population-level studies have yielded mixed effects,37 38 however, and a knowledgeable commentator has suggested that it is too soon for population-level evaluations of ERPOs.39 The same may have been true for this study of APPS, especially given the relatively brief period of observation. Experience with DVRO prohibitions, which have been available for decades, suggests that population-level effects emerge over time.12 In such situations, analyses relying on individual-level data may provide the best estimates of interventions’ effectiveness. Such an analysis is underway for APPS. Limitations Our study did not include all cities in the state. Cities in counties that were subject to the intervention shortly before the study began were excluded, as were cities in remote areas and the city of Los Angeles. As mentioned, the relatively brief period of observation prior to onset of APPS interventions in group 2 cities limited our ability to detect intervention effects. Delayed intervention in seven small cities on the relatively remote Central Coast resulted in an unexpected number of group 2 protocol deviations in the Farm and Southern California regions (online supplemental figure 2). Advance or concurrent notice of an intervention team’s presence in a city could have provided an opportunity for individuals to evade intervention, but our surveillance generated no reports of such contamination. Conclusion APPS is a unique and promising intervention that seeks to prevent violence by recovering firearms from prohibited persons. This intention-to-treat analysis of a large, pragmatic, cluster randomised trial provides a rigorous assessment of its effects on important population-level outcome measures. There was no clear evidence of benefit at that level. Several concurrent mechanisms may account for this, the most important, in our judgement, being that intervention was completed more quickly than planned for in the treatment (group 1) cities, that only a small part of the population was eligible for intervention, that many eligibles did not receive the intervention and that many persons at risk were not eligible. Further study of the effects of the intervention as delivered, effects on suicide and effects at the individual level is needed before recommendations can be made on replicating the programme in other states. Supplementary material