Policy Evaluation Beyond Significance Testing: Evidence from Chicago’s ShotSpotter Contract

Paper by Terence Chau et al: “Program impact estimates that are not statistically significant are often treated as providing little guidance for policy choice, even when economically important effects remain plausible. In such cases, the decision-relevant task is not simply to test whether the effect differs from zero, but to estimate the probability distribution of programmatic effects. The importance of going beyond hypothesis testing is especially clear when evaluations are underpowered relative to the effect needed for benefits to exceed costs. We illustrate this point using Chicago’s contract for ShotSpotter, an acoustical gunshot detection system that reduces police response times and may therefore have improved survival among shooting victims. Using a geographic RD design, we estimate that ShotSpotter reduced the shooting fatality rate by 2.5 percentage points, equivalent to roughly 49 fewer deaths per year. Although the estimate is not statistically significant, its implied benefit far exceeds the cost of the service. To have enough statistical power under the hypothesis testing approach to detect an effect of this size would require 87 years’ worth of Chicago shooting data. We instead implement a decision-relevant perspective using Bayesian analysis, treating the life-saving effect as a random variable. We find positive expected net benefits, with posterior probabilities that benefits exceed costs ranging from 70% under a flat prior and a $0.5 million value of statistical life to 99.6% under a no-harm prior and a $13.6 million value. The results illustrate how reliance on statistical significance can discard decision-relevant evidence when economically meaningful effects are difficult to detect precisely…(More)”.