Methodological problems in every black-box study of forensic firearm comparisons
Date
2024-12-05
Authors
Cuellar, Maria
Vanderplas, Susan
Luby, Amanda
Rosenblum, Michael
Major Professor
Advisor
Committee Member
Journal Title
Journal ISSN
Volume Title
Publisher
Oxford University Press
Abstract
Reviews conducted by the National Academy of Sciences (2009) and the President’s Council of Advisors on Science and Technology (2016) concluded that the field of forensic firearm comparisons has not been demonstrated to be scientifically valid. Scientific validity requires adequately designed studies of firearm examiner performance in terms of accuracy, repeatability, and reproducibility. Researchers have performed “black-box” studies with the goal of estimating these performance measures. As statisticians with expertise in experimental design, we conducted a literature search of such studies to date and then evaluated the design and statistical analysis methods used in each study. Our conclusion is that all studies in our literature search have methodological flaws that are so grave that they render the studies invalid, that is, incapable of establishing scientific validity of the field of firearms examination. Notably, error rates among firearms examiners, both collectively and individually, remain unknown. Therefore, statements about the common origin of bullets or cartridge cases that are based on examination of “individual” characteristics do not have a scientific basis. We provide some recommendations for the design and analysis of future studies.
Series Number
Journal Issue
Is Version Of
Versions
Series
Academic or Administrative Unit
Type
Article
Comments
This article is published as Maria Cuellar, Susan Vanderplas, Amanda Luby, Michael Rosenblum, Methodological problems in every black-box study of forensic firearm comparisons, Law, Probability and Risk, Volume 23, Issue 1, 2024, mgae015, https://doi.org/10.1093/lpr/mgae015. Posted with permission of CSAFE.
Rights Statement
© The Authors (2024). This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Funding
This work was supported by the Center for Statistics and Applications in Forensic Evidence (CSAFE) through [Cooperative Agreement 70NANB20H019] between the U.S. National Institute of Standards and Technology (NIST) and Iowa State University, which includes activities carried out at Carnegie Mellon University, Duke University, University of California Irvine, University of Virginia, West Virginia University, University of Pennsylvania, Swarthmore College and University of Nebraska, Lincoln; and a Nexus Award from Johns Hopkins University to M.R.