Hierarchical Bayesian non-response models for error rates in forensic black-box studies

Thumbnail Image
Khan, Kori
Major Professor
Committee Member
Journal Title
Journal ISSN
Volume Title
© 2023 The Authors
Carriquiry, Alicia
Distinguished Professor
Research Projects
Organizational Units
Journal Issue
Is Version Of
StatisticsCenter for Statistics and Applications in Forensic Evidence
Forensic science plays a critical role in the United States criminal legal system. Historically, however, most feature-based fields of forensic science, including firearms examination and latent print analysis, have not been shown to be scientifically valid. Recently, black-box studies have been proposed as a means of assessing whether these feature-based disciplines are valid, at least in terms of accuracy, reproducibility and repeatability. In these studies, forensic examiners frequently either do not respond to every test item or select an answer equivalent to ‘don’t know’. Current black-box studies do not account for these high levels of missingness in statistical analyses. Unfortunately, the authors of black-box studies typically do not share the data necessary to meaningfully adjust estimates for the high proportion of missing responses. Borrowing from work in the context of small area estimation, we propose the use of hierarchical Bayesian models that do not require auxiliary data to adjust for non-response. Using these models, we offer the first formal exploration of the impact that missingness is playing in error rate estimations reported in black-box studies. We show that error rates currently reported as low as 0.4% could actually be at least 8.4% in models accounting for non-response where inconclusive decisions are counted as correct, and over 28% when inconclusives are counted as missing responses. These proposed models are not the answer to the missingness problem in black-box studies. But with the release of auxiliary information, they can be the foundation for new methodologies to adjust for missingness in error rate estimations.
This article is part of the theme issue ‘Bayesian inference: challenges, perspectives, and prospects’.
This article is published as Khan K, Carriquiry A. 2023 Hierarchical Bayesian non-responsemodels for error rates in forensic black-box studies. Phil. Trans. R. Soc. A 381: 20220157. https://doi.org/10.1098/rsta.2022.0157. Posted with permission of CSAFE.

Published by the Royal Society under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/4.0/, which permits unrestricted use, provided the original author and source are credited.