Combining reproducibility and repeatability studies with applications in forensic science
Date
2023-10-19
Authors
Arora, Hina
Kaplan-Damary, Naomi
Stern, Hal S.
Major Professor
Advisor
Committee Member
Journal Title
Journal ISSN
Volume Title
Publisher
Oxford University Press
Abstract
Studying the repeatability and reproducibility of decisions made during forensic examinations is important in order to better understand variation in decisions and establish confidence in procedures. For disciplines that rely on comparisons made by trained examiners such as for latent prints, handwriting, and cartridge cases, it has been recommended that ‘black-box’ studies be used to estimate the reliability and validity of decisions. In a typical black-box study, examiners are asked to judge samples of evidence as they would in practice, and their decisions are recorded; the ground truth about samples is known by the study designers. The design for such studies includes repeated assessments on forensic samples by different examiners and additionally, it is common for a subset of examiners to provide repeated assessments on the same evidence samples. We demonstrate a statistical approach to analyse the data collected across these repeated trials that offers the following advantages: i) we can make joint inference about repeatability and reproducibility while utilizing both the intra-examiner and inter-examiner data, ii) we can account for examiner–sample interactions that may impact the decision-making process. We demonstrate the approach first for continuous outcomes such as where decisions are made on an ordinal scale with many categories. The approach is next applied to binary decisions and results are presented on the data from two black-box studies.
Series Number
Journal Issue
Is Version Of
Versions
Series
Academic or Administrative Unit
Type
Article
Comments
This article is published as Hina Arora, Naomi Kaplan-Damary, Hal S Stern, Combining reproducibility and repeatability studies with applications in forensic science, Law, Probability and Risk, Volume 22, Issue 1, 2023, mgad007, https://doi.org/10.1093/lpr/mgad007. Copyright The Authors (2023). Posted with permission of CSAFE. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.