Reliability for Binary and Ordinal Data in Forensics
Date
2022-08-08
Authors
Arora, Hina
Kaplan-Damary, Naomi
Stern, Hal
Major Professor
Advisor
Committee Member
Journal Title
Journal ISSN
Volume Title
Publisher
Copyright 2022, The Authors
Abstract
Black-box studies are a crucial part of assessing the accuracy and reliability of subjective decisions in forensics. The extant black-box studies have generally had two components. In the first study, examiners are asked to assess forensic samples (typically questioned and known samples with ground truth known by the study designers) just like they would in real casework and report decisions on a scale that is pre-defined. That addresses reproducibility (consistency) and validity (accuracy) of examiner decisions. In the second study, examiners are asked to re-assess a small subset of the samples that they observed in the first study. This allows researchers to assess repeatability or within-examiner consistency.
Motivated by these studies, this project explores statistical methods for the analysis of black box studies. In particular, we have used these examples to demonstrate a statistical approach to inferences for reliability that combines the reproducibility and repeatability studies while allowing for inference regarding examiner-sample interactions.
Motivated by these studies, this project explores statistical methods for the analysis of black box studies. In particular, we have used these examples to demonstrate a statistical approach to inferences for reliability that combines the reproducibility and repeatability studies while allowing for inference regarding examiner-sample interactions.
Series Number
Journal Issue
Is Version Of
Versions
Series
Academic or Administrative Unit
Type
Presentation
Comments
This presentation is from the 2022 Joint Statistical Meetings (JSM), Washington, D.C., August 6-11, 2022. Posted with permission of CSAFE.