Determining an inter-rater agreement metric for researchers evaluating student pathways in problem solving

dc.contributor.advisor John Jackman
dc.contributor.author Sullivan, Austin
dc.contributor.department Industrial and Manufacturing Systems Engineering
dc.date 2018-08-11T15:14:24.000
dc.date.accessioned 2020-06-30T02:51:18Z
dc.date.available 2020-06-30T02:51:18Z
dc.date.copyright Wed Jan 01 00:00:00 UTC 2014
dc.date.embargo 2001-01-01
dc.date.issued 2014-01-01
dc.description.abstract <p>A new inter-rater agreement metric (IRRA) was developed for measuring agreement between multiple research coders when they code activities as they observe student problem-solving sessions. The complex nature of the student data includes activities that 1) are dependent on prior activities, 2) are ordinal data types, 3) can occur at any point in time, and 4) can reoccur. The assumptions used in traditional inter-rater agreement metrics are violated in this context and may lead to erroneous conclusions in particular datasets. In this study, coded activities are considered to be a sequence codes that can be analyzed using a string matching algorithm. We evaluated the metric's performance by simulating the variability of coders in a controlled fashion. The results show that the algorithm performed well as an inter-rater agreement metric over a wide range of conditions.</p>
dc.format.mimetype application/pdf
dc.identifier archive/lib.dr.iastate.edu/etd/13665/
dc.identifier.articleid 4672
dc.identifier.contextkey 5777354
dc.identifier.doi https://doi.org/10.31274/etd-180810-2820
dc.identifier.s3bucket isulib-bepress-aws-west
dc.identifier.submissionpath etd/13665
dc.identifier.uri https://dr.lib.iastate.edu/handle/20.500.12876/27852
dc.language.iso en
dc.source.bitstream archive/lib.dr.iastate.edu/etd/13665/Sullivan_iastate_0097M_14066.pdf|||Fri Jan 14 19:58:01 UTC 2022
dc.subject.disciplines Cognitive Psychology
dc.subject.disciplines Industrial Engineering
dc.title Determining an inter-rater agreement metric for researchers evaluating student pathways in problem solving
dc.type article
dc.type.genre thesis
dspace.entity.type Publication
relation.isOrgUnitOfPublication 51d8b1a0-5b93-4ee8-990a-a0e04d3501b1
thesis.degree.level thesis
thesis.degree.name Master of Science
File
Original bundle
Now showing 1 - 1 of 1
Name:
Sullivan_iastate_0097M_14066.pdf
Size:
831.21 KB
Format:
Adobe Portable Document Format
Description: