Personality and mental health as predictors of rater bias in observational data

Abstract

<p>Observational coding has become a common form of data collection in the social sciences. This popularity has largely been fueled by the advantage of objectivity these systems are said to provide over more traditional self- and other-report data collection techniques. However, it has been demonstrated that observational coding systems can suffer from a type of systematic error known as rater bias, which is variance in the data that is not attributable to the observational target (i.e., true variance), but rather is attributable to the rater (i.e., error variance). This study sought to better understand the causes of rater bias. A group of 47 observational coders employed by a large research institute were administered the SCL-90-R and the NEO-PI-R. From these results, personality traits and mental health could be estimated for each coder. A large, longitudinal observational data set was analyzed to estimate the amount of rater bias exhibited by each coder, and this was correlated with the results from the SCL-90-R and NEO-PI-R. Two trends were evident from these results. First, coders who were introverted introduced more rater bias than those who were extroverted. Second, coders who experienced poor mental health introduced more rater bias than coders who were psychologically healthy. These findings were generally true with respect to scales measuring negative or neutral attributes of the targets. The effects were not found on scales measuring positive attributes of the target.</p>