Reliability and Criterion Validity of Two Algebra Measures: Translations and Content Analysis-Multiple Choice

Thumbnail Image
Olson, Jeannette
Major Professor
Committee Member
Journal Title
Journal ISSN
Volume Title
Research Projects
Organizational Units
Organizational Unit
Project AAIMS
Project AAIMS, funded from January 2004 through December 2007, was designed to achieve two objectives related to the teaching and learning of algebra for students with and without disabilities. First, we examined algebra curriculum, instruction, and assessment for students with and without disabilities and determine the extent to which they were aligned. Second, we developed algebra assessment tools that can be used for monitoring the progress of students with and without disabilities as they learn algebra. We then investigated the measures' reliability, validity, and sensitivity to growth. This website has information about the activities, research, and products associated with Project AAIMS. Our district partners include Fort Dodge Community School District, South Tama County Community School District, and Ballard Community School District.
Journal Issue
Is Version Of

This technical report summarizes the results of a study in which we examined the technical adequacy of two potential measures for algebra progress monitoring. Eighty-seven students (11 of whom were receiving special education services) completed two forms of a Translations measure and two forms of a Content Analysis-Multiple Choice measure during each of two data collection sessions. In addition, we gathered data on criterion variables including grades, overall grade point average, teacher ratings of student proficiency, and scores on districtadministered standardized tests, as well as a measure of algebra aptitude. We examined both test-retest and alternate form reliability for both single probe scores and aggregated scores (computed by averaging two individual scores). Criterion validity was examined by computing correlations between students’ single and aggregated scores on the probes with their scores on other indicators of proficiency in algebra. The results of this study suggest that the Translations measure is more promising than the Content Analysis-Multiple Choice measure in terms of both reliability and criterion validity. The strength of the relations obtained in this study were in the low to moderate range and were not as strong as the relations obtained with a different sample in this district using three other algebra measures (see Project AAIMS Technical Report 2 for details of the earlier study). Both measures produced acceptable distributions that were free from floor and ceiling effects. Students had roughly similar means and standard deviations on both measures. Reliability estimates for both measures fell short of expected levels for both single probes and aggregated scores. The Translations measure produced stronger correlations than the Content Analysis-Multiple Choice measure, but did not demonstrate a level of reliability that would be acceptable for instructional decision making. The majority of the criterion validity relations were in the low to moderate range. Aggregated scores produced improvements in the criterion validity estimates for the Translations measure, but not for the Content Analysis-Multiple Choice measure. The strongest relations were identified between the Translations measure and eighth graders’ performance on the district’s math achievement test, as well as between the Translations measure and all students’ performance on the algebra aptitude test. These two relations were in the moderate to strong range; relations between the Translations measure and the remaining criterion variables were in the low range.


Project AAIMS is funded by the U.S. Department of Education, Office of Special Education Programs, Grant # H324C03006; Technical Report #6.