Using retest data to evaluate and improve effort-moderated scoring
This session from the National Council on Measurement in Education 2020 virtual conference presents new research findings on understanding and managing test-taking disengagement (presentation begins at 22:55).
Wise, S., & Kuhfeld, M. (2020, September). Using retest data to evaluate and improve effort-moderated scoring. National Council on Measurement in Education virtual conference.
There has been increased research into managing test-taking disengagement. Some research focused on methods for adjusting scores for the effects of rapid guessing, which is a validated indicator of disengaged item responding. One example is the effort-moderated (E-M) model (Wise & DeMars, 2006), in which rapid guesses are identified and excluded during scoring. E-M scoring is intended to estimate the score a disengaged test taker would have received had they been fully engaged. Although it is challenging to evaluate the accuracy of E-M score adjustments, we developed a unique dataset that permits such an evaluation.
Study 1 analyzed a dataset based on archived RIT-score data from MAP Growth assessments in math and reading from students who had been retested within a short period of time. This allowed us to study the accuracy of E-M scoring. Across two testing terms, instances of students being retested within one day were identified. Within this group, over 5,000 cases were found in which students exhibited disengagement (10%+ rapid guesses) on the first testing and zero rapid guesses on the retest. Table 1 shows that E-M scores partially accounted for the observed RIT score differences.
Study 2 will investigate, using the same dataset, a purification method for improving E-M score accuracy. Specifically, the use of various secondary time thresholds for identifying responses excluded during E-M scoring will be investigated. The new method is expected to improve our ability to estimate the score distortion caused by test-taking disengagement, thus enhancing the value of E-M scoring.See More