Using retest data to evaluate and improve effort-moderated scoring

September 2020

By: Steven Wise, Megan Kuhfeld


This session from the National Council on Measurement in Education 2020 virtual conference presents new research findings on understanding and managing test-taking disengagement (presentation begins at 22:55).

Wise, S., & Kuhfeld, M. (2020, September). Using retest data to evaluate and improve effort-moderated scoring. National Council on Measurement in Education virtual conference.

There has been increased research into managing test-taking disengagement.  Some research focused on methods for adjusting scores for the effects of rapid guessing, which is a validated indicator of disengaged item responding.  One example is the effort-moderated (E-M) model (Wise & DeMars, 2006), in which rapid guesses are identified and excluded during scoring.  E-M scoring is intended to estimate the score a disengaged test taker would have received had they been fully engaged.  Although it is challenging to evaluate the accuracy of E-M score adjustments, we developed a unique dataset that permits such an evaluation.

Study 1 analyzed a dataset based on archived RIT-score data from MAP Growth assessments in math and reading from students who had been retested within a short period of time. This allowed us to study the accuracy of E-M scoring.  Across two testing terms, instances of students being retested within one day were identified.  Within this group, over 5,000 cases were found in which students exhibited disengagement (10%+ rapid guesses) on the first testing and zero rapid guesses on the retest.  Table 1 shows that E-M scores partially accounted for the observed RIT score differences.

Study 2 will investigate, using the same dataset, a purification method for improving E-M score accuracy.  Specifically, the use of various secondary time thresholds for identifying responses excluded during E-M scoring will be investigated.  The new method is expected to improve our ability to estimate the score distortion caused by test-taking disengagement, thus enhancing the value of E-M scoring.

See More
See the presentation

Related Topics