When we assess a student’s achievement using MAP® Growth™, we assume they remain fully engaged throughout the test. This assumption allows us to view their RIT score as an accurate indicator of what they know and can do, which can help their teacher plan instruction.
As teachers are aware, however, sometimes a student becomes disengaged during testing, which usually results in a score that underestimates a student’s achievement level and can mislead the teacher’s interpretation of the instructional needs. Thus, the presence of disengagement threatens the validity of interpretations teachers make about their students’ RIT scores.
What causes disengagement?
Disengagement can be due to a variety of factors. Some students become bored. Others aren’t feeling well, experience fatigue, or become highly anxious during testing. Disengagement can therefore be viewed as the presence of any student-related factors that prevent a student from being fully engaged when they take MAP Growth.
At NWEA, two indicators have been used to detect disengagement. The first is rapid-guessing behavior, in which a student answers a question much faster than the time required to read it, understand and solve its challenge, and submit an answer. Rapid guessing indicates that the student answered without really engaging with the question. The second indicator is performance decline, in which the student’s test performance shows a large drop during testing. Performance decline implies that the student did not maintain full engagement throughout their test event.
What the research tells us about avoiding disengagement
One way of managing the problem posed by test-taking disengagement is to identify and avoid testing conditions under which disengagement is especially prevalent. My colleagues and I recently completed a study, which will be published soon, in which we investigated the relationship between test-taking disengagement and the time of day that testing occurred. The results of this study show that time of day clearly matters.
More than half a million MAP Growth test events in both math and reading were grouped by the hour of the school day that testing began. Two indicators of disengagement were studied. A student’s MAP Growth test event was deemed disengaged if at least 10% of their responses were classified as rapid guesses or their performance during the second half of the test showed a statistically significant decline from first-half performance.
Our results showed that schools tend to test early, with most test events (over two-thirds) beginning during the first two hours of the day. However, there were many instances of test events occurring throughout the school day.
The line graph below shows the relationships for math testing between hour of testing and the prevalence of disengagement. The rate of substantial rapid guessing tripled—from 2.7% to 8.5%—from 7 a.m. and 1 p.m., and the rate of increase rose after noon.
Similarly, in reading, the prevalence of rapid guessing doubled—from 8.5% to 15.5%—with the rate of increase accelerating during the last two hours. In contrast, the rates of performance decline were relatively unchanged during the day and remained about 3% in math and 4% in reading.
There was no evidence of an hour-of-testing effect for performance decline. This was somewhat surprising, as performance decline suggests that a student did not show persistent high engagement throughout their test event. We expected that such persistence would be more difficult for students to maintain later in the day. Our research has shown, however, that performance decliners tend not to be those who rapid guess. Apparently, testing later in the day encourages more students to rush through MAP Growth without engaging with the questions, but without their performance changing during the test event.
Test in the morning
What does this research mean for MAP Growth schools? One conclusion is clear: the later in the day you administer MAP Growth, the more likely that substantial rapid guessing will occur.
While we understand that sometimes circumstances require tests to be administered later in the day, be aware of the potential costs of less trustworthy RIT scores. Test earlier in the school day, when you can.
