Presentation
A cessation of measurement: Identifying test taker disengagement using response time
September 2020
Description
In this session from the National Council on Measurement in Education 2020 virtual conference (from 1:06:40 in video), Dr. Wise shares research insights into test disengagement and how disengagement should be managed in scoring.
There is a long tradition of using achievement tests to assess an individual’s knowledge, skills, and abilities. During such tests, a series of items is administered to a test taker, and the responses to each of these items provide information that is aggregated to produce a test score. The validity of inferences made based on the resulting score depends, in part, on the assumption that each item response reflects the construct or domain of interest. The reality, however, is that test takers do sometimes disengage from their test and give responses that do not reflect their knowledge, skills, and abilities. When computer-based tests are used, such item responses can be identified as rapid-guessing behavior.
The presence of rapid guesses in test data invites the question of what to do about them. In this presentation, we contend that because rapid guesses do not reflect what a test taker knows and can do they do not contribute to measurement and therefore should be excluded from scoring. This assertion is based on belief that rapid guessing on an item represents a test taker’s choice to “opt out” of the measurement process. We provide evidence to support this assertion through both data analyses and previous research findings. Collectively, this evidence shows that rapid guesses reflect a distinctly different, construct-irrelevant responses process. Moreover, if rapid guesses are included in scoring, they introduce systematic measurement errors that can seriously distort scores.
See MoreAssociated Research
Blog article
Related Topics
High dosage tutoring for academically at-risk students
This brief provides a review of the research on high dosage tutoring as an intervention strategy for supporting at-risk students. It highlights the benefits and the non-negotiable factors for effective implementation and usage.
Topics: COVID-19 & schools, Equity, Growth, Informing instruction
Typical learning for whom? Guidelines for selecting benchmarks to calculate months of learning
To describe the impact of the COVID-19 pandemic on students, researchers have translated test scores into months of learning to claim how many months/years students are behind in school. Despite its perceived accessibility, there are major downsides to this translation. To inform future uses by researchers and media, we discuss in this brief how to calculate this metric as well as its trade-offs.
Topics: COVID-19 & schools, Equity, Growth, Growth modeling, Seasonal learning patterns & summer loss
Exploring the educational impacts of COVID-19
This visualization was developed to provide state-level insights into how students performed on MAP Growth in the 2020–2021 school year. Assessments are one indicator, among many, of the student impact from COVID-19. Our goal with this tool is to create visible data that informs academic recovery efforts that will be necessary in the 2022 school year and beyond.
Topics: COVID-19 & schools, Innovations in reporting & assessment
Education’s long COVID: 2022–23 achievement data reveal stalled progress toward pandemic recovery
New research shows progress toward academic recovery stalled in 2022-23. This research brief covers data from 6.7 million US students examining academic gains relative to pre-pandemic years as well as tracking the gap in achievement between COVID year student groups compared to their pre-pandemic peers.
Products: MAP Growth
Topics: COVID-19 & schools, Equity
Technical appendix: 2022-23 achievement data reveal stalled progress toward pandemic recovery
The purpose of this technical appendix is to share more detailed results and describe the sample and methods used in the research in Education’s long COVID: 2022-23 achievement data reveal stalled progress toward pandemic recovery report.
Topics: COVID-19 & schools, Equity
Executive Summary: Content proximity spring 2022 pilot study
This executive summary outlines results from the Content Proximity spring 2022 pilot study, including information on the validity, reliability, and test score comparability of MAP Growth assessments that leverage this new item-selection algorithm.
Products: MAP Growth
Topics: Computer adaptive testing, Innovations in reporting & assessment, Test design
Content Proximity Spring 2022 Pilot Study Research Report
The purpose of this research report is to provide detailed information about updates to the MAP Growth item-selection algorithm. This brief includes results from the Content Proximity pilot study, including information on the validity, reliability, and test score comparability of MAP Growth assessments that leverage this new item-selection algorithm.
Products: MAP Growth
Topics: Computer adaptive testing, Innovations in reporting & assessment, Test design
