Multiple choice assessments are good enough for nuclear power, why not education?

As the concluding post to a four-part series on how assessments can be well-aligned to state or common core standards, I am introducing a new phrase into the education ecosystem – a reasonable level of assurance. In my 20 years in nuclear power, a “reasonable level of assurance” was the standard by which many decisions were made. People make judgments about “reasonable” because getting to an “absolute” level is financially prohibitive or otherwise totally impractical. You’ll see why I’m introducing this in a minute.

Did you know that your health and safety is protected by multiple choice tests? Directly from Nuclear Regulatory Commission (NRC) Guidance (NUREG-1021, Revision 9, Operator Licensing Examination Standards For Power Reactors):

In the case of operator licensing, the NRC’s written examination yields a key measure that allows the agency to make a confident decision regarding the safety-significant performance of the individual seeking a license.

Multiple-choice items are the most common and most popular of the select-type items. For reasons of consistency and reliability, they are currently the only type of items acceptable for use on NRC initial licensing examinations. Although multiple-choice items are not as easy to construct as other forms, they are very versatile, can be used to test for all levels and types of knowledge, and minimize the likelihood that the examinee will obtain the correct answer by guessing. Scoring multiple-choice examinations is also considerably more reliable and less time-consuming than scoring open-ended response items. Furthermore, since each item requires less time to answer, more items can be used to test a larger sample of K/As [Knowledge/Abilities – a direct translation to “Standards” in the education sector]. This provides better content coverage, which also increases test reliability.

If you don’t believe the NRC, check out any educational measurement text to learn more about the advantages and disadvantages of multiple choice tests. Regardless of application, creating high quality multiple choice tests is challenging. That’s why so much time and effort is required to train people, understand standards, review item performance, and confirm quality.

I think many people in education are not following the wisest path. Sure we should be continuously improving our assessments, especially if it’s done with minimal additional burden to the students and teachers to reap the added benefits; however, there is such a push for more and more standard coverage and measurement precision, the added costs of many options are growing well beyond “reasonable” for the intended uses of the assessment data.

What an assessment should provide is “reasonable assurance” that students are on track to learn the standards. We want data from our assessments that we can use to guide instruction. Provide useful feedback on progress. Understand areas of strength and opportunity so we can grow as students and educators. Support improving our processes at all levels of our education organizations.

People are pushing so hard to get to “absolute assurance” that the cost is untenable – lost instruction time, diverted financial resources, distracted teachers and leadership, and stressed and disengaged students.

I agree it’s helpful to understand each student’s achievement in the context of some standard. The question is how best to do this in a “reasonable” way? Rather than focusing on new assessments with different types of assessment items designed to get more coverage of the grade level standards within a test whose goal is to determine proficiency, why don’t we simply move to assessments designed to determine what each child really knows. Simply using standard multiple choice items in this way will result in richer data with much better precision for each child (see this previous blog for more on this topic). Let me say it simply: let’s measure what all kids know and stop measuring “proficiency”. This is my preferred path to significantly enhance the support the data provides for most educators’ desired uses without imposing an unreasonable burden.

If performance on a multiple choice test is a key factor in deciding whether to grant a nuclear power plant operator a license, surely high quality multiple choice assessments designed to measure what each student knows can provide the reasonable assurances we need in education as well.

To see related blog posts in this series, go herehere, and here.