In the last few weeks a lot has been written about using computers to score student writing. Harvard and MIT offered through EdX, their joint venture to deliver massive open online courses (MOOCs), automated essay scoring (AES). In fact, they offer the software free to any university that wants to employ it. This has animated those critical of using computer algorithms to score writing to launch a renewed round of critiques of the practice. Coincidentally, the National Council of Teachers of English (NCTE) just released a position paper that, not surprisingly, states that AES is not adequate and cannot replace the teacher as an evaluator of student writing.
This is an issue that should be of interest to those implementing the Common Core State Standards because of the emphasis on student writing that the standards demand. Embedded in the drive for career and college readiness is a focus on students being prepared for the writing demands they will encounter in college. In the NCLB environment writing, like many other subjects, often got shortchanged in the effort to focus on reading and mathematics and on achieving AYP in those two areas. As state assessments shift from NCLB assessments to Common Core assessments, the time devoted to writing will increase in all districts where tests help drive how instructional time is allocated.
The questions around this shift are many. Are teachers prepared for the increase in writing and the scoring demands that increase implies? Where will the time come from in a teacher’s already busy schedule? How prepared are teachers to evaluate student writing? How prepared are teachers to provide meaningful and useful feedback to students? And finally, is there a place for AES in this picture?
The issue of the role of AES in a writing program is a broad issue with many ways to think about it and much confusion around it. For example, one bit of confusion concerns the two educational assessment consortia. Some of the reading I am doing about this issue assumes that because the consortia plan to offer computer based assessment, they are planning to use AES to evaluate student writing. This is not the case at all. Much of the new consortia developed tests can and will be machine scored, but the plan is to use human raters to score writing.
A second bit of confusion around AES—and here I want to note that critics often call AES “machine scoring,” which strikes me as having more negative connotations—is the well-documented fact that AES algorithms can be gamed. That is, a critic of AES can write a nonsensical piece that the AES engine will score with a high score point. Critics cite this fact as a fatal flaw in AES. However, to write the “hot mess” that receives a high score, the critic must be fully versed in many of the aspects that make writing strong: a wide vocabulary, a variety of sentence lengths, a variety of sentence types, use of transitions, grammatical correctness, etc. In other words, an AES can only be tricked by a good writer. What is true is that users of AES need to be fully aware what the AES does well and what it doesn’t do well.
I believe if a curriculum developer is willing to take the time to understand both the strengths and weaknesses of AES, they can likely find ways to implement AES in their instructional program in ways that benefit both evaluation of student writing (assessment of learning) and the writing feedback loop necessary in good formative practice (assessment as learning). With an increase in student writing almost mandated by the CCSS and the upcoming assessments, anything that can help teachers manage the paper load should be explored. In future posts, I will suggest the uses and limitations around AES in both roles mentioned and discuss some of the specific findings of NCTE.