Write by numbers

The Times reports on a new study claiming to show that computers are as good at grading student essays on standardized tests as people are – proving once again that it’s easy to achieve reliability if you toss validity out the window.

As a counterpoint, the article quotes Les Perelman, who studies the algorithms that the Educational Testing Service uses. According to Perelman, ETS’s “automated reader can be easily gamed, is vulnerable to test prep, sets a very limited and rigid standard for what good writing is, and will pressure teachers to dumb down writing instruction.”

The e-Rater’s biggest problem, he says, is that it can’t identify truth. He tells students not to waste time worrying about whether their facts are accurate, since pretty much any fact will do as long as it is incorporated into a well-structured sentence. “E-Rater doesn’t care if you say the War of 1812 started in 1945,” he said.

To prove his point, Perelman wrote an essay explaining how the high costs of college are the fault of greedy graduate teaching assistants.

“The average teaching assistant makes six times as much money as college presidents,” he wrote. “In addition, they often receive a plethora of extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures.”

The computer gave his essay the highest score, as it did another Perelman essay that was “padded with more than a dozen nonsensical sentences.”

ETS officials are unfazed by Perelman’s criticisms. “E.T.S. officials say that Mr. Perelman’s test prep advice is too complex for most students to absorb; if they can, they’re using the higher level of thinking the test seeks to reward anyway.” Think about the implications of that statement: It doesn’t matter if the test really measures writing ability, or whether it will cause teachers to turn English classes into test-prep sessions. It’s fine if you can game the test, because test-gaming is just as important an ability as writing well. It’s the ultimate, all-purpose justification for turning schools into test-prep centers!
.