Scientific findings vs. pseudo-scientific conclusions

The most emailed article in New York Times this morning is about a study of the effectiveness of testing in helping students learn. “Several cognitive scientists and education experts said the results were striking,” the article notes. What’s most striking, though, is the contrast between what the researchers found and what the article concludes.

What the study found was:

[S]tudents who read a passage, then took a test asking them to recall what they had read, retained about 50 percent more of the information a week later than students who used two other methods.

Got that? One week later, the kids had a better memory of the content of what they read than if they had studied in a different way for that second test.

Now, according to the Times, this is earthshaking stuff.

Taking a test is not just a passive mechanism for assessing how much people know, according to new research. It actually helps people learn, and it works better than a number of other studying techniques. . . .

“I think that learning is all about retrieving, all about reconstructing our knowledge,” said the lead author, Jeffrey Karpicke, an assistant professor of psychology at Purdue University. “I think that we’re tapping into something fundamental about how the mind works when we talk about retrieval.” . . .

“[W]hen we use our memories by retrieving things, we change our access” to that information, Dr. Bjork said. “What we recall becomes more recallable in the future. In a sense you are practicing what you are going to need to do later.” . . .

The Purdue study supports findings of a recent spate of research showing learning benefits from testing, including benefits when students get questions wrong. . . .

Dr. Kornell said that “even though in the short term it may seem like a waste of time,” retrieval practice appears to “make things stick in a way that may not be used in the classroom.

“It’s going to last for the rest of their schooling, and potentially for the rest of their lives.”

(Emphases mine.)

Now, to this wet-blanket reader, a few questions leap to mind. The first one is: Why was one week chosen as the relevant period? How much of this effect is still apparent a month later? How about eighteen years later? Another is: Is retention of content really the goal of our reading comprehension instruction? Is education really just the search for effective memory strategies? Another is: What educational values will be sacrificed if we make raising test scores our supreme goal? And what unintended consequences will ensue if teachers and schools are penalized for failing to get the kids’ test scores up? And so on.

Empirical evidence is useless unless we first settle on our goals, which requires a discussion of values. But in so much of what passes for educational science, the evidence drives the goal, instead of the other way around. Testing improves one-week retention of content; therefore we must test! Exaggerating scientific findings, examining them in isolation, and accepting conclusions uncritically only compounds that basic error. If the Times article is any indication of how empirical evidence gets used in our school policy discussions, our educational system is failing even more than we thought.
.