Friday, January 21, 2011

Scientific findings vs. pseudo-scientific conclusions

The most emailed article in New York Times this morning is about a study of the effectiveness of testing in helping students learn. “Several cognitive scientists and education experts said the results were striking,” the article notes. What’s most striking, though, is the contrast between what the researchers found and what the article concludes.

What the study found was:

[S]tudents who read a passage, then took a test asking them to recall what they had read, retained about 50 percent more of the information a week later than students who used two other methods.
Got that? One week later, the kids had a better memory of the content of what they read than if they had studied in a different way for that second test.

Now, according to the Times, this is earthshaking stuff.

Taking a test is not just a passive mechanism for assessing how much people know, according to new research. It actually helps people learn, and it works better than a number of other studying techniques. . . .

“I think that learning is all about retrieving, all about reconstructing our knowledge,” said the lead author, Jeffrey Karpicke, an assistant professor of psychology at Purdue University. “I think that we’re tapping into something fundamental about how the mind works when we talk about retrieval.” . . .

“[W]hen we use our memories by retrieving things, we change our access” to that information, Dr. Bjork said. “What we recall becomes more recallable in the future. In a sense you are practicing what you are going to need to do later.” . . .

The Purdue study supports findings of a recent spate of research showing learning benefits from testing, including benefits when students get questions wrong. . . .

Dr. Kornell said that “even though in the short term it may seem like a waste of time,” retrieval practice appears to “make things stick in a way that may not be used in the classroom.

“It’s going to last for the rest of their schooling, and potentially for the rest of their lives.”
(Emphases mine.)

Now, to this wet-blanket reader, a few questions leap to mind. The first one is: Why was one week chosen as the relevant period? How much of this effect is still apparent a month later? How about eighteen years later? Another is: Is retention of content really the goal of our reading comprehension instruction? Is education really just the search for effective memory strategies? Another is: What educational values will be sacrificed if we make raising test scores our supreme goal? And what unintended consequences will ensue if teachers and schools are penalized for failing to get the kids’ test scores up? And so on.

Empirical evidence is useless unless we first settle on our goals, which requires a discussion of values. But in so much of what passes for educational science, the evidence drives the goal, instead of the other way around. Testing improves one-week retention of content; therefore we must test! Exaggerating scientific findings, examining them in isolation, and accepting conclusions uncritically only compounds that basic error. If the Times article is any indication of how empirical evidence gets used in our school policy discussions, our educational system is failing even more than we thought.
.

6 comments:

StepfordTO said...

Great post, Chris! I just tweeted a link to it.

Chris said...

Thanks northTOmom. By the way, here's another article about a study with a one-week time horizon on the subject of how people learn. Notice which one ended up in the New York Times.

Seth Coster said...

There's a simple function at play in the commercial media environment. If you tell your readers what they need to hear rather than what they want to hear, you may lose readers. If you lose readers, you lose advertising revenue. Lose too many readers, and you're finished.

Sadly, I've stopped taking almost any commercial news source seriously; dedicated bloggers often seem to be more interested in the truth. But that's all right; I think it won't be long before news corporations are struggling to keep up with freelance bloggers and the viral spread of media through social networking.

Chris said...

Stoz -- Thanks for commenting! I have as many gripes about the MSM as anyone else, but I still feel a little anxious about the death of newspapers. I don't know that I want to depend on a bunch of spare-time bloggers like me to fill the void.

Chris said...

[Sarah tried to post this comment, but Blogger seems to be giving people a hard time, so I'm posting it for her:]

Chris,

Have you seen Alfie Kohn's response to this article? The Answer Sheet on the Washington Post ran it here. I thought Kohn was right in discussing how the media can really misinterpret studies. The NYTimes was not clear that the study was done on undergrads. They also did not emphasize how the study was not about high-stakes testing, but about the basic recalling of facts. I think the study could be interpreted to show that ongoing assessment can be more helpful in determining a child's learning than taking one big test a year.

Chris said...

Thanks, Sarah, for that great link. I also like his article on the abuse of research (which focuses on research about the value of homework), which he links to in that post.