Wednesday, April 4, 2012

Educational thalidomide, or What’s wrong with “evidence-based education”

Pregnant women should take thalidomide because research shows that it’s effective.
What’s wrong with that assertion? It’s true that thalidomide has been proven effective, as measured by its ability to reduce morning sickness. Why shouldn’t pregnant women take it, then? We all know the answer: because it has other effects, too – some of which, as we all now know, are terrible. Before taking a drug, you’d want to know what all of its effects were, and whether the good effects outweighed the bad. No one would argue that we should reduce morning sickness at all costs.

But that basic principle is completely absent from most discussions of empirical research on education. Educational “reformers” are eager to proclaim that their proposals “work” based on very incomplete information about their effects on children. If an intervention appears to cause short-term standardized test scores to rise, it is enthusiastically embraced, regardless of what its other effects may be. But does anyone think that we should raise test scores at all costs? If beating the kids made their test scores go up, is that all you would want to know about the practice?

Ultimately, test scores are just a proxy for something much harder to measure: whether we have achieved our educational goals by the time our kids reach adulthood. But as soon as you put it that way, you realize how pathetically inadequate those proxies are. Education has lots of goals: we want our kids to be proficient in reading and math, but we also want them to grow up to be independent, to be intellectually curious, to be appropriately skeptical, to be self-supporting, to be mentally and physically healthy, to participate willingly and capably in democratic self-government, to be ethical, and assertive, and honest, and brave, and kind, and fair, and to lead lives they find fulfilling. Many of these goals are impossible to assess empirically, especially over the long-term. Even if we could measure them, a rise in one value would often come at the expense of another. Trying to raise test scores too single-mindedly, for example, can produce stressed-out unhappy people who dislike learning. Similarly, an exclusive focus on reducing disciplinary referrals can turn schools into obedience training academies that teach authoritarian values. Where is the assessment of those effects?

“Evidence-based education” has an army of enthusiasts, but I have never heard a response from any of them on this basic point. There are gaping holes in their evidence, and they don’t seem to care at all. Yes, analysis is easier if you don’t hold more than one value in your head at a time, and if you don’t ask any questions that you don’t want to know the answer to. But is that the sign of a scientific mindset, of a respect for evidence and empiricism?

What is the response?
.

5 comments:

Chris said...

Karen W. has a good post on other limitations of educational research here.

Billy Zelsnack said...

"Yes, analysis is easier if you don’t hold more than one value in your head at a time, and if you don’t ask any questions that you don’t want to know the answer to."

5 thumbs up!

Chris said...

Thanks, Billy. Maybe that's just another way of saying that education is more of an art than a science.

Duane Swacker said...

Chris,

I hope you don't mind my postings as some can be a tad long but I believe that there are some very important ideas that need to be "out in the dialogue".

What you say in “Ultimately, test scores are just a proxy for something much harder to measure:. . . Many of these goals are impossible to assess empirically, especially over the long-term. Even if we could measure them, a rise in one value would often come at the expense of another.” is certainly true. No, the educational deformers will not answer your questions/concerns-it cuts into their profit making schemes. And no, I do not believe for a moment that the deformers have altruistic reasons for saying what they say.

But I would like to take your thoughts back a step to a more foundational problem with any system of grades, educational “standards” and standardized testing and that is: Teaching and learning belong in the logical category of “quality”. Logically speaking one cannot determine quality in terms of quantity, they are two different things. To do so obtains a logical error.

“There are ten (quantity) apples” does not tell us anything about whether those apples are any good to eat (quality). Johny got 11 out of 20 answers correct on the test tells us absolutely nothing about the quality/characteristics of Johny’s learning. But in most school systems that score would be labelled “failing”. We add to the falsehood by adding the label. Any attempt to quantify student learning-grading, standardized testing, “real data”, etc. . . is a falsehood. When one bases a practice on a falsehood the results will, more likely than not, be false. (Every now and then one may obtain a correct answer/conclusion by chance).

The educational practices of grading, labeling, etc . . . are so ingrained into us that to question the practices is to be considered by most to be a question of a mentally deranged person. I know, I’ve seen the looks teachers and administrators have given me when I bring this up. When I point out the fact that so much of what we in public education do, especially with regards to “data driven decisions”, in essence is a falsehood and therefore a huge waste of time and energy that could be better spent on the actual teaching and learning process, they think I’m nuts. And sometimes I do to because people don’t want to know that “the emperor has no clothes”.

For the definitive destruction of educational standards and standardized testing please see N. Wilson’s “Educational Standards and the Problem of Error to be found at: http://epaa.asu.edu/ojs/article/view/577 .

For a shorter essay on the invalidity of standardized testing see Wilson’s “A Little Less than Valid: An Essay Review”.
http://www.edrev.info/essays/v10n5.pdf .

Chris said...

Duane -- I agree. Education is a sufficiently complex human interaction that attempting to use quantitative methods to dictate policy are likely to do more harm than good. Moreover, any effort to quantify would have to be preceded by a discussion of goals and values. What do we want for our kids? Just high test scores, at any and all costs? That seems to be the assumption behind most of the quantitative measurement we hear about today.