The “streetlight effect” and the Great and Powerful Oz

Esther Quintero sums up the misguided use of data that is the central characteristic of so much of today’s educational policymaking:

Remember the parable about the drunk man searching for his wallet under a streetlight? When someone comes to help, they ask “Are you sure you dropped it here?” The drunk says, “I probably dropped it in the street, but the light is bad there, so it’s easier to look over here.” In science, this phenomenon – that is, researchers looking for answers where the data are better, “rather than where the truth is most likely to lie” – has been called the “streetlight effect.”

Quintero questions whether people are “develop[ing] the ideas to fit the data they have, rather than finding the data to test the most important ideas.” She concludes that:

Excessive faith in data crunching as a tool for making decisions has interfered with the important task of asking the fundamental questions in education, such as whether we are looking for answers in the right places, and not just where it is easy (e.g., standardized test data).

I think Quintero’s post is terrific (read the whole thing), but I wish she had gone further. Why are so many people attracted to using data in this utterly unscientific way? Quintero generously assumes that everyone is acting in good faith in trying to bring data to bear on policy, and concludes that many people just aren’t thinking deeply about what data can tell us. I wish she had considered whether some people might be using data for other purposes. Yes, if your purpose is to shed light on policy questions, much of today’s discussion of data is very misguided. But if your purpose is to justify preconceived conclusions, and to deter laypeople from examining them closely, and to squelch discussion, then it’s enough that your data look impressive on the surface and be accompanied by an academic-looking citation. Some people use data as a light; others use it as a club.

Are our educational policymakers stumbling drunkenly in the wrong area, or are they more like the Wizard of Oz, fraudulently extracting allegiance with smoke and mirrors? Pay no attention to that man behind the curtain!
.