Friday, January 9, 2015

No price too high for standardized tests?


[This post appears as a guest opinion in the Des Moines Register and Iowa City Press-Citizen; I’m posting the published version here with some additional notes afterward.]

Should we spend three times as much on standardized testing as we currently do, or eight times as much?

That question captures the range of current establishment opinion. The establishment answer, of course, is eight times as much—and more.

Currently, Iowa requires schools to administer the Iowa Assessments every year. Because those tests are developed by the University of Iowa, we get them at cost: about $4.25 per student.

But in 2013, the state created a task force to determine whether to require a different set of assessments as of 2016-17. The task force considered two possibilities: the Next Generation Iowa Assessments and the Smarter Balanced Assessments. The former would cost $15 per student. The latter cost significantly more and would need to be supplemented with a separate science test, for a likely total cost between $30 and $40 per student. Those tests would also require unknown millions to buy and maintain the necessary technology; the task force—incredibly—didn’t even try to estimate the cost.

The tests we’ve used for years are now inadequate, we’re told. We need tests that incorporate “constructed responses” that require human graders, or tests that the students will take on computers, with technology that can adjust the difficulty of the questions to the student’s responses. We need to make the kids sit through twice as many hours of testing as they do now.

The task force recommended the most expensive tests (though not unanimously). If you suspect there are diminishing returns from all this expensive additional testing, you’re a Luddite.

What will get cut to pay for these tests? No one will say. Don’t believe for a minute that the state will fund these new tests at no sacrifice to other educational funding. Every dollar the state spends on testing will be one less dollar available for general school funding—at a time when many districts are already making severe budget cuts. No matter how it’s spun, the school districts are about to get hit with an enormous unfunded mandate.

Standardized testing is now the Defense Department of the school budget: only the most deluxe, big-ticket, exorbitant program will do. Never say no, regardless of what has to be sacrificed. The testing companies get richer; your kids’ education gets poorer.

The only thing standing between us and this enormous increase in spending on standardized testing is our legislature. If your school district has to make class sizes larger, cut band and orchestra, eliminate foreign language classes, or worse—all so we can have the shiniest new standardized tests—you’ll know who’s responsible: state legislators who decided that no price was too high.

________________


Here’s some additional information that I couldn’t fit in the guest opinion:

By not even trying to quantify the costs of the technology that will be necessary, the task force violated the legislature’s explicit charge. The act creating the task force provided that “the task force shall consider the costs to school districts and the state in providing and administering such an assessment and the technical support necessary to implement the assessment.”

The dissent to the report focuses largely on the fact that the task force did not assess the full costs of the Smarter Balanced tests. You can read it in full here.

________________


The task force report works hard to minimize the actual cost of the Smarter Balanced Assessments, stating that the tests will costs about $22.50 per student. In fact, the cost will be significantly higher, for several reasons:

First, the $22.50 figure is an estimate that comes from the Smarter Balanced Assessment Consortium—that is, by the people who have a major interest in getting the state to adopt the tests. In fact, as the Consortium admits, the actual cost will depend on what is charged by the private vendors (Pearson?) that will administer and score the tests. If those charges come in higher than the Consortium’s estimate, no one will be surprised.

Second, the Consortium’s additional “interim” tests add another $4.80 per student to the cost. The report refers to those tests as “optional,” but its inclusion of this chart seems to promote their use. (These interim tests would also add to the number of hours the kids have to sit for tests.)

Third, Iowa law requires a science assessment. Unlike the (far less expensive) Next Generation Iowa Assessments, the Smarter Balanced tests do not include a science assessment, so the state will have to pay for a separate science test. That test could cost another $10 per student.

Fourth, if some districts are not ready to conduct the tests via computer, they can use paper-and-pencil versions of the tests—for $10-12 more per student than the online versions.

For those reasons, I used an estimate of $30 to $40 per student, and that’s just for the tests themselves, not for the necessary technology. I would have preferred not to estimate at all, but the task force didn’t assess those costs itself.

________________


Then there are the technology costs. Even the task force report admits that its estimate of the per student cost does not include the tech costs that will be necessary in districts across the state of Iowa—which, as the dissent points out, are likely to be “significant and ongoing.” What will those costs be? Don’t ask the task force.

The tech costs won’t just be for equipment and bandwidth, either. They will almost certainly require ongoing expenses for tech staff to maintain the tech infrastructure, as well as ongoing upgrades. There goes a teacher.

On the lack of information about the tech-readiness of Iowa’s school districts—and on some of the debacles experienced in other states—read Karen Woltman’s post here.

________________


The task force report argues that using the Smarter Balanced tests could enable districts to save money by eliminating some other tests they are currently using. But if that’s true, it’s also true of the much less expensive Next Generation Iowa Assessments, and the task force doesn’t claim otherwise. Moreover, this argument conveniently ignores the fact that the Smarter Balanced tests do not include the statutorily-required science assessment—which means the Smarter Balanced tests will actually cause expenditures on additional tests.

________________


Though it works hard to minimize the cost of the new tests, the report also works hard to make the current tests look more expensive. Although the Iowa Testing Program lists the cost of the current tests as $3.50 per student, the report says that they cost “$4.25 to $6.25” per student, citing a document that is not available online.

The $4.25 figure supposedly comes from adding a $.50 processing fee and a $.25 barcode fee. I’m willing to trust the report about the existence of those fees, though a source would be nice.

The $6.25 figure is what districts would pay if they didn’t agree to allow some field-testing of new questions—which means no one has to pay it.

Then the report asserts that the current Iowa Assessments have a hidden $2.25 additional per-student cost because of “additional data managing and reporting” costs borne by the state. I didn’t include that figure in my estimate, for three reasons. First, the report gives no citation at all for that assertion. Second, there is no way to tell whether the report is comparing apples with apples: will there be similar “data managing and reporting” costs if we use the Smarter Balanced tests? Third, if the report is just counting money the state spends using the test results, it can’t fairly include that as part of the price of the tests.

Moreover, even if you include the $2.25, you’re only up to $6.50—a far cry from what the Smarter Balanced tests cost.

________________


One of the most striking paragraphs in the entire task force report is worth reprinting here in full:
The other costs that cause concern are related to districts’ technology readiness to support online testing. While there is no doubt that some districts are behind in technology readiness, schools will not be required to make devices and internet connections available to each and every child simultaneously. As the Consortium notes, “A 600-student middle school could test its students using only one 30-computer lab.” In these ways, the costs of upgrading school technology infrastructures are not likely to be overly burdensome on the whole. Besides, these are costs the state of Iowa should shoulder. We must better incorporate technology into the delivery and conduct of not just our assessments, but our instruction as well. The future of assessments is online, but so is the future of teaching and learning. Investing in devices and bandwidth is necessary and should be done by the state regardless.
Well, the Consortium assures us that we can test 600 students with just 30 computers, so we don’t have to bother quantifying technology costs. You might think giving 600 seven-hour-long tests on just 30 computers would be “overly burdensome,” or, say, an unimaginable nightmare. But the salesperson lobbyist Consortium told us it’s okay, so it’s cool.

But anyway, we should buy a lot of technology! We’re sure it will be useful and necessary, for lots of stuff that we can’t actually identify because we’re just an assessment task force. But that’s what the future’s all about, right—computers? So there’s no need to consider how much it will cost, even though the legislature told us to.

Okay?

________________


Don’t even get me started on the assertion that “the future of teaching and learning” is online.

________________


Notice the double standard. When the report wants to make the existing tests look expensive, it includes extra costs that it claims are borne by the state. But when it wants to make the Smarter Balanced tests look less expensive, it argues that “Besides, these are costs the state of Iowa should shoulder”—as if that somehow means they don’t cost the districts anything. Of course, any money the state spends on testing is that much less money it can spend on other educational needs, including supplemental aid to the districts.

________________


Although the task force didn’t quantify the total cost of adopting the new tests, it assures us that this unknown cost is “proportionately small,” compared to the total education budget. So what? It’s also small compared to the national debt, or to the gross national product of Bulgaria. What matters isn’t how much of the total budget it is; what matters is how we’re going to pay for the increase. What will get cut to pay for these tests? If we don’t know, how can we know whether the tests are worth having?

Instead of giving us a cost-benefit analysis, the task force gave us a benefit analysis—really, closer to an advertisement. Is that what the legislature asked it to do?

________________


One reason the Smarter Balanced tests are so expensive is that they use “computer-adaptive testing,” which means that the technology will vary the difficulty of the questions in response to a student’s answers as the test goes along. There is not universal agreement that computer-adaptive testing is even desirable, though, let alone worth paying for. Commenters make some interesting points about it here.

________________


The report emphasizes that computer-based tests will enable districts to get the test scores back much more quickly. Is that worth paying a lot of money for? The tests are going to be given toward the end of the school year. If Susie gets a lower-than-hoped-for “problem solving” score, will the school somehow re-teach that year’s math curriculum to her in the last month of school? How would they even know what to fix?

If there is a connection between the quick score reporting and the improvement of your child’s education, you won’t find it explained in the task force report. Am I wrong to think that, at best, the scores will be used primarily to evaluate curricular choices and recommended teaching strategies, which schools don’t change overnight? (Large bureaucracies aren’t exactly known for being nimble.) If the ultimate point of the scores is to improve the curriculum next year and beyond, what is gained by getting the scores in days instead of weeks?

________________


The report also asserts that the new tests will provide “more precise scores for Iowa students.” I have a lot of concerns about K-12 education, but I have to admit that I have never complained about my kids’ scores on the annual standardized tests not being precise enough. In any event, call me skeptical that any test will ever “precisely” measure a kid’s “academic performance.” (See this post.)

The report also asserts that the tests will measure kids’ “ability to transfer their learning to real-world situations”—by giving them a standardized test in a classroom. Again, call me a skeptic. (See this post.)

The report also asserts that the tests will help our students “compete against students across the country and beyond” in “today’s global economy,” because the same tests have been adopted by “21 states and a U.S. Territory.” Now Iowa can finally prevail in its age-old rivalry with the U.S. Virgin Islands.

On those topics, more in upcoming posts. In the meantime, I can’t resist raising one question: how much do you want to bet that the same kids who do well on the current (inexpensive) tests will also do well on the new (very expensive) ones? If so, then what is actually being measured?

________________


What’s missing from the report is any concrete explanation of how these tests will be used to improve your child’s education. There are all kinds of fun, expensive things you can do with data. Many of them have no meaningful bearing on real life. A marginally more “precise” “total writing” score might be a victory for the test designers, but just how does it make your child’s life better? Through what series of events does that occur?

I came away from the report with the discouraging sense that our state education bureaucracy has really lost its way. Nobody wants to admit that the ability of empirical data to tell us how to turn kids into capable, independent adults is severely limited. Standardized testing has legitimate uses, but it’s not vested with magical powers. It’s almost as if education officials have constructed a reassuring, imaginary universe in which their job is to manipulate abstractions—like in those infernal box-folding questions on the tests—rather than to deal with the messiness of the living, breathing, infinitely varied bodies and minds who arrive in the classrooms every morning.

They can dream. But if they want us to spend millions of dollars, they owe us much more specific, realistic information about costs, benefits, and what will get cut.
.

6 comments:

Chris said...

A commenter at the Register argues that education is expensive and so I shouldn't be worried about spending $35 to find out "if it worked."

What I'm worried about is spending $35 dollars (and significantly more) when we could get information that's just as useful for $4.25, and about multiplying that by nine grades and hundreds of thousands of kids every year, and about what will get cut from our kids' school programming as a result.

Whether any tests will tell us "whether it worked," and what use we will make of that information, are topics it would be interesting to hear discussed further -- keeping in mind that variations in test scores are probably explained as much by household income as by anything else.

Chris said...

On Twitter, Matt Townsley wishes that I had mentioned that the current tests are not aligned with the Iowa Core standards. He’s right that the desire for new tests is driven by Iowa’s adoption of the Common Core—though many people will not hear that as a point in its favor.

I see the Common Core and the push for more (and more expensive) standardized testing as essentially one enterprise. As I wrote in last year’s blogathon, I’m not a fan. The project strikes me as a huge misdirection of money and effort in ways that ultimately undermine education rather than furthering it. It seems to put an almost utopian faith in the ability of father-knows-best central planners to use data and standards to improve kids’ lives, and it has no connection to any reality that I can see. I can see how it gives bureaucrats something to do and makes money for private vendors, but I don’t see how it’s worth millions of dollars and sacrifices in actual teaching and programming. More in a future post.

Chris said...

A commenter at the Register assumes that I’m against the new tests because I don’t want a tax increase. First: No one has suggested that these tests will be paid for with tax increases. In fact, Governor Branstad, whose Department of Education wants the new tests, has pushed tax cuts, not increases. It’s clear that the tests will be paid for with cuts elsewhere, not with tax increases. I think it’s fair to ask where those cuts will be.

Second: I’m happy to pay more taxes for a better world. If it were up to me, we’d raise taxes on people who can afford it (including me) and use the money to make a serious dent in child poverty. That would do far more for education than the very speculative benefits of gold-plated standardized testing would.

On that topic, interesting posts here and here.

Anonymous said...

Hello!

We need to ditch the essays and stuff that needs to be "human scored"... they increase the price of the test many times over and aren't reliable items.

We need norm-referenced test at national level so the parents can kids can see where they fit in a large sample... instead of the gerrymandered CRT tests that the psychometricians can make say whatever the state wants the results to say (by changing demo information or including or discluding kids).

Then the price would be about 1.80 a kid... and the information would be USEFUL -- not political.

If Lindquist knew what has happened to his "achievement tests" of yore, he'd be spinning in his grave.

Peace to you. Friend. Cornelius.

Steve Peterson said...

You said: "just how does it (SBA) make your child’s life better? Through what series of events does that occur?"

As a teacher, yours are the key questions to ask of ANY assessment whether they be SBA, IA Assessments, or my own curriculum-based assessments. Every bit of data I gather (and data-gathering is constant in a classroom) needs to be gathered in order to make the learning of that particular child, or children in general, better. Data are easy to gather; using them to better educate each child is the hard part that requires systemic change (and a deeper sense of what a "good education" looks like and how one might "measure" it.) We need to consider the answers to your questions from the very beginning.

Chris said...

I saw some concern on Twitter that it’s wrong to use the current Iowa Assessments as a baseline for cost comparisons, because the job of the task force was to adopt tests that are aligned with the Common Core, and the current tests are not.

But the post is about the impending increase in the amount of money being spent on the tests. Using the current costs as the baseline makes perfect sense. Otherwise, you couldn’t possibly discuss the increase in cost.

If we don’t discuss the cost increase when we decide to adopt the Common Core, and then we don’t discuss it when we’re choosing what test to use (because now we have to have Core-aligned tests!), when do we ever discuss the cost increase?

In retrospect, I do wish I had included “We need tests aligned with the Common Core” as one of the rationales in the fifth paragraph. But that paragraph is not talking about the rationale for having new tests, it’s talking about the rationale for those tests costing three, eight, or more-than-eight times as much as the tests we use now. As I understand it, it’s the costs of human graded questions and/or computer-adaptivity, not the new subject matter per se, that is largely responsible for the increased cost. (And let’s not exaggerate the newness of the “subject matter.” The Common Core hasn’t utterly transformed the content of math and reading comprehension.)

But there is certainly a connection to the Common Core. I assume the defenders of the new tests would justify their expensive features on the grounds that they are the only way to assess the “twenty-first century skills” that the Core claims to identify. It’s also true that the sheer volume of the Core standards (four hundred pages!) might be cited as a reason for spending so much more time and money on testing.

(Defenders of the Common Core standards often argue that the need for uniform standards is a separate issue from the role of standardized testing. But if the answer to “Why do we need to spend so much more time and money on testing” is “because we need the Common Core,” then it’s hard to see them as separate enterprises.)

In any event, I don’t think saying “We need Core-aligned tests” would do much to explain the cost increase to the average person. There are, of course, whole opinion pieces (and blogathons) that could be written about the desirability of adopting the Common Core, but that’s not what this piece was intended to be. I do wonder how many pro-Common-Core pieces ever bothered to mention that they would necessitate huge increases in spending on standardized testing.