Don't Pause – STOP!

ISTEP FAIL

The Indiana Statewide Testing for Educational Progress, called ISTEP, is a mess.

First it was too long, then it was too hard, then it was too late. And now the too-long, too-hard, too-late test has just been revealed to have been the subject of scoring errors. The Indy Star explains...

Computer glitch could have misscored thousands of ISTEP tests, scoring supervisors say

Scores on thousands of student exams could be incorrect because of a computer malfunction that inadvertently changed grades on Indiana’s high-stakes ISTEP test, according to scoring supervisors familiar with the glitch.

But the company that scored the exam on behalf of the state — testing giant CTB McGraw Hill — decided to leave those potentially faulty scores in place, even after the problem was brought to management’s attention, the supervisors said.

Company executives would not speak with The Indianapolis Star, but in a letter Tuesday to the Indiana Department of Education, Executive Vice President Ellen Haley downplayed the problem. She said the issue “was very rare” and “did not affect student scores.”

Seven supervisors who spoke with The Star disagreed. All said they believed the problem was more widespread. Two estimated that tens of thousands of test questions were likely given incorrect scores. Others said it is difficult to put a number on the problem, but it was pervasive enough to merit rescoring the potentially impacted tests.

This has inflamed bloggers, pundits, and educators all over the state. Dave Bangert sums it up...

Bangert: The ISTEP Dumpster fire

ISTEP has tumbled and crashed the way educators across the state predicted as obvious back in January and February. In those days, the dire warnings from classrooms around the state were blown off by those with Statehouse clout as just more excuses about the testing culture of school reform. That’s dereliction of a different sort.

The incompetence of CTB McGraw Hill simply removes all doubt, pounding one more nail in a coffin for a round of testing that should have been lowered into the ground long ago.

Everyone, from your local school superintendent, to the Indiana State Teachers Association, is calling for the legislature to "pause accountability" since the tests used to measure schools and evaluate teachers are so messed up. Even the Governor has agreed to ask the legislature to not use the test scores to grade teachers.

I disagree...we ought not "pause accountability." We ought to end it.

ACCOUNTABILITY

Most "reformers" don't read my blog, but if there are any out there, I'm not calling for an end to all accountability for schools and teachers. I'm calling for an end to "reform-style" accountability.

I'll explain.

Standardized tests, assuming that they're valid and reliable to begin with (another post for the future), ought to be used for that which they have been developed. In other words, student achievement tests ought to be used to measure student achievement, not to evaluate teachers and schools. By definition, student achievement tests are invalid when used to measure anything else.

James Popham, UCLA Professor Emeritus, wrote in 2001 that there were only four appropriate uses for standardized student achievement tests.

Informing parents about their children’s relative achievements
Informing teachers about their students’ relative achievements
Selecting students for special programs
Allocating supplemental resources

He also listed four inappropriate uses.

Evaluating schools
Evaluating teachers
Promoting or grading students
Making classroom instructional decisions

Even earlier (1982) was this from "Ability Testing: Uses, Consequences, and Controversies"

A test score is a numerical description of a sample of performance at a given point in time. A test score gives no information as to why the individual performed as reported.

Claiming that it does, whether intended as a positive attribute or a criticism, is tantamount to test misuse. Furthermore, no statistical manipulation of test data, even though combined with the best additional data, will permit more than probabilistic inferences about causation or future performance.

In other words, a student's score on a student achievement test should not be used to show that a teacher (or school) was the cause of the student's score.

In order to use a test in a way other than the purpose for which it was intended, the user must be able to show how the test retains its validity. To my knowledge, the developer of ISTEP has never shown how it is a valid tool for evaluating schools and teachers other than the non-research based explanation of "it shows what the kids know, therefore the teachers and schools must be responsible." (For just one of many reasons this isn't true, see Poverty and Potential: Out-of-School Factors and School Success

"Reform-style" accountability, then, insists that tests do more than they were developed to do. It insists that tests be used to evaluate schools and teachers. This is a common, though blatant misuse of tests.

I'd love to see proof that the ISTEP has been developed to include measuring the effectiveness of schools and teachers, in addition to measuring student achievement.

If there is none (and I don't think there is), we need to stop, not just pause, using it for those purposes.

For more information about ISTEP see:
A CLOSER LOOK: ISTEP SCORES, A HISTORY AND WHAT IT MEANS FOR YOUR SCHOOL

###