What's Bugging Me Today: Testing Ignorance - RTFM

What's bugging me today is people who interpret standardized test results without understanding what the scores, and the labels attached to those scores, mean.

Even from people I agree with...

THE STATE OF AMERICA'S CHILDREN

Late last year, the Children's Defense Fund issued its 2017 report on the State of America's Children.

The State of America's Children® 2017 Report

This is a very challenging and scary time for America’s children. As new policies threaten to eliminate the safety net that millions rely on to survive, the reality is millions of America’s children today are still suffering from hunger, homelessness and hopelessness. The Children’s Defense Fund’s new report, The State of America’s Children® 2017, provides a comprehensive look at the status of America’s children in 11 areas: child population, child poverty, income and wealth inequality, housing and homelessness, child hunger and nutrition, child health, early childhood, education, child welfare, juvenile justice and gun violence.

Indiana's numbers in those areas are as disturbing as the numbers for the nation as a whole. It's shameful that a nation as wealthy as the United States allows one-fifth of its children to live in poverty. It's horrifying that more children have been killed by guns since Sandy Hook than U.S. soldiers in combat since 9/11. We need to make the children of our nation, and locally, of Indiana, a priority. Our future, literally, depends on it.

You can click either of these links to reach the Children's Defense Fund website to download

• the entire report

• the state summaries

WRONG ABOUT THE NAEP

I agree with the Children's Defense Fund about the state of children in the U.S., but they're wrong about testing. They're not wrong that schools need to improve, but they're wrong about how they interpret the need for improvement based on the National Assessment of Education Progress (NAEP).

The report's section on education begins with this graphic:

The statement that 67% of eighth graders are unable to read at grade level is based on results from the NAEP, the "Nation's Report Card."

The number, 67%, is reported on page 59 of the Children's Defense Fund full report. Below is the table from page 59, cropped to show the national results...

Note that the table purports to display, according to its title, the percentage of eighth graders who were performing below grade level.

You can see that 67% is listed under All Students on the Reading side of the table. Below the table, we're told that the source for this data is the

U.S. Department of Education. 2016. "2015 Mathematics and Reading Assessments Report Card: Summary Data Tables with Additional Detail for Average Scores and Achievement Levels for States and Juristictions."

This is followed by a link to the information

https://www.nationsreportcard.gov/reading_math_2015/#reading/scores?grade=4

If we dig a little deeper, however, we find that the authors have mistakenly equated the term Grade Level with scoring Proficient on the NAEP. This is incorrect. Proficiency, on the NAEP, has nothing to do with "grade level."

Misrepresenting the achievement levels leads to graphics like the one above, and headlines like the one below. Fortunately, the article below also explains that the achievement level, proficient, doesn't equal grade level. I only hope that the Senator read the article.

See also: What Does 'Proficient' on the NAEP Test Really Mean?

PROFICIENT vs BASIC

The NAEP Glossary of Terms defines NAEP's three achievement levels, Basic, Proficient, and Advanced.

Basic is defined as...

...denoting partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade assessed...

Proficient is...

...representing solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter...

In the pamphlet, A Closer Look at NAEP, the National Assessment Governing Board (NAGB, an independent, bipartisan organization that oversees the NAEP) debunks myths about the NAEP.

Myth: The NAEP Proficient level is like being on grade level.

Fact: Proficient on NAEP means competency over challenging subject matter. This is not the same as being “on grade level,” which refers to performance on local curriculum and standards. NAEP is a general assessment of knowledge and skills in a particular subject.

When Diane Ravitch, a former member of the NAGB, discussed this topic in her book Reign of Error, she wrote,

From what I observed as a member of the NAGB who reviewed questions and results over a seven-year period, a student who is "proficient" earns a solid A and not less than a strong B+.

"Basic," as defined by the NAGB, is "partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade." In my view, the student who scores "basic" is probably a B or C student.

This means that the 67% of students who scored below proficient on the NAEP's 8th grade reading test were not honor students, not that they were "below grade level." Students who are "proficient" are high achieving students. Students who are "basic" are average, and students who are "below basic" are the ones who are at risk of failure. 67% of students below "proficient" does not mean that 67% failed the test!

In fact, 76% of eighth graders scored at "Basic" or above on the NAEP nationally. That's still not perfect...and some might argue that it's not even acceptable, but it's much better than the mistaken assumption that "67% of eighth graders score below grade level."

A more serious issue, is that, while 85% of white students scored at Basic or above, the number of black, hispanic, and other minority students scoring Basic or above is much lower. That should be the focus of the Children's Defense Fund's report on Education...

For further reading on this issue see:

• Reports show Indiana school funding lagging

• We read new reports on the state of school funding in America so you don’t have to. Here’s what we learned.

• Report finds most states shortchange high-poverty schools

THE DANGER OF MISUSING WORDS

The Children's Defense Fund is not entirely at fault, however. When the National Assessment Governing Board (NAGB) releases scores and reports, they use the "proficient" level as the default level of discussion which makes education in the U.S. – as measured by the NAEP – sound much worse than it really is.

This is a serious problem and at least one educator, James Harvey, has suggested that NAEP change the wording of the achievement levels to prevent further distortion of our students' achievement levels [emphasis added].


The Problem with "Proficient"

...recommends replacing the terminology NAEP currently applies to its performance levels (Below Basic, Basic, Proficient, and Advanced) with the performance levels employed in international assessments: Low, Intermediate, Advanced, and Superior. This simple change in terminology would go a long way toward reducing the confusion the term proficient has introduced into the national discussion of school performance. And we should educate the public about the flaws embedded in these benchmarks and emphasize to everyone the caution that Congress has always assigned to them. It would also be highly desirable if the views of independent psychometricians and assessment experts guided NAEP's thinking about other technical judgments that could improve NAEP.

In the meantime, the Children's Defense Fund, and everyone else who reports U.S. students' achievement based on NAEP test results, should take the time to learn what Basic, Proficient, and Advanced, actually mean.

In other words, before you try to interpret test scores...

📝✏️📊