Researchers: Test scores released today mislead the public

For a real picture of how schools are doing, experts say forget about the "percent meeting and exceeding" standards.

November 1, 2011

Download Story

School-by-school test score results are here.

Today is the day the public gets to see how its schools are doing. Illinois is releasing test scores for every school in the state. Increasingly, researchers say the way the state reports those scores is misleading. WBEZ’s Linda Lutton tell the story.

So part of my job as an education reporter is to slice and dice test scores. And I have some big-time researchers I talk to about this stuff.

Meet Stuart.

LUPPESCU: Ah, Stuart Luppescu, chief psychometrician, Consortium on Chicago School Research.

That’s at the University of Chicago. It’s a big deal. Stuart sorts through hundreds of thousands of student test scores--all the time. He looks at them on his two computer screens. 

LUPPESCU: Yeah, I’ve got a 24-inch monitor and a rotated 17-inch monitor put together.
LUPPESCU: All right, so what’s the story? 

I ask Stuart what sort of attention he thinks people should pay to the scores released today.

LUPPESCU: So Hhhhhhh. [long sigh] Oh, geez. I don’t know—you know. It’s, um…I wouldn’t make a big deal about it, OK?

See, the scores the state releases to the public show the percentage of kids who meet and exceed state standards. Stuart says that way of reporting kids’ test scores is…

LUPPESCU: Misleading, inaccurate, and sends the wrong message to the public.

Stuart works with the same scores, but in raw data--kid by kid, school by school. Not just the percent that "meet or exceed standards."

LUPPESCU: What’s wrong with "percent meets and exceeds"?

Stuart and his colleagues just finished a big analysis that touches on this. It was front-page news

Their numbers showed elementary students barely improved, but the state reported skyrocketing percentages of kids meeting and exceeding standards during that period.

And Stuart found widening achievement gaps between white and minority kids. The way the state reports those scores, the gaps are narrowing.

So how can the same test scores reported two different ways be so different? Stuart told me to think of a race. A qualifying race.

COACH: We’re at Washington Park here, and the 5th and 6th grade boys are about to run a mile to qualify for the city championships next Saturday.

LUPPESCU: You know if you talked about the percentage of people who qualified for a race, it doesn’t really tell you how fast people ran.

Stuart says the public basically never finds out how fast kids run. Schools don’t even know what their average score is. We’re just always finding out how many kids qualify for the race—how many “meet or exceed” standards.

Then there’s this:  kids could be improving. They might be running a lot faster. But their effort doesn’t show up unless they cross that line and meet standards.

The superintendent in Evanston-Skokie District 65 has another analogy. Hardy Murphy talks about kids jumping over a stream.

MURPHY: There’ll be some students that can just land on the other side, and not land in the water. There’ll be some students that jump far beyond the other bank.

District 65 has begun reporting scores that show how far kids actually jump at its schools—not just whether they get over the stream.

It’s the only district reporting how its students stack up compared to others in the state.

And it’s also calculated the scores students need to get to college—and tells the public how many students hit those goals, grade-by-grade.

MURPHY: The percent meeting standards? That standard was about whether kids can read, whether they can do math. This new standard is about whether they can read well enough to go to college. There’s a big difference.

Much of this has happened because of another pesky researcher.  Paul Zavitkovsky of UIC says Illinois could be reporting everything Evanston is. Instead, the state is mostly looking forward, to new tests coming in 2015.

Zavitkovsky would rather not wait. These scores help make lots of decisions in education—from school closings to a teacher’s focus in the classroom.

ZAVITKOVSKY:It’s very frustrating to watch policies be built on numbers that really don’t mean much.

The percent of students meeting or exceeding standards at all schools are posted to the internet.  If you really want to know how well your school is doing, maybe I can put you in touch with Stuart.