If you look at a graph of test scores for Chicago elementary schools, you see a line that goes up and up over the years. Reading scores are soaring, math scores too.

Forget that, says researcher Elaine Allensworth.

ALLENSWORTH: In the elementary schools, reading scores have really been flat for the last 20 years.

That statement right there turns on its head everything we’ve been told about Chicago’s test scores. 

Allensworth and her colleagues at the University of Chicago’s Consortium on Chicago School Research have done something no one else has. The highly respected group released a dramatic study Friday linking the major standardized tests kids took over the past two decades, the Iowa Tests of Basic Skills and the Illinois Standards Achievement Test, known as ISAT.

Using a complex statistical analysis, they came up with some indicting findings:

For one, African-American elementary school students are reading at exactly the same level they were 20 years ago.

Chicago has been home to big reforms in that time—reforms that attracted national attention. Elected councils at every school. High-stakes testing and accountability. Failing schools shut down, and over 100 new schools opened.

ALLENSWORTH: We’ve had some dramatic reforms that have been enacted over these 20 years, many of them aimed at those poorest-performing African-American schools. I think we can say there are no silver bullet solutions, that dramatic reforms don’t necessarily result in dramatic improvements.

DUNCAN: This work is tough, it takes time.

Arne Duncan was CEO of Chicago Public Schools during eight of the years covered in the report. He’s now U.S. Secretary of Education, and some of the strategies tried in Chicago—like completely overhauling schools—are now national education policy.

DUNCAN: Anyone who thinks you should stand idly by and perpetuate the status quo—that’s gone on for far too long in this nation. And across the country you now have about 1,000 schools that are being turned around.

Another bombshell dropped by this study: Achievement gaps between white and minority students actually widened, especially during the past decade.

TEACHER: I want everyone to put their books away and go sit on the carpet.

The Consortium study found that one type of school showed bigger gains than others: schools with at least 30 percent white students.

Alcott Elementary in Lincoln Park was one of those. Teacher Donna Kaufman has worked here since the 1980s. She says Alcott was transformed when the principal opened the door to parents, encouraged collaboration among teachers, and convinced neighborhood families to try the school.

KAUFMAN: My first parent was after 9/11. And she said she wanted her child to be close to home because if anything ever happened she wanted to be able to run from her house and pick her child up at school. So she put her daughter in my classroom. She said, ‘This school needs to make some changes.’ She bought me all new desks for my classroom, gave me supplies. It was pretty amazing.

The bright, colored chairs still furnish Kaufman’s first grade classroom. Parents here fundraise to buy extra teachers. They even staff the library. Consortium researchers say having that sort of social capital makes it easier to improve schools.

Not everyone is buying the Consortium’s analysis. Peter Cunningham has been the spokesman for Arne Duncan since he was Chicago’s school CEO.

It’s hard for Cunningham to believe that years of test scores issued by the state—scores that showed improved reading and narrowing achievement gaps—well, it’s hard for him to believe that was all wrong.

CUNNINGHAM: These were the results we were seeing. And based on those result we were pushing forward with our efforts to improve schools. By all indications we were seeing real gains. Now a report comes out and says the test wasn’t really valid and those gains really didn’t happen—and I think if anything this is a challenge to the state of Illinois to come up with a better test.

Researchers wouldn’t disagree with that. They blast the way Illinois reports test scores, saying  the scores don’t accurately reflect progress students make—for all kinds of reasons. The tests change, passing scores are lowered, kids are excluded.

The report was not all bad news. For years, high school test scores have appeared stubbornly flat, frustrating school officials. But this report shows steady improvement in ACT scores, and huge gains in graduation rates.

The Consortium researchers say test score data will continue to shape school policy. But that data better be sterling, they say. To come up with the right fix, you need to know precisely what’s broken.