Chicago Public School officials and former Mayor Rahm Emanuel spent much of the last five years touting growth every year on the NWEA — the all-important district-mandated reading and math tests that are factored into teacher evaluation, school ratings and whether students get into elite selective enrollment high schools.
But at the Chicago Board of Education meeting on Wednesday, there promises to be a heated discussion about a controversial report by the inspector general that calls out “unusual patterns” in NWEA test score results.
School district officials released the report last week and then attacked it, saying they are outraged that it raises the specter of cheating and gaming. District leaders insist the report “does not call into question the accomplishments of our students and school communities.” Still, they said they will implement recommendations made in the report, which include hiring a testing security expert and doing more auditing of test results.
Meanwhile, the inspector general has said little about it. He plans to present his findings at Wednesday’s meeting.
Here are three important questions raised by the inspector general’s report.
Does the inspector general’s report prove cheating or gaming?
The quick answer is no. But it does raise red flags that gaming and cheating could be going on unchecked.
The basis of the report is a data analysis of test score growth. It identifies dozens of schools where in some grades the test score growth from Spring 2017 to Spring 2018 was so great that there was one in a million or one in a billion chance it would occur.
The inspector general also analyzed two unique aspects of the NWEA: Students can take as much time as they want on the computer-based test and they can actually pause it, which can trigger a different question.
The inspector general found a “connection” between students with unusually large test gains and excessive test length and pauses. For example, 12 percent of the students whose scores skyrocketed spent six hours or more on the test, which has 53 questions.
But school district officials strongly dispute there is any connection. It put out a counter analysis looking at all test scores, not just high growth, and found no correlation between time taken on the exam or pauses and the results.
This counter analysis led board members to conclude the report was faulty and to discount it.
Chavez Principal Barton Dassinger said high growth in third grade math at his school was driven by a pair of “super star” teachers. He said he is confident nothing nefarious is going on. While investigators found substantial growth at Chavez, students did not take an unusually long time on the test, nor did they pause a lot.
“There is just absolutely incredible teaching going on,” he said.
In fact, the report clearly states that high growth could be happening for benign, even positive, reasons.
But there are some schools with high-growth results in more than one grade where students did spend a long time on the exams and often paused them. This includes a South Side elementary school with three grades of unusually high growth where students spent more than six hours on the test and paused an average of 10 times.
It also includes a West Side elementary school where Emanuel and CPS CEO Janice Jackson went to tout improved NWEA scores.
And investigators interviewed students and teachers at some of the schools and found some troubling patterns .
Some students said they were told to write down hard questions for later lessons and said sometimes teachers would read passages and questions out loud and even nod or shake their heads, indicating the correct answer.
According to the inspector general’s office, the NWEA testing company, or the Northwest Evaluation Association, warns that the validity of the test could be compromised if the test is paused with the intent of getting a new question. And indeed that’s what students told investigators they were allowed to do.
The OIG points out that even if there’s no misconduct, allowing students to spend an inordinate amount of time on the test is a waste of time. While some students took days to complete the test, those who finished in less time said they were left doing nothing. The OIG found that tens of thousands of CPS students are taking at least twice the national average duration to complete their exams.
What is the school district doing to prevent cheating?
School district officials say they have put in some safeguards to prevent cheating, including, in recent years, providing a manual on proper testing procedures and requiring principals and those administering the test to vouch that they will follow the rules.
They also say they have done some proactive school visits.
But principals tell WBEZ they are only required to fill out a generic form and there is no follow up.
One big problem: the manual does not spell out that the test should not be administered alone by the classroom teacher. The inspector general found many times the reading or math teacher — the teacher whose evaluation is partly based on the students’ results — was proctoring the test.
In addition, the inspector general found that neither the testing company, nor the school district, kept track of who was administering the tests to which students. Without this accounting, it is impossible to see whether a high-growth test result can be tied to a particular proctor.
It also found that internal examinations done by CPS that identified schools with unusually high growth did not always lead to an audit or any repercussions.
Though school district officials won’t admit they could have done more, they say they are accepting all the recommendations of the inspector general, including hiring a test security expert and promising to do more monitoring and audits of unusual results.
Is the test the problem?
Board member Elizabeth Todd-Breland bashed the report for raising the idea of cheating, but she said it correctly questions how the NWEA exams are being used.
Chicago Public Schools, like many school districts, began using the NWEA sometime in the late 2000s as a benchmark exam, meaning it was used to give teachers information about student deficits. Students take the NWEA three times a year and teachers reported liking that they could get immediate, useful feedback about student progress.
The NWEA was created for this purpose, or, as NWEA put it, the test was “designed to be an instructionally informative assessment that provides actionable data for teachers to develop effective learning pathways for their students.” Because it is supposed to help teachers find out what students know, it was untimed.
At the time, the school district used the state-mandated standardized test for school ratings and teacher evaluations. Then, the state switched to a new exam aligned with new difficult national standards called the Common Core. But the results were delayed by months and students struggled with the state test, known as PARCC.
It was at this point that the school district switched to using the NWEA to hold schools and students accountable, using it to evaluate teachers, decide whether students were promoted to the next grade and to rate schools.
Bob Schaeffer, interim executive director of the group FairTest, said he questions why the NWEA testing company is allowing school districts like Chicago to use its test in a high-stakes manner when it wasn’t designed for that purpose.
“Ethical testing companies would cancel the test, refuse to sell it to the district, if they know it is being used in an improper way,” Schaeffer said. “If a test has not been validated for a specific purpose it should not be used.”
Schaeffer notes that standardized testing is a big business.
Since 2014, Chicago Public Schools has paid NWEA more than $15 million and, this year, it is expected to pay out $2.2 million.
Over the past decade, NWEA, which is based in Portland, Oregan, has seen its budget triple from $50 million to $150 million. Its top executives make more than $350,000.
NWEA would not make an official available to comment, but it released a statement, noting it is a not-for-profit organization that has been committed to helping students learn for 40 years. It also said it would provide more guidelines for Chicago Public Schools around proper test duration.
Click here to read the full inspector general report.
Sarah Karp covers education for WBEZ. Follow her on Twitter at @WBEZeducation and @sskedreporter.