Updated March 1
At his final Chicago Board of Education meeting, the school district’s inspector general sparred with board members over a report his office produced about irregular patterns in elementary test score results.
Board members thanked Inspector General Nick Schuler for his service at the meeting last week, but then attacked his report. At the heart of the discussion was the question of whether his findings proved cheating or gaming and whether it cast any shadow of the academic progress made by Chicago Public School students in recent years.
School district leaders and former Mayor Rahm Emanuel have spent the past five years boasting about gains on the test at the center of Schuler’s report, called the NWEA. NWEA results factor into teacher evaluations, school ratings and admissions for the school district’s selective enrollment high schools.
But, as Chief Education Officer LaTanya McDade pointed out at the meeting, studies based on other exams are the basis of the biggest pride points for the district. Studies by Stanford University’s Sean Reardon and the University of Illinois at Chicago have shown that Chicago Public Schools students improve at a faster rate than almost any other school district. However, these studies are based mostly on a state standardized test that hasn’t been given for about five years. National test data was also analyzed up until 2015.
Here are the important questions raised by the report and how board members and the inspector general responded to them.
Does the inspector general’s report prove cheating or gaming?
The quick answer is no, and board members laced into the inspector general for even raising the possibility that cheating or gaming was involved. Board member Amy Rome called it irresponsible, given the absence of concrete evidence in the report.
But the inspector general and his team shot back that school district leadership would be “naive” if they thought these variables did not factor in.
Rosalind Rossi, the chief performance analyst with the inspector general’s office, said she and her team interviewed dozens of students. Many of them described questionable behavior, including proctors rewording questions, telling students to rethink answers and providing math formulas, she said.
“We saw many red flags here,” she said. “Rather than have the board bury its head in the sand, we thought it was best to proactively try to address any situations, either benign or not. That was our goal here: to try to improve the system, not to prove individual cases.”
Board members also blasted the conclusions reached based on the data analysis.
The basis of the report is an analysis of test score growth. It identifies dozens of schools where in some grades the test score growth from spring 2017 to spring 2018 was so great that there was one in a million or one in a billion chance it would occur.
The inspector general also analyzed two unique aspects of the NWEA: Students can take as much time as they want on the computer-based test and they can actually pause it, which can trigger a different question.
The inspector general found a “connection” between students with excessive test length and pauses and those who have extraordinary growth. The inspector general said experts suggested their analysis focus on students who took long and paused a lot and not all students.
But board members said that was the wrong way to look at it. They pointed to a school district analysis looking at all test scores. That analysis found no correlation between time taken on the exam or pauses and the results.
This counter analysis led board members to conclude the report was faulty and to discount it.
In an interview, Chavez Principal Barton Dassinger said high growth in third grade math at his school was driven by a pair of “superstar” teachers. He said he is confident nothing nefarious is going on. While investigators found substantial growth at Chavez, students did not take an unusually long time on the test, nor did they pause a lot.
“There is just absolutely incredible teaching going on,” he said.
In fact, the report clearly states that high growth could be happening for benign, even positive, reasons.
Board member Lucino Sotelo also pointed to one South Side elementary school, Dixon, which was identified as having several classes with growth classified as one-in-a-million. Earlier in the board meeting, a student from Dixon read her award-winning inspirational black history month speech.
Sotelo said he saw her performance as evidence that the growth was the result of learning.
“When I see the work she is putting in, when I see the effort she is putting in, [I see why she is able] to deliver that growth,” he said. “There is a reason why there is no correlation in this data.”
But Dixon is unusual, even among other schools identified in the report. It had three grades of unusually high growth where students spent more than six hours on the test, which experts say should take about an hour, and paused an average of 10 times.
According to the inspector general’s office, the NWEA testing company, or the Northwest Evaluation Association, warns that the validity of the test could be compromised if the test is paused with the intent of getting a new question. And indeed that’s what students told investigators they were allowed to do.
The inspector general’s office pointed out that even if there’s no misconduct, allowing students to spend an inordinate amount of time on the test is a waste of time. While some students took days to complete the test, those who finished in less time said they were left doing nothing. The inspector general’s office found that tens of thousands of CPS students are taking at least twice the national average duration to complete their NWEA exams.
What is the school district doing to prevent cheating?
School district officials say they have put in some safeguards to prevent cheating, including, in recent years, providing a manual on proper testing procedures and requiring principals and those administering the test to vouch that they will follow the rules.
They also say they have done some proactive school visits.
But principals tell WBEZ they are only required to fill out a generic form and there is no follow up.
One big problem: The manual does not spell out that the test should not be administered alone by the classroom teacher. The inspector general found many times the reading or math teacher — the teacher whose evaluation is partly based on the students’ results — was proctoring the test.
In addition, the inspector general found that neither the testing company, nor the school district, kept track of who was administering the tests to which students. Without this accounting, it is impossible to see whether a high-growth test result can be tied to a particular proctor.
It also found that internal examinations done by CPS that identified schools with unusually high growth did not always lead to an audit or any repercussions.
Though school district officials wouldn’t say if they could have done more, they told board members they want to accept most of the recommendations of the inspector general, including recording proctors, hiring a test security expert and promising to do more monitoring and audits of unusual results.
McDade said the school district will continue to allow classroom teachers, whose evaluations depend on these test scores, to administer the test. She said scheduling different proctors would be too hard. Also, she said younger students need the familiarity of their teacher when taking the test.
Also, one board member said he was not sure the report proved hiring a security expert was necessary.
Is the test the problem?
Board member Elizabeth Todd-Breland bashed the report for raising the idea of cheating, but she said it correctly questions how the NWEA exams are being used.
At the board meeting, she pressed school district officials about whether the test should be ditched.
McDade said the school district is in the midst of a process to reexamine how it rates schools. She said that process will also involve discussions about whether the NWEA is the right test.
Chicago Public Schools, like many school districts, began using the NWEA sometime in the late 2000s as a benchmark exam, meaning it was used to give teachers information about student deficits. Students take the NWEA three times a year and teachers reported liking that they could get immediate, useful feedback about student progress.
The NWEA was created for this purpose, or, as NWEA put it, the test was “designed to be an instructionally informative assessment that provides actionable data for teachers to develop effective learning pathways for their students.” Because it is supposed to help teachers find out what students know, it was untimed.
At the time, the school district used the state-mandated standardized test for school ratings and teacher evaluations. Then, the state switched to a new exam aligned with new difficult national standards called the Common Core. But the results were delayed by months and students struggled with the state test, known as PARCC.
It was at this point that the school district switched to using the NWEA to hold schools and students accountable, using it to evaluate teachers, decide whether students were promoted to the next grade and to rate schools.
Bob Schaeffer, interim executive director of the group FairTest, said he questions why the NWEA testing company is allowing school districts like Chicago to use its test in a high-stakes manner when it wasn’t designed for that purpose.
“Ethical testing companies would cancel the test, refuse to sell it to the district, if they know it is being used in an improper way,” Schaeffer said. “If a test has not been validated for a specific purpose it should not be used.”
Schaeffer notes that standardized testing is a big business.
Since 2014, Chicago Public Schools has paid NWEA more than $15 million and, this year, it is expected to pay out $2.2 million.
Over the past decade, NWEA, which is based in Portland, Oregon, has seen its budget triple from $50 million to $150 million. Its top executives make more than $350,000.
NWEA would not make an official available to comment, but it released a statement, noting it is a not-for-profit organization that has been committed to helping students learn for 40 years. It also said it would provide more guidelines for Chicago Public Schools around proper test duration.