Lies, Damn Lies, and Public Education

Standford University just published a study which claims that charter schools are not doing as well as traditional public schools. What most people who repeat the claims of this report do not understand is that it is not a measure of actual performance of the respective schools. It only measures how much students of comparable backgrounds improved test scores over previous years.

The differences in improvement which are measured are tiny: between 1 and 3 percent of a standard deviation. And they admit it: "This small difference — less than 1 percent of a standard deviation — is significant statistically, but is meaningless from a practical standpoint. Differences of the magnitude described here could arise simply from the measurement error."

To get some better looking results, they break out the measurements into lower, middle, and high school levels, and find that charter school students in high schools improve 2-5% of a standard deviation less.

"we did look at performance at elementary, middle, high school and multi-level schools and found that actually elementary and middle schools were actually outperforming their traditional public school peers.
The way—the measure that we used is growth on their—each state’s test. And because testing is generally from third to eighth grade, so we can’t even really look at the schools who just have kindergarten through third grade, because we haven’t gotten a growth score on those yet."
(Kenneth Surratt, assistant director of the Center for Research on Education Outcomes at Stanford University and one of the lead authors of their report, in an interview on Democracy Now)

Wait, did he just say testing is only generally available through eight grade, and they are still able to assert they have good measurements for high schools somehow? This deserves a closer look. On page 7 of the technical appendix there is a table showing the measurements for math and reading broken down by grade level. 8 of 12 of the grades improved in math, and 7 of 12 of the grades improved in reading. If I take a simple average of their numbers, for all grades, for grades 3-8 (when the test scores are most available), or even for high school only, I get positive results! What isn't included in their table is the sample sizes for each grade, so a simple average is not exactly accurate, but I find it incredulous that their sample sizes in the negative scoring grades would be so much larger as to throw the actual results that far off the average.

I can get a negative result if I throw out the 1st and 12th grade scores, which show charter schools doing very much better compared to other grade. I'm not sure what happened with the 1st graders, but 12th grade is when students are taking college entrance exams and all, so while testing may be optional at the other high school grade levels, we should have a much larger sample size from 12th grade because they are taking SATs and ACTs. Yet the only way I can get results similar to those alleged by this report is to assume a very tiny sample size of 12th graders. The words "deeply flawed" come to my mind here.

Personally, I don't think charter schools are going to solve any big problems. They are still public schools, after all. But it is really depressing to see the education unions going to such lengths to try to stifle the one prospect we have for some healthy competition in public education. Charter schools give us an alternative to throwing more money at traditional public schools when they are failing. Not that its really better, but at least the administrators can see the students (and the per-student funding) leaving their school.