John Merrow Takes the Road Most Traveled, By Corporate Charter Advocates

John Merrow is better served when he sticks to putting the best news face possible on corporate education reform atrocities, all in the name of fair and balanced reporting. But he has a predictable habit of interjecting himself into educational debates that he feels he should know something about, even though his latest entry into the charter controversy shows once again that Mr. Merrow should take fewer naps and do some current reading in areas for which he obviously cares more about than he knows.

Merrow's latest commentary in Ed Week, "When Roads Diverge: Tracking the Charter Movement," shows Merrow doing a little backtracking, in fact, to a time in late summer 2009, before Caroline Hoxby's shoddy charter "research" for Mike Bloomberg had been thoroughly discredited as an expensive advocacy piece by an expensive advocate. Merrow, in this commentary, offers us a celebration of the "gold standard" that Hoxby might have brought to bear on the charter question, had she not been compelled to distort, manipulate, exaggerate, and extrapolate her findings to fit her commission. And the small issue of the Hoxby "study" never receiving peer review before becoming the gospel according to Eli Broad and the corporate press? No problem, for Merrow had lunch with Caroline, and she assured him the peer review would happen soon. Which it already has, but which Merrow never mentions for fear of damaging his own variety of George Babbitt boosterism that drives this pro-charter screed to its illogical conclusion.

If Merrow had written all this out of ignorance, I could look the other way. After all, it is common practice for education writers to go for a couple of months without reading anything about the subjects they report on. But it is in the way that Merrow abuses the PDK/Gallup findings that convinces me clearly he has a few things to teach even the most polished propagandist like Caroline Hoxby. From Merrow:

The general public clearly wants more charter schools—64 percent in the 2009 Gallup poll on education. And a 2009 survey conducted by Education Next reports that more than a third of public school teachers support charters, a number that jumps to nearly half when respondents are told of President Obama’s support.

First off, if I wanted to know what teachers think, I would not go to Education Next, which is sponsored by the union busters at the sludge tank, the Hoover Institute. Secondly, there are no teacher questions in the survey that is connected by Merrow's link. Thirdly and most importantly, Kappan/Gallup poll did not ask the public if they "want more charter schools." The question actually asks, "do you favor or oppose the idea of charter schools:"

TABLE 11. As you may know, charter schools operate under a charter or contract that frees them from many of the state regulations imposed on public schools and permits them to operate independently. Do you favor or oppose the idea of charter schools?

Quite a different question, wouldn't you say, Mr. Merrow?

Hoxby Study Gets Kicked After Smackdown

On November 14, we reported on an assessment of Caroline Hoxby's unpublished charter study paid for by Mike Bloomberg. The independent evaluation boiled down to these specific criticisms:

• In measuring the effects of charter schooling on students in grades 4 through 12, the study relies on statistical models that include test scores from the previous year, measured after the admission lotteries take place. Yet because of that timing, those scores could be affected by whether students attend a charter school. As a consequence, the statistical models "destroy the benefits of the randomization" that is a strength of the study's design. (The use of a different model makes the results for students in grades K-3 more credible, he notes.)

• The report's claims regarding the cumulative effects of attending a New York City charter school from kindergarten through eighth grade are based on an inappropriate extrapolation.

• It uses a weaker criterion for statistical significance than is conventionally used in social science research (0.05), referring to p-values of roughly 0.15 as "marginally statistically significant".

• The report describes the variation in charter school effects across schools in a way that may distort the true distribution of effects by omitting many ineffective charter schools from the distribution.

This month's District Administrator has another level of criticism offered by Jonathan Gyurko:

A report issued by the New York City Charter Schools Evaluation Project in September, and since been held up as clear evidence that charter schools are doing a better job than traditional schools, is now facing criticism that its claim of being an “apples to apples” study just isn’t true.

The report, How New York City’s Charter Schools Affect Achievement, examines the performance of students who applied for openings in New York’s charter schools, which are 94 percent filled through random lotteries. When researchers compared the academic performance of those who were “lotteried in” with those who were “lotteried out,” they discovered a higher rate of achievement in the charter group.

Critics claim the report does not take into account the “peer effect,” whereby a child learns not only from teachers but from fellow students. Writing in Edwize, a blog sponsored by New York’s United Federation of Teachers, Jonathan Gyurko says, “Charter schools benefit from the fact that 100 percent of their students hail from motivated families; as a result, a charter student is surrounded by peers who are there by choice—rather than by attendance zone.”

Alexander Hoffman, writing in GothamSchools, an online news source about the New York City public schools, says the report is flawed because, in contrast to a medical study, it has no placebo group. Both groups—students in charter schools and students in traditional schools—know what kind of education they are getting. “I know from my own experience teaching that students who get their choice of schools take a bit more ownership,” Hoffman says. “If they get their second choice, or last choice, or somehow do not get their choice, that’s a big hurdle for their teachers and parents to overcome.”

Led by well-known school choice advocate Caroline Hoxby of Stanford University, the researchers claim that since these two groups were essentially the same—both comprised of students who sought admission to charter schools—they were able to make a comparison in which where students were educated, charter school or traditional school, was the only variable.. . . .

Hoxby Charter Study Turns Hoaxby

I was at Substance News this morning enjoying the work of the Schmidt Team when I came across some important news I had missed. No doubt I would still be missing it if it were not for Substance News, for the research review by EPIC and EPRU announced below remains ignored by corporate editorial boards everywhere.

Jennifer Medina did have a brief, buried blog post in the NYTimes Saturday regional news, but you can be sure that the deflation of the gaseous Hoxby hoax will not be reported as news anywhere--it simply does not conform to the corporate education reform narrative, particularly since it is, yet, another loud smackdown to the anti-scientific approach to education policy being taken by Team Obama/Gates/Broad.

When the history is written on this sad episode of American educational policy, Arne Duncan will, no doubt, make Margaret Spellings seem, well, 99.44% pure.

The Press Release from EPIC at the University of Colorado:

Headline-Grabbing Charter School Study Doesn’t Hold Up To Scrutiny
Reviewer finds serious statistical flaws in research on NYC charter schools

Contact: Sean Reardon, (650) 736-8517 (office); (617) 251-4782 (cell); sean.reardon@stanford.edu
Kevin Welner, (303) 492-8370; kevin.welner@colorado.edu
Gary Miron, (269) 599-7965; gary.miron@wmich.edu

BOULDER, Colo. and TEMPE, Ariz. (November 12, 2009) -- A recent report on New York City charter schools found achievement results at the charters to be better than comparison traditional schools. But that report relies on a flawed statistical analysis, according to a new review.

The report is How New York City's Charter Schools Affect Achievement and was written by Caroline Hoxby, Sonali Murarka, and Jenny Kang. When it was released in late September, it was enthusiastically and uncritically embraced by charter advocates as well as media outlets. The Washington Post offered an editorial titled, "Charter Success. Poor children learn. Teachers unions are not pleased." The editorial's first paragraph reads:

"Opponents of charter schools are going to have to come up with a new excuse: They can't claim any longer that these non-traditional public schools don't succeed. A rigorous new study of charter schools in New York City demolishes the argument that charter schools outperform traditional public schools only because they get the 'best students.' This evidence should spur states to change policies that inhibit charter-school growth. It also should cause traditional schools to emulate practices that produce these remarkable results."

The editorial argues throughout that the study provides unquestionable evidence that charters result in improved student achievement. It ends, "Now the facts are in."

The New York Daily News was no less effusive: "It's official. From this day forward, those who battle New York's charter school movement stand conclusively on notice that they are fighting to block thousands of children from getting superior educations."

Because of the declared importance of the new report, we asked Professor Sean Reardon to carefully examine the report's strengths and weaknesses for the Think Tank Review Project and write a review that would help others use the study in a sensible way. Reardon, like the report's lead author Hoxby, is a professor at Stanford University. He is an expert on research methodology.

The Hoxby report estimates the effects on student achievement of attending a New York City charter school rather than a traditional public school. A key finding, repeated in press reports throughout the U.S., compares the cumulative effect of attending a New York City charter school for nine years (from kindergarten through eighth grade) to the magnitude of average test score differences between students in Harlem and the wealthy New York community of Scarsdale. The report estimates this cumulative effect at roughly 66% of the "Scarsdale-Harlem gap" in English and roughly 86% of the gap in math.

In his review, Reardon observes that the report "has the potential to add usefully to the growing body of evidence regarding the effectiveness of charter schools." New York charter schools' use of randomized lotteries to admit students to charter schools offers the possibility that the study of those schools can roughly approximate laboratory conditions.

But Reardon points out that the report's key findings are grounded in an unsound analysis -- an inappropriate set of statistical models -- and that the report's authors never provide crucial information that would allow readers to more thoroughly evaluate "its methods, results, or generalizability."

Reardon's review notes these shortcomings in the report:

• In measuring the effects of charter schooling on students in grades 4 through 12, the study relies on statistical models that include test scores from the previous year, measured after the admission lotteries take place. Yet because of that timing, those scores could be affected by whether students attend a charter school. As a consequence, the statistical models "destroy the benefits of the randomization" that is a strength of the study's design. (The use of a different model makes the results for students in grades K-3 more credible, he notes.)

• The report's claims regarding the cumulative effects of attending a New York City charter school from kindergarten through eighth grade are based on an inappropriate extrapolation.

• It uses a weaker criterion for statistical significance than is conventionally used in social science research (0.05), referring to p-values of roughly 0.15 as "marginally statistically significant".

• The report describes the variation in charter school effects across schools in a way that may distort the true distribution of effects by omitting many ineffective charter schools from the distribution.

Reardon explains that, as a result of the flaws in the report's statistical analysis, the report "likely overstates the effects of New York City charter schools on students' cumulative achievement, though it is not possible -- given the information missing from the report -- to precisely quantify the extent of overestimation." This, as well as the lack of detailed information in the report to assess the extent of that bias, make it impossible for readers to know whether the report's estimated charter school effects are in fact valid.

"Policymakers, educators, and parents should therefore not rely on these estimates until the bias issues have been fully investigated and the analysis has undergone rigorous peer review."

According to Professor Kevin Welner, director of the University of Colorado at Boulder's Education and the Public Interest Center (EPIC): "Readers of this review will understand that, while Hoxby's charter school study is a contribution, it has significant flaws and limitations. Unfortunately, the editorial reaction of otherwise-respectable media outlets trumpeted the New York City findings as the final and faultless word on charter school performance. In fact, the study used inappropriate methods that overstate the performance of the charter schools it studied."

Welner notes that the Think Tank Review Project also recently reviewed another charter school study, released in June by Stanford's CREDO policy center. That study encompassed 65-70% of the nation's charter schools. "Our review pointed out a number of limitations but also noted the relative strength and comprehensiveness of the data set, the solid analytic approaches of the CREDO researchers, and the important fact that the CREDO results were consistent with a large body of research showing charter schools overall to be performing no better than (and perhaps worse than) traditional public schools," Welner says. But he added that "the CREDO and Hoxby reports used different designs and covered different schools. They are not directly comparable, nor are we able to say which is 'better.' Neither report is definitive or without notable weaknesses."

Welner concludes, "the important thing to understand is that if, after an appropriate reanalysis of the data, we still find that New York City's charter schools are in fact bucking the national trend, the sensible next step is for researchers to explore the causes rather than to jump to broad conclusions that fly in the face of the overall research base. It would be irresponsible to use the NYC results -- even if they were valid and reliable -- to drive policy in places throughout the U.S. where charters are apparently underperforming their competition."

Find Sean Reardon's review on the web at:
http://epicpolicy.org/thinktank/review-How-New-York-City-Charter

Find the NYC report by Hoxby and her colleagues at:
http://www.nber.org/~schools/charterschoolseval/

CONTACT:
Sean F. Reardon
Associate Professor of Education and (by courtesy) Sociology
Stanford University
(650) 736-8517 (office); (617) 251-4782 (cell)
sean.reardon@stanford.edu

Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
(303) 492-8370
kevin.welner@colorado.edu

Gary Miron, Professor of Education
Western Michigan University
(269) 599-7965
gary.miron@wmich.edu

About the Think Tank Review Project

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the University of Colorado at Boulder Education and the Public Interest Center (EPIC) and the ASU Education Policy Research Unit (EPRU), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

EPIC and EPRU collaborate to produce policy briefs in addition to think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).

Hoxby's Fool's Gold Standard

It was a huge embarrassment to Gates, Broad, Bloomberg, and other corporate school charterizers when the first peer-reviewed national study of charter schools showed that charters outperform public schools in less than 20 percent of the cases. Since then the oligarchs have brought in the hired gun economist, Catherine Hoxby, who never saw a charter school she didn't like, to slime the study as best she can.

Bloomberg has even sweetened Hoxby's pot, asking her for a good-news study of New York's charter schools, a study whose methodology Hoxby immodestly refers to as reflecting the "the gold standard." Here, then, is a post from More Thoughtful that does a little rubbing to find out what is underneath Hoxby's shiny surface:

Did you hear about the big report that came out this week? You know, the one that "shows" that NYC charter schools are better than traditional non-charter public schools? It has gotten a ton of attention, probably because it uses "'the gold standard' method[ology]." The report is not subtle about this. It is right there in the very first sentence of the executive summary, "The distinctive feature of this study is that charter schools' effects on achievement are estimated by the best available, "gold standard" method: lotteries." It even uses the term "gold standard" four more times throughout the report.

Everyone wants to follow The Gold Standard -- or at least be able to say that they do. Of course! I mean, who wouldn't? But I do not think that we actually have a gold standard in education research. In fact, I am quite sure that we do not, and appropriating biomedical research's gold standard does not make it appropriate for us.

However, if we are going to borrow their standard, can we not at least get it right?

The biomedical standard uses double-blind experimental studies with random assignment. That means that some research participants get the experimental treatment and some get a placebo, and both are assigned randomly. It also means that neither the researchers nor the participants know who is getting which treatment. After all, expectations are important, and the mind can set us up for all kinds of things.

*****************

One of the latest ideas about The Gold Standard in educational research concerns charter schools.

We all want to know whether charter schools are better than traditional non-charter public schools. On one level, we certainly do want to know about individual schools. But on the policy level, we want to know about the average charter school, because we want to figure out if "charterness" helps a school be better. If it does, then we want more charter schools. If it does not, then we want fewer or none. And if we cannot be sure, we want to keep checking.

Let me say this quite clearly: Some charter schools are better than most non-charter public schools, and some are worse. And some non-charter public schools are better than most charters, and some are worse.

The Gold Standard crowd have a favorite method for comparing charter schools to non-charter public schools, one of which they are quite proud, but one that is so full of problems that I am shocked that they keep using it.

They rightly want to control for self-selection bias among charter school students. We know that children and families that apply to charter schools are different from those who do not, even if we do not know what all those differences are. This seems like the perfect time to do a randomized assignment, because that is the best method to make sure that these differences cancel out between groups. Luckily, we have some randomized assignments. Oversubscribed charter schools are virtually always supposed to accept students using a random lottery. This allows researchers to compare the outcomes of those were were randomly accepted, and those who were not.

Sounds good, right?

Well, it does sound good. But serious issues remain. Some are more obvious than others, and some are correctable by those interested in getting the correct answer, rather than the one that fits their pre-ordained conclusions.

Issue #1: No Placebo

Biomedical research does not just include randomization of treatment. It also is at least single blind. If some patients know that they are getting the new treatment, they might react differently. The mind is a powerful thing. They might be more diligent. Perhaps do their rehab exercises more often. Maybe pay more attention to diet. Who knows? And those who know that they are getting the new treatment might not lose hope.

If a student and/or his family do not get into the school of his/her/their choice, how might they react? I know from my own experience teaching that students who get their choice of schools take a bit more ownership. If they get their second choice, or last choice, or somehow do not get their choice, that's a big hurdle for their teachers and parents to overcome. If parents do not get their choice of schools for their students, are they going to be as supportive of their child's teachers? Of the assignments? Are they going to have the same kind of faith in their child's school? I think that the answer is really quite obvious.

The problem with these studies is that the students and families who "lose" these lotteries are no longer like the students and families who "win" these lotteries. There simply is no basis for thinking that their views of their schools are like those of the lottery "winners." In fact, one could quite simply argue that this method of analysis ensures that the "winning" charter school students are being compared to students who did not want to go to the schools they attend.

Obviously, that's not an even comparison.

Issue #2: Peer Effects

We all know that peer effects matter. Research, experience and common sense back this up. If you put a student in a class with a "better" group of peers, s/he will do better than s/he would have done in a class with a "worse" group of peers. The other kids all do their homework, or their parents were more likely to read to them, or they are somehow smarter, or harder working, or bring more cultural capital to school with them, or however else you think "better students" might be defined.

We also know that charter school students are not, as a group, like non-charter school students. That is how the Gold Standard crowd justifies their approach here. So: trying to control for applicant differences, but not controlling for ongoing peer effects? I don't know if that is just lazy or actually dishonest. The importance of peer effects is so well recognized that I tend to think it is the latter, especially because techniques to control for them are so well established.

Of course, there is another way to look at this. From a personal level, if you have a child, you don't care about controlling for peer effects. Actually, you want the effects to be left in so that you can take advantage of them. If charter schools have "better" students, that's a reason to send your own child to a charter school. However, if this analysis is done for policy purposes, to influence policy-makers, then peer effects do matter. If you are thinking about all students, not just the select few who can get into the "better" school, you need to control for peer effects.

Issue #3: Selection Bias on the School Level

The goal of this lottery-based study design is to avoid self-selection bias in the data. However, those who use it do not acknowledge the additional selection problems they create.

The most important problem is that not all charter schools are oversubscribed, so not all charter schools can be included in these studies. This wouldn't be a problem if we had good reason to believe that a random selection of charter schools were included, but that is obviously not the case. Clearly, the "better" charter schools are far, far, far more likely to be oversubscribed than the "worse" charter schools. This biases the sample rather severely towards better charter schools.

Unfortunately, the sample bias problem doesn't stop there.

A really strong traditional non-charter public school is not going to lose a lot of students to a simply above-average charter school. In order to be oversubscribed, a significant number of students and/or families have got to believe that the charter school option is superior to the non-charter public school option, which suggests a level of dissatisfaction with the local traditional public schools. This biases the sample towards inferior non-charter schools.

Issue #4: Generalizability

The hardest thing in educational research -- and perhaps research overall -- is to be able to generalize one's results to the broader population or wider world. And yet, that is usually the end goal of policy-oriented research.

These kinds of lottery-based studies only include the kinds of students and families that apply to charter schools in the first place. Even if the previous issues could be corrected, how can one know that other sorts of students and families would see the same benefits? The fact is that different populations might benefit less or more from going to a charter school. It is simply impossible to know from this kind of study. Of course, if you are only concerned about benefitting the kids of families who already opt for charter schools, then this is not a problem. But if you aim to help a broader population than that, you need a better methodology.

These generalizability concerns also apply to schools. Oversubscribed charter schools might well be better than average non-charter public schools, and I do not really question whether they are better than their local traditional alternatives. But on a policy level, we need to be concerned with charters more generally than that. If we raise or lift caps on charter schools, or approve new charter schools, we have to expect an average charter school to result, not an exceptional one. But these studies really tell us nothing about the majority of charter schools that are not oversubscribed. Nor do they tell us anything about the relative quality of non-charter public schools that lack charter school alternatives.

*****************

I understand the desire to find a Gold Standard for educational research. But simply grabbing that label because a methodology has some resemblance to biomedical research is not good enough, despite what Prof. Caroline Hoxby may claim. Moreover, the popular press really must do a better job of examining these claims critically, rather than cheerleading for them like this.

This, of course, means that researchers, journalists and the rest of us must be sure to take a more thoughtful stance than has become our habit.