The authors of this report created “efficiency scores” for Milwaukee schools and, on the basis of those scores, draw conclusions about the relative efficiency of Milwaukee’s charter and traditional public schools. A first set of scores was generated by dividing test scores in math and science by per-pupil funding. In a second analysis, “efficiency scores” were estimated for school types using a regression model that also accounted for selected student demographic characteristics. Several major problems arise from the authors’ approach: First, test scores do not comprehensively represent the purposes of schools. Second, threats to the validity of the author’s assumption that there is uniform financial accounting across schools and types are not addressed. Third, the analytic description of the study was incomplete, making interpretation difficult. Fourth, the report makes strong but unmeasured claims about the superior “efficiency” of charter schools based on their having greater autonomy. Finally, the authors did not appear to adjust for selection effects – effects that would prove fatal to their conclusions.  As a result of its manifold flaws, and because its conclusions are not supported by the evidence presented, the report is of little if any use to policymakers.

Update: Will Flanders and CJ Szafir, the report's authors, have posted a response to the review. The response can be found immediately below Cobb's original review of their report and at: http://www.will-law.org/will-blog-will-responds-criticism-colorado-group/.

Casey Cobb's rejoinder to the authors' response to his review is posted immediately below the authors' response.

Milwaukee’s voucher program, the Milwaukee Parental Choice Program (MPCP), was created by state legislation in 1990 and enables low-income residents of the Milwaukee Public School district (MPS) to enroll at taxpayer expense in private schools that have been certified by the state Department of Public Instruction.  The School Choice Demonstration Project at the University of Arkansas released in 2012 the final reports based on its five-year longitudinal growth study of this voucher program.  The following reviews are of three of these reports:  Report #29 compares voucher and MPS elementary and middle school test-scores; Report #30 compares graduation and post-secondary enrollment and persistence rates of MPS and MPCP students; and Report #32 compares test performance of MPCP and MPS students in grades 4, 8 and 10 in reading, math, and science.  The reviewed reports use largely sound methods, but the data they assemble provide little to support the 22-year-old school voucher program.  To some extent, this is because of specific methodological or analytical shortcomings. But it's also because the data and the reports simply fail to demonstrate that voucher schools are associated with improved outcomes. 

The report under review attempts to counter claims made by two Arizona newspapers that Arizona’s tuition tax credit voucher program has largely failed to increase private school access for low-income families. The report concludes that many scholarship recipients come from families whose incomes qualify them as below the state median and even below the poverty line. The analysis is generally transparent and thorough, and the results probably do approximate the distribution of the vouchers. However, the report overstates the newspaper claims. Moreover, certain design issues--particularly the possibility of sampling bias, a lack of uniformity in data collection, and self-reported data by the voucher-granting organizations--cause some uncertainty about the final estimates of family income. The Arizona law was recently amended to require official reporting, so more definite numbers will soon be available.

This review examines a research study that compares high school graduation rates of students who used vouchers to attend private high schools in Milwaukee and students who attended public high schools in that same city. The study reports that for the most recent year of data, a sample of voucher students had estimated graduation rates 12 percentage points higher on average than a sample of public school students. Overall, the trend favoring voucher students was observed in five of the previous six years. The analysis is technically accurate and makes defensible assumptions as necessary for the final calculations. However, although the results are descriptively useful, any real claims about whether the voucher program is actually causing higher graduation rates would depend on a much stronger research design.

Suggested Citation: Cobb, C. D. (2010). Review of “Graduation Rates for Choice and Public School Students in Milwaukee, 2003-2008.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-graduation-rates-choice

The study under review is the second-year evaluation report of the Milwaukee Parental Choice Program (MPCP), a publicly funded voucher program that allows low-income students in Milwaukee to attend secular and religious private schools in that city. Its primary finding is that there were no overall statistically significant differences in achievement growth in reading or math between MPCP and Milwaukee Public School (MPS) students over a one-year period. The study design and methods of analysis are sound overall. The relatively short duration of the treatment, however, raises questions about the usefulness of the findings.

Correction: On June 24, 2009, the authors of the report under review brought to our attention an error in the original May 28, 2009 review of their work by Casey Cobb. Professor Cobb has revised his review accordingly. In the initial review, Professor Cobb made an incorrect assumption about the sampling procedures used in the study. As a result, he drew the incorrect conclusion that the external validity of the findings was tempered by a non-random sample of MPCP students, and hence a non-representative sample of MPCP schools. As the study authors pointed out, the MPCP sample was in fact drawn randomly.

Suggested Citation: Cobb, C.D. (2009). Review of “The MPCP Longitudinal Educational Growth Study Second Year Report.” Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-MPCP-longitudinal