In March 2019, the Florida Department of Education published a report titled Student Achievement in Florida’s Charter Schools. The report consists almost entirely of simple graphs comparing achievement levels, achievement gaps, and achievement gains on statewide tests among charter school students to those among traditional public school students. Beyond the odd exercise of counting the number of comparisons that appear favorable to charter schools, the report offers no discussion. The comparisons are not even explained. The fact that the report merely presents comparisons required by law without putting any policy “spin” on them might be considered a virtue. The danger is that the report might encourage erroneous conclusions. The simple comparisons reveal very little about the relative effectiveness of charter schools and still less about other policy questions. At the very least, the report should have clarified the purposes of its comparisons and cautioned the reader against drawing unwarranted conclusions. 

An annual report from EdChoice is designed to provide a yearly updated list and synthesis of empirical studies exploring the impacts of school vouchers across a set of outcomes. EdChoice presents itself as a clearinghouse of “evidence” that school vouchers “work” and that school choice is an effective and efficient reform. Along those lines, the report showcases the purported personal and community benefits that arise from voucher implementation, such as an increase in test scores, educational attainment, parental satisfaction, increased civic values, improvements in racial segregation, and fiscal benefits through governmental cost savings. However, the report is a limited collection of cherry-picked studies chosen from an “overwhelming” number, largely from sources that are not peer-reviewed and primarily authored by voucher advocates. For these reasons, the report is so misleading that it is not useful for decision-making or research purposes.

The Center on Reinventing Public Education recently released a series of three policy briefs on the financial impact of charter schools on nearby school districts in California. The briefs are intended to inform ongoing debates over charter school financing and expansion in the state of California. This review finds that the briefs fail to accurately or fully apply the relevant research and data. They are useful only in pointing to some important issues that policymakers should consider; their analyses of those issues are, however, generally superficial and misleading.

This review offers a critique of a teacher effectiveness experiment conducted by investigators from the Mathematica Policy Group and published by the Institute of Education Sciences. The Mathematica experiment was designed to provide evidence about the effectiveness of teachers who were themselves high-achieving students and trained by either Teach for America (TFA) or the Teaching Fellows programs. 

A Mackinac Center for Public Policy report, The Michigan Context and Performance Report Card: High Schools 2018, seeks to measure and publicize high school performance by ranking schools according to their test scores. Although this has been done previously in many contexts, this publication touts as its major contribution taking socioeconomic status into account in its school rankings. While the stated goal of the report is laudable, the reality falls far short due to several shortcomings, detailed in this review. Given these shortcomings, the rankings presented in this report should be given no weight in any discussions of policy or practice. In fact, this report does a disservice by introducing questionable information in an easily readable form that is not substantiated by any credible analysis.

Update: Ben DeGrow and Michael Van Beek posted a response to the review in a blog post at: https://www.mackinac.org/critique-of-cap-report-card-fires-blanks.

John T. Yun’s response to the blog is posted immediately below the review. There is a corresponding newsletter at:
https://nepc.colorado.edu/publication/newsletter-yun-response-050219.

A report from the Brookings Institution makes the case for the importance of 21st century skills as goals for education systems and provides international examples. It focuses specifically on the development of new assessment methods as a primary means to help countries integrate 21st century skills – such as critical thinking, problem solving, collaboration, and communication – into curricular reforms. The report is a review of research and a policy analysis rather than an empirical study. Its main contributions are (a) the identification of three challenges to implementing a 21st century skills agenda, and (b) the organization of key ideas and research to possibly address those challenges. Policymakers should find the report useful as an overview highlighting the importance of 21st century skills internationally, although the report’s omission of discipline-specific considerations makes it less helpful as a guide for local curricular and pedagogical reforms. 

This Urban Institute research report is aimed at assessing the impact of the Florida Tax Credit (FTC) scholarship program on college enrollment and graduation. Through matched comparisons, the study finds consistently positive effects of the program for both two-year and four-year colleges, with relatively stronger effects for four-year private college enrollment and for students who stayed in the program longer. This review acknowledges the study’s contributions, using expanded data tracking to inform school voucher policy debates, but the review also raises three critical questions about the validity of some methods, findings and conclusions. These concerns call into question the use and misuse of this study by voucher advocates; proper use of the study begins with an understanding of its limitations and of the need for confirmatory research and future exploration of potential mechanisms driving any increased college attainment.

A report from Bellwether Education Partners contends that more funding should be given for charter school facilities. Focusing on a series of case studies in Idaho, the report argues that charter schools are unfairly denied funding for the construction and renovation of their school buildings. The examples the report relies on, however, are not “apples-to-apples” comparisons, and this makes any statewide generalizations suspect. Further, the report’s calculation of “costs-per-seat” ignores the reality that different students have different needs. Consequently, public district schools, which enroll proportionally more English language learners and students with disabilities, will likely have greater facilities expenses per pupil than charter schools. The report bemoans the fact that charter school facilities are not part of local school districts’ bonds and tax levies, yet it does not acknowledge that charter facilities are often owned by private entities. Mandating that local taxpayers support charter facilities would, therefore, force them to pay for buildings they would not own. Given these limitations, the report provides little guidance for policymakers and other stakeholders at a time when Idaho is working to overhaul its school funding system.

For over a quarter-century, researchers and others have vigorously investigated and debated the impact of school vouchers and voucher-like programs (such as education savings accounts and tuition tax credits/deductions). The result is a developed and sophisticated research literature on different aspects of these programs. But a report from the Institute for Justice does not take advantage of this body of research, instead offering little more than a simplistic and one-sided treatment of the empirical record. Setting out 12 “myths” about vouchers, the report then proceeds to systematically dismiss each “myth” in turn by presenting only evidence—much of it highly questionable—on the advantages of vouchers. Based largely on previous reports from other advocacy groups that curated evidence in order to present vouchers in a most positive light, the report then repeats many of those claims, even when flaws in those reports have already been publicly explained. In doing so, the report makes claims that are not supported, and in fact sometimes contradicted, by evidence in the sources it cites. The report provides a textbook case of echo-chamber advocacy, drawing primarily on reports from other voucher advocates. Consequently, it offers nothing useful in furthering our understanding of school vouchers. 

A TNTP report aims to expose what it labels the “opportunity myth” in American education: that while schools purport to prepare students well, they don’t deliver. It paints a dramatic picture of American students being misled by false promises of opportunity, when they could make significant learning gains if they experienced grade-level content, strong instruction, deep engagement, and high expectations. The report contends that these negative experiences are primarily the result of educators’ daily decisions and are magnified for students of color and low-income students. Though the report presents an array of qualitative and quantitative data, some of its particular claims are not fully supported by evidence, and it is unclear how key constructs are measured. Importantly, in describing educators’ decisions, the report does not sufficiently account for larger systemic and societal impediments to opportunity that serve to establish and maintain many of the obstacles and problematic patterns the report observes. 

Public Agenda compiled public opinion surveys to show where the American public stands on education issues. Their report notes the importance of gauging current public understanding of education in the nation, particularly given the three major policy changes in the past 10 years: Race to the Top, the Common Core State Standards, and the Every Student Succeeds Act. While this publication makes a unique contribution in its gathering of many surveys into a single report and its inclusion of interviews with employers, the lack of clarity in its methods makes many of its conclusions questionable. The report is inconsistent in how it addresses disaggregated data and respondents from different demographic groups (e.g., race, class, political affiliation), resulting in an inability to generalize to the population or to any subgroup. Therefore, the conclusions are cursory and incomplete, requiring further study and research. 

A report from the National Council on Teacher Quality (NCTQ) highlights six teacher evaluation systems claimed to be “yielding substantial benefits.” This comes at the end of a decade when reformed teacher evaluation systems that link teacher performance to measures of student growth have been at the center of educational debate. Disagreements range from the theoretical (e.g., is teacher quality fundamentally related to inequalities in student outcomes?) to technical (e.g., which measures should be included and how should they be defined?) to practical (e.g., how should ratings be used for personnel decisions?). Overall, the research regarding teacher evaluation is mixed, at best. Most notably, a recent multi-year RAND report suggests that a $500 million investment in teacher evaluation that heavily weighted student growth measures, with considerable funding from the Gates Foundation, did not improve student outcomes and, in some cases, exacerbated unequal access to effective teachers for low-income students and students of color. The NCTQ report, while clearly promoting such teacher evaluation, does not seriously counter the groundswell of academic literature critiquing these systems. It does not address the relevant literature, present a compelling justification for its site selection or the inclusion criteria for evidence, or adequately consider disconfirming or contradictory evidence. These methodological flaws limit the validity of the report’s findings and conclusions, which ultimately diminishes the usefulness of the report for policy and practice. 

The libertarian Reason Foundation recently published a policy brief that offers an alternative ranking of states’ education systems. The brief is based on a working paper from the Department of Finance and Managerial Economics at the University of Texas at Dallas (UTD). These two reports begin with the presumption that high average test scores combined with lower school spending should be the basis for state rankings, which are reasonable premises, depending upon how the analyses are approached. But the reports then head off the rails. Offering a ‘corrected’ representation of student outcomes and a crude analysis asserting that spending has no relation to those outcomes, the reports declare states such as New Jersey and Vermont to be poor-performing, highly inefficient systems by comparison to states like Texas. The reports then estimate a regression model to confidently assert that the higher performing states are those with a) weaker teachers’ unions and b) more children in charter schools. However, the reports’ corrected outcome measures, weighting significantly unbalanced racial groups as equal and treating racial groups as equated across states without regard for economic status, are specious at best. Regressing multiple, highly related, interdependent measures against a specious outcome measure leads to even more suspect findings, and would only mislead policymakers. 

The cost and productivity of schools is hotly debated across the nation. Left-leaning groups argue for equitable funding and equality of opportunity. Right-leaning organizations contend that costs are too high and money is unwisely used. The Wisconsin Institute for Law and Liberty (WILL) joins this debate with Money for Nothing, which claims that Wisconsin does not get a good return on its educational investment. The report is based on three analyses: (a) of the ratio of non-teachers to teachers, (b) of per-pupil spending, and (c) of teacher pay. The report suggests there are too many non-teachers, per-pupil spending is not linked to higher outcomes, and teacher pay makes no difference in test scores. But critical errors in study design fundamentally negate these conclusions. The report flounders in arguing causality from correlation and misinterpreting statistical significance as representing meaningful policy effects. While “statistically significant” in many cases, the results are minuscule. This leads to false or unsupported conclusions clouded by the omission of critical details that prevent replication or confirmation. Rife with undocumented policy claims, the results run contrary to the literature on spending, administrator effects, and teacher effects. Unfortunately, no literature review is provided. The report fails to address the efficacy of interventions such as class size and early high-quality childhood education. The off-point theoretical base, flawed assumptions and meager findings shows the report earned its title, “money for nothing,” which could leave unsuspecting policymakers in dire straits. 

A paper by two researchers at the Massachusetts Institute of Technology examines the consequences that follow from an expansion in the number of charter school places available for enrollment. The study uses data from Massachusetts, where charter school growth has been carefully managed and where there was significant excess demand for charter school places. In 2011, the state increased the cap on charter school enrollments in districts with low test scores, resulting in a large increase in charter school enrollment in some of these districts. The paper analyzes three outcomes: (a) changes across charter and non-charter public schools in funding (how much resource was available per student), (b) resource allocation (how schools spent their funds), and (c) achievement (how well students performed on academic tests). 

Amid burgeoning participation in AP coursework, a report by the Education Trust uses a case study of two exemplary high schools to address the question of how schools might support access to and success in AP programs by low-income students and students of color. It contends that a variety of interventions help promote access and success, including teacher support and development, analysis of class composition, careful scheduling, and provision of during- and after-school academic support. The report’s qualitative approach is well-suited to describing ways that schools can address the complex and deeply rooted problem of inequitable access to academic opportunity within secondary education. Encouraging underrepresented students to enroll in AP courses and then helping them succeed requires schools to engage in multilevel, holistic interventions that are not easily analyzed or described quantitatively. The report suffers, however, from lack of rigor in its description of methods and analysis. The scant detail on participants and data collection methods and the lack of discussion of how data were analyzed and used in the report weaken links between claims and evidence. The report, which focuses on two schools that enroll primarily Latinx students, would have also benefited from case studies of schools enrolling Black and Native American students—the groups that are most underrepresented in national AP enrollment and success rates. 

Overall, while the report provides some inspiring examples, more detailed and rigorous description of methods and analysis would make a stronger case for the highlighted interventions. 

A policy paper from ExcelinEd and Texas Public Policy Foundation examines the charter-school authorizing process in Texas. After surveying past Texas authorizing policies, the study claims Texas had been a leader in creating high-performing charter schools and that a low barrier of entry into the market was a contributing factor to this success. The study then contends that a 2013 legislative policy change has made the authorization process too restrictive, thus reducing the number of applicants and stifling innovation. The paper concludes with recommendations to create an easier authorization process to increase the number of charters granted and thereby to foster innovation. While the report is billed as a case study, it does not employ case study methodology. Moreover, the report fails to review or cite relevant research; it instead relies on unsubstantiated claims, anecdotes, misleading statements, and even demonstrably false statements to advance a particular viewpoint. In short, this paper is an ideological advocacy paper masquerading as a case study. Policymakers would be well advised to skip this paper and look for a more evidence-based review of the Texas charter authorizing process.

In Teacher Unions and Students’ Long-Term Economic Prospects, Emma García and John Schmitt review The Long-Run Effects of Teacher Collective Bargaining, a paper by Michael Lovenheim and Alexander Willén presented at the 2018 meeting of the Allied Social Science Associations (ASSA) paper session on "New Evidence on the Effects of Teachers' Unions on Student Outcomes, Teacher Labor Markets, and the Allocation of School Resources."

This review was originally published on the Economic Policy Institute website at: https://www.epi.org/blog/teacher-unions-and-students-long-term-economic-prospects/