School closings and the ever-increasing number of deaths provide the backdrop for a proposal by the Center for American Progress (CAP) to deny waivers of the federally mandated administration of standardized tests in spring 2021. Further, the federal government proposes to add to those assessments in ways that CAP argues would make the test results more useful. In its recent report, CAP sides with the Department of Education’s policy of denying such requests for waivers, and it calls for additional assessments that “capture multiple aspects of student well-being, including social-emotional needs, engagement, and conditions for learning” as well as supplementary gathering of student information. The reviewers find the CAP proposal to be ill-timed, unrealistic, and inappropriate for dealing with the exigencies arising from the pandemic.

A report from the Wisconsin Institute for Law and Liberty claims a chain of benefits will purportedly follow if the current cap on voucher enrollment for the state of Wisconsin is increased from four percent to 20%. The resulting improvement in graduation rates will, the report asserts, result in the employment of more people at higher wages, leading to increased personal wealth and government income of $3.2 billion over 20 years.The obvious failing of this logic is how this projected “ripple effect” will occur. Causal links are weakly explained and lack support. The claims cannot be verified because the methods are not described and, in key areas, the numbers literally do not add up. Other prominent unaddressed issues include social stratification, inequitable selection effects, the cost of running two school systems, and the effects on learning. With a large number of relevant variables at play and the study’s apparent reliance on descriptive methods, interpretations of the data are subjective and untrustworthy, and ultimately, the report offers no assistance to policymakers or others.

The cost and productivity of schools is hotly debated across the nation. Left-leaning groups argue for equitable funding and equality of opportunity. Right-leaning organizations contend that costs are too high and money is unwisely used. The Wisconsin Institute for Law and Liberty (WILL) joins this debate with Money for Nothing, which claims that Wisconsin does not get a good return on its educational investment. The report is based on three analyses: (a) of the ratio of non-teachers to teachers, (b) of per-pupil spending, and (c) of teacher pay. The report suggests there are too many non-teachers, per-pupil spending is not linked to higher outcomes, and teacher pay makes no difference in test scores. But critical errors in study design fundamentally negate these conclusions. The report flounders in arguing causality from correlation and misinterpreting statistical significance as representing meaningful policy effects. While “statistically significant” in many cases, the results are minuscule. This leads to false or unsupported conclusions clouded by the omission of critical details that prevent replication or confirmation. Rife with undocumented policy claims, the results run contrary to the literature on spending, administrator effects, and teacher effects. Unfortunately, no literature review is provided. The report fails to address the efficacy of interventions such as class size and early high-quality childhood education. The off-point theoretical base, flawed assumptions and meager findings shows the report earned its title, “money for nothing,” which could leave unsuspecting policymakers in dire straits. 

This report offers political advice to school superintendents. The authors draw solely on a collection of anecdotes from superintendents of large urban districts to offer a narrative replete with maxims, aphorisms, and pithy advice. Superintending is, in their view, a continuous string of almost unavoidable conflicts, so the reader is led through internal groups (central office, school board, unions, teachers and principals) and external groups (foundations, businesses and state politicians) and advised on how to cultivate support and deal with opponents. There is no research design, literature review, or systematic data collection. The report relies heavily on the authors’ experiences, yet neither author appears to have worked in a K-12 district or served as a superintendent. Moreover, the advice that follows is often contrary to contemporary professional practices. The report improperly generalizes from more politicized large urban districts and does not consider how the favored approaches may limit vision, flexibility and effectiveness. In the end, the report perpetuates antiquated methods and does not advance our knowledge.

A report from EdChoice documents two staffing trends in public schools. After a temporary pause during the Great Recession, school staffing in the U.S. resumed an upward trajectory, and hiring tilted toward non-teaching personnel as compared to teachers. The report concludes that staffing growth outpaces enrollment growth and that there has been no corresponding increase in student performance. The report then recommends increasing teacher pay, at the expense of non-teaching staff, and school choice. While the data on staffing trends are obtained from NCES publications, the report’s discussion of inputs, outcomes, and policy approaches is poorly grounded, leading to unsupported conclusions and policy prescriptions. It does not examine why there has been a staffing surge or whether it reflects a valid use of personnel. It evaluates the effectiveness of school staffing changes using concurrent achievement and finance measures, and it fails to acknowledge that educational outcomes have steadily improved, even though any benefits of staffing increases will be lagged and only show up gradually over time. The report presents no logical relationship between staffing trends and the solutions proposed. As a result, the report is irrelevant and is devoid of any important policy implications.

Program variation has long been touted as a charter school advantage, so a recent AEI report rates the diversity of charter school programs in 17 major cities. Examining charter school websites, the report finds the schools evenly split between “Specialized” schools (e.g., no excuses, STEM, or Arts) and “General” schools. It finds small to moderate correlations between city demographics and certain types of charter schools but also finds that specialized schools tend to morph into homogenized general schools. The report has several weaknesses. It claims superior diversity for charter schools but doesn’t empirically compare charter offerings with those of traditional public schools, which typically include many diverse options. Similarly, the introduction claims charter schools are hamstrung by red tape, but this is not addressed in the report. Also, as acknowledged in the report, coding schools based on website descriptions is an error-prone enterprise, yet no check of the accuracy of the data is provided. As the correlations between charter schools and city demographics are based on only 17 cases (cities are the unit of analysis), this is too weak a base to support the report’s conclusions. Fundamentally, the report does not make the case for its major claims, and thus has only minimal utility.

The irony of a recent report from the National Alliance for Public Charter Schools is that it purports to “separate fact from fiction” about charter schools. Unfortunately, in addressing 21 “myths,” it embraces fiction whenever useful to push advocacy goals, thus perpetuating its own myths and fictions about charter schools. Since it relies overwhelmingly on other advocacy documents, it does not give a balanced or thorough examination of any of the 21 “myths.” But the exercise provides a useful opportunity for this review to walk through the various claims and succinctly address each. Among the areas addressed are the financial equality of charter schools, lower teacher qualifications, student selection demographics, academic outcomes, segregation, and innovation. While the NAPCS report itself may provide only sound-bite fodder for advocates, we hope that the two documents combined—report plus review—offer an overview of issues that does advance comprehensive understanding.

Please note: This review was replaced on March 3, 2015, due to a substantive change in the content of Endnote 24, which is described in the endnote itself.

This review addresses two different proposals for reforming teacher training, neither of which is grounded in research. Further, neither provides useful evidence that the proposed programs have been, or promise to be, effective. TNTP’s Fast Start initiative essentially replaces teacher preparation with a five-week pre-service program followed by a closely monitored internship. As proof that compressing teacher preparation into basic survival training is effective, the authors report three weak correlations between the performance of program participants and TNTP’s certification evaluation rubric. The report concludes with three self-evident aphorisms: practice improves teaching, teachers who master teaching skills do better, and inadequate performers should be weeded out. Unfortunately, the TNTP report fails to show its policy prescription is effective or superior to other approaches. New America Foundation’s Time to Improve proposes a federal regulatory approach, rating teacher preparation programs based primarily on the k-12 test scores of the pupils their graduates go on to teach. It doesn’t address why policymakers should favor extending the use of high-stakes student test scores to teacher preparation programs in light of the long string of uncontrolled intervening factors that invalidate this approach. In light of these weaknesses, neither proposal provides useful guidance for teacher preparation policy.

The Gates Foundation’s Measures of Effective Teaching (MET) project was a multi-year study of thousands of teachers in six school districts that concluded in January 2013. This review addresses two of the final MET research papers. One paper uses random assignment to test for bias in teachers’ value-added scores. The experimental protocol was compromised, however, when many students did not remain with the teachers to whom researchers had assigned them; other students and teachers did not participate at all. This prevents conclusive answers to the questions of interest. The second paper examines how best to combine value-added scores, classroom observations, and student surveys in teacher evaluations. The data do not support the MET project’s premise that all three primarily reflect a single general teaching factor, nor do the data support the project's conclusion that the three should be given roughly equal weight. Rather, each measure captures a distinct component of teaching. Evaluating teachers requires judgments about which components are the most important, judgments that are not much informed by the MET’s masses of data. While the MET project has brought unprecedented vigor to teacher evaluation research, its results do not settle disagreements about what makes an effective teacher and offer little guidance about how to design real-world teacher evaluation systems.

This report offers recommendations for building community support for federal school turnaround approaches, particularly in communities that oppose these approaches. Parents, the report concludes, want improvement but cherish their local schools and distrust the turnaround options mandated from above by higher levels of government. Thus, community members rise up in anger when their school faces closure, conversion to a charter school, breaking up, or forced replacement of staff. Arguing that this resistance is due in large part to parents not understanding how bad their schools are, the report proposes that by engaging the public constructively and using eight communication strategies, parents will react more positively towards imposed turnaround approaches. The report does not address the body of research that shows school turnarounds to be generally unsuccessful. Further, even though parents in the study raise concerns that their schools are under-resourced and face significant social problems, the report fails to address these issues. By diverting attention from the real problems correctly identified by the parents and by possibly disrupting ongoing reforms, this communication strategy holds little promise for actually improving education and could prove harmful.

Former Florida Governor Jeb Bush and the Foundation for Excellence in Education have embarked on a well-funded campaign to spread selected Florida education reforms to other states. These reforms include assigning letter grades to schools, high-stakes testing, promotion and graduation requirements, bonus pay, a wide variety of alternative teacher credentialing policies, and various types of school choice mechanisms. This policy potpourri was recently presented by Gov. Bush in Michigan, and the documents used allow for a concrete consideration and review. Regrettably, Bush’s Michigan speech relies on a selective misrepresentation of test score data. Further, he offers no evidence that the purported test score gains were caused by the recommended reforms. Other viable explanations, such as a major investment in class-size reduction and a statewide reading program, receive no or little attention. Moreover, the presentation ignores less favorable findings, while evidence showing limited or negative effects of the proposed strategies is omitted. Considering the overwhelming evidence that retention is ineffective (if not harmful), it is troubling to see Mr. Bush endorse such an approach. Finally, Florida’s real problems of inequitable and inadequate education remain unaddressed.

Suggested Citation: Mathis, W. (2011). Review of “Florida Formula for Student Achievement: Lessons for the Nation.”  Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/thinktank/review-florida-formula

On May 4, 2010, the Obama administration released College- and Career-Ready Students, one of six research summaries supporting its proposals for reauthorizing the Elementary and Secondary Education Act (ESEA). This research summary addresses three key areas: common core standards, rewarding progress and success, and turning around low-performing schools. The proposed goals are laudable and the summary of the problems in American education is generally accurate. But the research foundation it provides is superficial and inadequate. Some of the significant proposals, such as the accountability system and "rewarding progress and success" are not addressed; and for two major issues, national standards and school turn-arounds, the research cited does not support the document’s conclusions. Fewer than 15% of the report’s references rely on independent, peer-reviewed research. The document advances rhetorical ends and political goals rather than providing a sound research base for the proposed policies. Overall, the document is of little or no value for those who seek evidence of the soundness of the Obama administration’s proposed legislation.

Suggested Citation: Ravitch, D. & Mathis, W. (2010). Review of “College- and Career-Ready Students.” Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/publication/college-and-career

President Obama and Education Secretary Arne Duncan have called for national "common core" curriculum standards. Some have argued that national standards are essential for reform, as they provide coherence, rigor, logic and organization. Others have contended they will narrow the curriculum, seize control from local districts and states, and distort the purposes of education. The Cato Institute’s Neal McCluskey argues that national standards will have only limited, if any, effect. The report contends there is only a weak theoretical case in favor of national standards and that the structure of schooling might be the real problem. It concludes that market models are the best way to reform education. While providing a useful summary and critique of the research on national standards, the non-sequitur in the report (standards do not work; therefore the free market will) presents readers with a conclusion not supported by the report’s evidence. Thus, the fundamental policy conclusions are not sustained.

Suggested Citation: Mathis, W.J. (2010). Review of "Behind the Curtain: Assessing the Case for National Standards." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-Behind-the-Curtain

This self-described "research study" and following press release are intended to influence the debate over the direction of the reauthorization of NCLB, offering a defense of the current test-based accountability system against the inclusion of "multiple measures." The report presents a review of the research on portfolios in large-scale school accountability systems, concludes that portfolio assessment is severely flawed, and then characterizes portfolios as a proxy for all non-test-based measures of student performance. The report has several glaring weaknesses, however. The literature review cherry-picks two studies, both conducted 13 years ago and, on the basis of those studies, concludes that portfolios are not reliable and are too expensive for large scale accountability systems. Yet other large-scale studies of portfolios – some of which are discussed in one of the two studies that the report itself relies on – come to different conclusions but are not examined or even mentioned. An even bigger problem with this new report (which is repeated in the press release), however, is the author’s decision to present portfolios as somehow representative of all non-test-based measures of student performance – which they clearly are not. This results in a document that is of little value for research or policy development.

Note that an "Issue Brief" (press release), published by the Lexington Institute after the main report, is available at:
http://lexingtoninstitute.org/1169.shtml

Suggested Citation:

Mathis, W. (2007). Review of "Portfolios: A Backward Step in School Accountability."  Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-portfolios-a-backward-step-school-accountability

This report documents "Two Wisconsins" separated by wealth. It concludes the difference in test scores between rich and poor is not attributable to school funding. To close the achievement gap, the author recommends that school districts require students to take more rigorous courses.

Suggested Citation:

Mathis, W. (2006). Review of "The State of High School Education in Wisconsin: A Tale of Two Wisconsins."  Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-the-state-high-school-education-wisconsin-a-tale-two-wisconsins