Category Archives: VAM

Education, education reform, Teaching, Testing, VAM

When the Shoe Is on the Other Foot: Lessons for Teachers in Misguided Accountability

September 19, 2013 plthomasedd 2 Comments

If we imagined a pictorial representation of the evolution of education accountability, similar to the standard image we associate with human evolution—

—then we’d have to confront that the accountability era begun in the early 1980s focused first on students, requiring them to pass exit exams (regardless of their having taken and passed all of the required courses for graduation) in order to receive their diplomas.

Next, schools were the target of accountability with the advent and distribution of school report cards.

By the end of the first and beginning of the second decades of the twenty-first century, teachers have found their place at the accountability table, with some suggesting that teachers are now being fed their just desserts. Merit pay linked to student test scores and the more recent flurry of implementing value-added methods (VAM) of teacher evaluation and retention in many ways bring teachers into decades-long predicaments faced by students and schools: the misguided and unfair weight of standardized testing used in dysfunctional and invalid ways.

When I posted about how absurd teacher accountability has become, I expected most on my Twitter feed to recognize the situation in New York as unfair and a harsh warning of the mounting weight of failed accountability:

A Bronx performing arts school’s dance instructor will be judged on students’ English exam scores. Physical education teachers at a transfer school in Brooklyn are going to teach Olympic history lessons to prepare students for the history tests that will help determine their ratings. And teachers in Queens are putting the fate of their evaluations into a final exam that they don’t teach, but yields high pass rates.

The scenarios are not unusual — across [New York City] this year, thousands of teachers will be rated in large part based on test scores of subjects and students that they do not teach.

Rather, the scenarios are examples of how schools have tried to comply with a new teacher evaluation system that must factor student performance into final ratings. They also represent how the original purpose of the evaluations, to differentiate teachers’ effectiveness, has been squeezed by restrictive state laws, limited resources, and a tight timeline for implementation.

“It’s insane to me that 40 percent of my evaluation is going to be based on someone else’s work,” said Jason Zanitsch, a high school drama teacher who will share the same “student growth” score with colleagues in his school this year.

However, the first response I received raised a much different point:

E knox ‏@Tim_10_ber

@plthomasEdD if teachers don’t like this then way assign all the group work which is just as bad for the kids? hmmm….

My first response was to note that holding teachers accountable for the work of other teachers and the test scores of students they do not even teach is not truly analogous to having students do group work, and then be graded for that group work.

As @Tim_10_ber and I exchanged tweets, I came to recognize that I was arguing from my idealized position on how best to implement group work (group work must require collaboration—or it is simply students sitting close to each other doing individual work—and any grades assigned to group work must be articulated to reflect participation) and @Tim_10_ber was confronting a position with which I agree—that group work is often implemented and graded carelessly and thus unfairly to students.

It is from that recognition, then, that I want to make an argument about the only potential positive outcome related to the unjustifiable use of merit pay and VAM in teacher evaluation, pay, and retention: teachers need to learn how to teach better now that the shoe is on the other foot. Some ironic lessons teachers should learn from invalid teacher accountability include the following:

Testing and grades often do far more educational harm than good; the time has come to consider de-testing and de-grading our teaching. Teacher feedback, student self-assessment, student-created rubrics, and re-imagined assessment situations (such as group assessments) and formats are all better alternatives to tests and grades, if our goal is equitable and effective learning opportunities for students.
The central flaw with teacher accountability being linked to student test scores and the standards movement is that teachers have experienced declining autonomy in both their content and pedagogy as well as the high-stakes tests themselves. Accountability without autonomy is tyranny. This lesson translates into how often student learning is reduced to mere compliance. Students being held accountable also must have their autonomy honored; thus, students deserve far more choice in their learning than they have been traditionally allowed.
As noted by @Tim_10_ber, teachers must be far more vigilant about designing, assigning, and assessing group work, with a keen eye on autonomy, engagement, and causation/correlation (what are fair associations between each student and the outcomes of the group).

The accountability era has nearly destroyed public education. Little about accountability based on standards and high-stakes testing can be embraced or endorsed.

But oppressive and even capricious mandates tend to be leveled at the least among us first; once those policies trickle up to those in power—in other words, when the shoe is on the other foot—living with inequity, unfair accountability, and unworkable conditions can open our eyes to our own flaws as teachers.

As we continue to fight for our professional autonomy and dignity, taking moral stands of non-cooperation, let’s be sure to bring that fight to our classrooms and honor the autonomy and dignity of all our students as a model for those in power who have yet to see the flaws of their ways through the distorting lens of privilege they wear.

In the words of Henry David Thoreau in “Civil Disobedience”:

If I devote myself to other pursuits and contemplations, I must first see, at least, that I do not pursue them sitting upon another man’s shoulders. I must get off him first, that he may pursue his contemplations too….

If the injustice is part of the necessary friction of the machine of government, let it go, let it go; perchance it will wear smooth — certainly the machine will wear out. If the injustice has a spring, or a pulley, or a rope, or a crank, exclusively for itself, then perhaps you may consider whether the remedy will not be worse than the evil; but if it is of such a nature that it requires you to be the agent of injustice to another, then, I say, break the law. Let your life be a counter friction to stop the machine. What I have to do is to see, at any rate, that I do not lend myself to the wrong which I condemn [emphasis added].

CCSS, Charter schools, Education, education reform, Educational Research, Incentives, KIPP, Media, Merit Pay, miracle schools, NCTQ, Poverty, SAT, School choice, Segregation, Standards, Statistics, Teach for America, Teacher Evaluation, Testing, VAM

What We Know Now (and How It Doesn’t Matter)

August 19, 2013 plthomasedd 52 Comments

Randy Olson’s Flock of Dodos (2006) explores the evolution and Intelligent Design (ID) debate that represents the newest attack on teaching evolution in U.S. public schools. The documentary is engaging, enlightening, and nearly too fair considering Olson admits upfront that he stands with scientists who support evolution as credible science and reject ID as something outside the realm of science.

Olson’s film, however, offers a powerful message that rises above the evolution debate. Particularly in the scenes depicting scientists discussing (during a poker game) why evolution remains a target of political and public interests, the documentary shows that evidence-based expertise often fails against clear and compelling messages (such as “teach the controversy”)—even when those clear and compelling messages are inaccurate.

In other words, ID advocacy has often won in the courts of political and public opinion despite having no credibility within the discipline it claims to inform—evolutionary biology.

With that sobering reality in mind, please identify what XYZ represents in the following statement about “What We Know Now”:

Is there a bottom line to all of this? If there is one, it would appear to be this: Despite media coverage, which has been exceedingly selective and misrepresentative, and despite the anecdotal meanderings of politicians, community members, educators, board members, parents, and students, XYZ have not been effective in achieving the outcomes they were assumed to aid….

This analysis is addressing school uniform policies, conducted by sociologist David L. Brunsma who examined evidence on school uniform effectiveness (did school uniform policies achieve stated goals of those policies) “from a variety of data gathered during eight years of rigorous research into this issue.”

This comprehensive analysis of research from Brunsma replicates the message in Flock of Dodos—political, public, and media messaging continues to trump evidence in the education reform debate. Making that reality more troubling is that a central element of No Child Left Behind was a call to usher in an era of scientifically based education research. As Sasha Zucker notes in a 2004 policy report for Pearson, “A significant aspect of the No Child Left Behind Act of 2001 (NCLB) is the use of the phrase ‘scientifically based research’ well over 100 times throughout the text of the law.”

Brunsma’s conclusion about school uniform policies, I regret to note, is not an outlier in education reform but a typical representation of education reform policy. Let’s consider what we know now about the major education reform agendas currently impacting out schools:

What we know now about grade retention: Grade retention is growing in popularity across the U.S., represented by accountability policies in Florida. But grade retention has been shown by four decades of research not to achieve the goals advocates claim, and to cause harm.
What we know now about charter schools: Despite the increased support and funding for charter schools, “charterness” has not been shown to be a determining factor in school quality (when compared to traditional public schools [TPS]), charter schools have produced a range of outcomes essentially indistinguishable from TPS, but charter schools have increased segregation (by class and race) as well as underserved English language learners and special needs students (see annotated research here).
What we know now about school choice (and competition): Decades of a variety of commitments to school choice (notably vouchers) have resulted in a growing body of evidence that school choice fails to achieve the goals of its proponents (see a critical analysis of choice here). Choice, however, has been associated, like charter schools, with shuffling populations of students and increasing segregation. More broadly, the research on competition shows that it causes harm, and not the positive outcomes choice advocates claim.
What we know now about value added methods (VAM) of teacher evaluation: Although it is fair to say that the jury is still out on VAM, even advocates for exploring the potential for VAM have expressed caution about using it in high-stakes policies (see cautious considerations of VAM validity and reliability). Broadly, high-stakes implementation of VAM is certainly premature, and likely a significant waste of time and money better spent on problems more pressing and clearly defined.
What we know now about teacher quality: Teacher quality matters, but teacher quality is dwarfed by factors outside of school and outside the control of schools. The real teacher quality problem in schools is teacher assignment since impoverished students, African American students, Latina/o students, English language learners, and special needs students are disproportionately assigned to inexperienced and un-/under-certified teachers.
What we know now about merit pay: Simply stated, merit pay doesn’t work (and it may often have powerful negative consequences). It doesn’t work in education (see Pink also), but the business world has recognized that as well.
What we know now about Teach for America (TFA): The growing research base on TFA reveals a mixed picture, but it also shows that TFA advocacy is misleading (see also “three biggest lies”). Further TFA contributes negatively to some central problems public education faces: teacher attrition/turnover and inequitable teacher assignments (high-poverty and minority students being assigned disproportionately new and uncertified teachers).
What we know now about the SAT: The SAT remains a weaker predictor of freshman college success than GPA and also does not contribute positively to the public perception of school quality. SAT-prep classes also create a drain on school time and resources that could be better used addressing other needs. As well, the SAT remains race, class, and gender biased*.
What we know now about accountability, standards, and high-stakes testing: After thirty years of accountability, standards, and high-stakes testing in 50 separate state experiments, the research base is clear: “the absence or presence of rigorous or national standards says nothing about equity, educational quality, or the provision of adequate educational services, there is no reason to expect CCSS or any other standards initiative to be an effective educational reform by itself” (Mathis, 2012).
What we know now about miracle schools: Virtually every school designated “miracle” by advocates or the media (Texas miracle, Chicago miracle, Harlem miracle, Florida miracle, etc.) has been debunked by close analysis. Claims that some high-poverty schools excel (and thus all should excel) has also been exposed as misleading: “only 1.1 percent of high-poverty schools were identified as ‘high flyers.'”
What we know now about education as a social change agent: Simply stated: “you are better off being born rich regardless of whether you go to college than being born poor and getting a college degree” (Bruenig, 2013, based on data from “Pursuing the American Dream,” Pew Charitable Trusts).

Well into the second decade of the twenty-first century, then, education reform continues a failed tradition of honoring messaging over evidence. Neither the claims made about educational failures, nor the solutions for education reform policy today are supported by large bodies of compelling research.

As the fate of NCLB continues to be debated, the evidence shows not only that NCLB has failed its stated goals, but also that politicians, the media, and the public have failed to embrace the one element of the legislation that held the most promise—scientifically based research—suggesting that dodos may in fact not be extinct.

* Santelices, M. V., & Wilson, M. (2010, Spring). Unfair treatment? The case of Freedle, the SAT, and the standardization approach to differential item functioning. Harvard Educational Review, 80(1), 106-133.; Spelke, E. S. (2005, December). Sex differences in intrinsic aptitude for mathematics and science? American Psychologist, 60(9), 950-958; See page 4 for 2012 SAT data: http://media.collegeboard.com/digitalServices/pdf/research/TotalGroup-2012.pdf

CCSS, education reform, VAM

A Call for Non-Cooperation: So that Teachers Are Not Foreigners in Their Own Profession

May 7, 2013 plthomasedd 21 Comments

Gandhi’s views on enhancing the vernaculars…so that Indians are “not foreigners in their own land” are directly tied to his opinions on developing communities (for “the poorest of the poor” ) and making community service an integral part of any education. (Ramanathan, 2006, pp. 235-236)

Standing in the middle of the road offers some statistical advantage to avoiding being run over since you aren’t in the prescribed lanes of traffic, but standing in the middle of the road can never assure the safety that refusing to walk into the road to begin with does.

Writing about a call for a moratorium on implementing and testing Common Core State Standards (CCSS) from union leadership, Anthony Cody ends his blog post with three questions:

What do you think? Should we join Randi Weingarten in pushing for one year’s delay in the harsh consequences attached to Common Core assessments? Will this year put the project on sound footing?

These questions about CCSS have been joined by two other calls for compromise and civility—Matthew Di Carlo challenging charges that value-added methods (VAM) of teacher evaluation are “junk science” and Jennifer Jennings penning an apology to Secretary of Education Arne Duncan for protests at his 2013 talk at American Educational Research Association (AERA). [1]

Weingarten, Di Carlo [2], and Jennings share a call for standing in the middle of the road, a quest for ways to compromise, and these all appear reasonable positions. Ultimately, however, moratoriums, compromise, and civility are all concessions to the current education reform movement and the policies at the center of those reforms, specifically CCSS and VAM.

Teachers as Foreigners in Their Own Profession

Briefly, I want to identify how arguments about a CCSS moratorium, implementing VAM properly and cautiously, and the need for civility are concessions that render teachers foreigners in their own profession.

As long as the debate about CCSS and VAM remain how best to implement them, the essential questions remain unasked, and the agenda behind both are assured success. While I want to address the civility argument next, let me note here that calls for CCSS and VAM are inherently civil and derogatory, exposing the myopic concern for the civility of those rejecting Duncan’s discourse and policies.

The implied and stated messages of calls for CCSS and more high-stakes testing include the following: (1) Teachers do not know what to teach, or how, and (2) teachers are unlikely to perform at the needed levels of effort in their profession unless they are held accountable by external and bureaucratic means.

The implied and stated messages of calls for VAM and merit pay include the following: (1) The most urgent problem at the core of educational outcomes is teacher quality, and (2) teachers are unlikely to perform at the needed levels of effort in their profession unless they are held accountable by external and bureaucratic means.

Calls for CCSS and VAM also share another implied and stated message: Failed educational outcomes are the result of in-school deficiencies; in effect, out-of-school factors are irrelevant in the pursuit of education reform.

These messages are factually false and, despite the civility of the language, irrevocably offensive.

Standing in the middle of the road of bureaucratic, accountability-based school reform, then, may decrease the likelihood of being run over, but it concedes the road itself to those who have built it, to those who govern the laws of transportation.

To answer Cody’s second and third questions, then, No. And now to his first.

Civility: Standing in the Middle of the Road of Accountability

The call for civility exposes a foundational problem with the current education reform debate because, for all practical purposes, there is no debate.

Civility, CCSS, and VAM may all have some appeal in theory, but all of them fall apart in reality, in their implementation.

Civility is the last recourse of the powerful, those who can afford to appear civil because they hold all the power.

Through the lens of history, we must recognize that CCSS will become “what is testing is what is taught,” as all standards movements have shown.

VAM also sits in a long history of the corrosive consequences of stack ranking, merit pay, and competition.

And this brings us back to standing in the middle of someone else’s road.

Education reform and policy have been historically and are currently under the control of political and corporate leadership who are not educators—many of whom did not even attend public schools, many of whom send their own children to schools unlike the environments they promote and implement.

The locus of power in education is catastrophically inverted; thus, we do not need more or different mechanisms for accountability-based education reform, but we do need a new era of non-cooperation.

The goal of non-cooperation must include seeking ways in which to shift the priorities of the locus of power:

First, the central locus of power in education is the student, situated in her/his home and community.
Next in importance is the locus of power afforded the teacher in her/his unique classrooms.
These must then merge for a locus of power generated within the community of the school.
Finally, the locus of power in this school-based community must radiate outward.

A Call for Non-Cooperation

Non-cooperation, as found in the philosophy and actions of Gandhi, represents another inversion—away from in-school only education reform and toward, as Ramanathan explains, “communal and educational change”:

As is evident, the take on “education” presented here is not the usual one—of teaching and learning in formal contexts of classrooms and institutions—but one that is intended to move us toward becoming collectively open to realizing that very valuable “education” often goes on outside the constraints of classrooms: in ashrams, in madrassas, in extracurricular programs, by local, politically minded youth, all drawing on local vernacular ways of healing rifts. Indeed, “education” in both these institutions is civic and community education that seems to assume Gandhian ideals of “Non-Cooperation” (and nonformal education) and that is aimed at primarily effecting changes in the community, sometimes before addressing issues relevant to formal education. (p. 230)

Non-cooperation, then, moves beyond a call for teacher autonomy; instead, non-cooperation is the act of the autonomy by “people directly involved” (Ramanathan, p. 231):

Not only do they have Gandhi’s larger philosophy of Non-Cooperation against political hegemonies [emphasis added] at their core…, but they also opened up for me a way of understanding both how Gandhianism is situated and how particular dimensions of the identities of participants (Kanno, 2003; Menard-Warwick, 2005; Norton, 2000; Pavlenko & Blackledge, 2004) get laminated. I was able to see how Gandhianism is first collaboratively interpreted in workshops, then applied and translated on the ground in most local of contexts, and then recast and reinterpreted by individuals and groups as they regroup. (Ramanathan, p. 232)

Non-cooperation is a new paradigm that begins with those most directly impacted by the institution (here, education)—parents, students, teachers. In other words, the people most directly impacted ask the foundational questions: Do we need formal education? And if so, what does that include and how should that be implemented?

This is not about seeking compromise at someone else’s table, not about standing still in the middle of someone else’s road.

The purposes of universal public education, then, is refocused in the ways that address the needs of the least among us, as Gandhi envisioned:

[Nonformal education] … will check the progressive decay of our villages and lay the foundation for a juster social order in which there is no unnatural division between the “haves” and the “have nots” and everybody is assured a living wage and the rights to freedom.…It will provide a healthy and a moral basis of relationship between the city and village and will go a long way towards eradicating some of the worst evils of the present social insecurity and poisoned relationship between the classes. (Harijan, 9-10-37, cited in Prasad, 1924…). (qtd. in Ramanathan, p. 236)

Bureaucratic accountability-based reform is ill equipped to address inequity, mismatched with goals of social justice since the paradigm is authoritarian, the locus of power exclusively with the “haves.”

Non-cooperation seeks instead, as Ramanathan explains:

[an orientation] toward viewing education in broader, community-oriented terms to draw out “the best in children,” to build a “healthy and moral” base for both “the city and the village,” to be entirely secular in its orientation (with “no room … for sectional religious training,” and to eventually transform the “homes of the pupils”[)]. (p. 237)

As well, this call for non-cooperation reframes the civility debate, as Gandhi recognized: “We must welcome them to our political platforms [emphasis added] as honoured guests. We must meet them on neutral platforms as comrades” (qtd. in Ramanathan, p. 237). Civility then follows the re-imagining of the locus of power: “Non-Cooperation…emerges as a deeply historicized awareness committed to doing the opposite of repressive, silencing ills. The quiet way in which both projects bridge perceived gulfs are reminiscent of Gandhi’s insistence on responding to tyranny by searching for nonviolent, quiet alternatives that tap the moral instincts of humans” (Ramanathan, p. 238).

Currently, since calls for CCSS, VAM, and civility all work as “repressive,” “silencing,” and “tyranny,” non-cooperation is the only alternative remaining.

The results must be “interpreting all education as ‘civic education’ and on attending to the most basic of human needs—food, clothing, shelter—before addressing any issues related to formal learning” (Ramanathan, pp. 241-242) as direct action refusing to compromise on in-school only education reform that drives arguments for how best to implement CCSS and VAM:

This close attention to “educating oneself,” of figuring out and questioning one’s own default assumptions, has echoes of Gandhi’s Non-Cooperation, and finds interesting articulation in the idea that we each need to “not cooperate” with our default views but attempt to step outside them by “educating ourselves” by learning from others. (Ramanathan, pp. 244)

In the West, specifically in the United States, we are deeply entrenched in our “default views,” most of which are tinted by commitments to competition, authoritarian structures, and the sanctity of the individual. This call, however, is a call to recognize the importance of community and social justice in our national pursuit of democracy.

Arundhati Roy confronts the tensions at the core of why compromise, moratoriums, and civility fail the narrow education debate as well as the broader democracy:

Fascism is about the slow, steady infiltration of all the instruments of state power. It’s about the slow erosion of civil liberties, about unspectacular, day-to-day injustices.…It means keeping an eagle eye on public institutions and demanding accountability. It means putting your ear to the ground and listening to the whispering of the truly powerless. It means giving a forum to the myriad voices from the hundreds of resistance movements across the country that are speaking about real issues….It means fighting displacement and dispossession and the relentless, every violence of abject poverty. (Roy, 2002; qtd. Ramanathan, pp. 246)

Now is the time for non-cooperation, not moratoriums, not compromise, and not civility on other people’s terms.

Now is the time for non-cooperation so that teachers are not foreigners in their own profession and students are not foreigners in their own classrooms.

[1] See also Jeff Bryant.

[2] Of the three calls for moderation, I do not place Di Carlo’s position as essentially equal to those by Weingarten and Jennings. Di Carlo’s nuanced and detailed discussion of VAM contributes a credible position that I find compelling to a point (such as Di Carlo conceding: “Now, I personally am not opposed to using these estimates in evaluations and other personnel policies”); however, Weingarten and Jennings present far more problems and suffer from a much greater degree of lacking credibility.

CCSS, Education, education reform, Poverty, Standards, Teacher Evaluation, Testing, The State, VAM

Conservative Leadership Poor Stewardship of Public Funds

March 8, 2013 plthomasedd 2 Comments

In South Carolina and across the U.S., conservative leadership of education reform has failed to fulfill a foundational commitment to traditional values, good stewardship of public funds. [1]

The evidence of that failed stewardship is best exposed in commitments to three education reform policies: Adopting and implementing Common Core State Standards (CCSS), designing and implementing new tests based on CCSS, and proposing and field-testing revised teacher evaluations based on value-added models (VAM).

SC committed a tremendous amount of time and public funding to the accountability movement thirty years ago as one of the first states to implement state standards and high-stakes testing. After three decades of accountability, SC, like every other state in the union, has declared education still lacking and thus once again proposes a new round of education reform primarily focusing on, yet again, accountability, standards, and high-stakes testing.

Several aspects of committing to CCSS, new high-stakes tests, and teacher evaluation reform that are almost absent from the political and public debate are needs and cost/benefit analyses of these policies.

More of the Same Failed Policies?

If thirty years of accountability has failed, why is more of the same the next course of reform? If thirty years of accountability has failed, shouldn’t SC and other states first clearly establish what the problems and goals of education are before committing to any policies aimed at solving those problems or meeting those goals?

Neither of these questions have been adequately addressed, yet conservative political leadership is racing to commit a tremendous amount of public funding and public workers’ time to CCSS, an increase in high-stakes testing never experienced by any school system, and teacher evaluations proposals based on discredited test-based metrics.

Just as private corporations have reaped the rewards of tax dollars in SC during the multiple revisions of our accountability system, moving through at least three versions of tests and a maze of reformed state standards, the only guaranteed outcomes of commitments to CCSS, new tests, and reformed teacher evaluations are profits for textbook companies, test designers, and private consultants—all of whom have already begun cashing in on branding materials with CCSS and the yet-to-be designed high-stakes tests that will eventually be implemented twice a year in every class taught in the state.

SC as a state and as an education system is burdened by one undeniable major problem, inequity of opportunities in society and in schools spurred by poverty.

Numerous studies in recent years have shown that schools across the U.S. tend to reflect and perpetuate inequity; thus, children born into impoverished homes and communities are disproportionately attending schools struggling against and mirroring the consequences of poverty.

Commitments in SC to CCSS, new high-stakes tests, and reforming teacher evaluations based in large part on those new tests are at their core poor stewardship of public funding in a state that has many more pressing issues needing the support of state government.

A further problem with conservative leadership endorsing these education reforms is that much of the motivation for CCSS, new test, and reforming teacher evaluations comes from funding mandates by the federal government.

Misguided education reform is not only a blow to conservative economics but also a snub to traditional trust in local government over federal control.

Recently, as well, a special issue on VAM from Education Policy Analysis Archives (EPAA) includes two analyses that should give policy makers in SC and all states key financial reasons to pause if not halt commitments to education reform based on student test scores—the potential for legal action from a variety of stakeholders in education.

Baker, Oluwole, and Green explain: “Overly prescriptive, rigid teacher evaluation mandates, in our view, are likely to open the floodgates to new litigation over teacher due process rights. This is likely despite the fact that much of the policy impetus behind these new evaluation systems is the reduction of legal hassles involved in terminating ineffective teachers.”

Further, Pullin warns: “For public policymakers, there are strong reasons to suggest that high-stakes implementation of VAM is, at best, premature and, as a result, the potential for successful legal challenge to its use is high. The use of VAM as a policy tool for meaningful education improvement has considerable limitations, whether or not some judges might consider it legally defensible.”

Do schools across SC need education reform? Yes, just as social policy in the state needs to address poverty as a key mechanism for supporting those schools once they are reformed.

But in a state driven by traditional values and conservative political leadership, current commitments to CCSS, new high-stakes tests, and reforming teacher evaluations are neither educationally sound nor conservative.

[1] Expanded version of Op-Ed published in The State (Columbia, SC), March 8, 2013: “Conservatives poor stewards of education funds”

Daily Kos, Teacher Evaluation, VAM

Daily Kos: Misreading Teacher Evaluation and Retention

January 29, 2013 plthomasedd Leave a comment

Daily Kos: Misreading Teacher Evaluation and Retention

The League of Women Voters of South Carolina has released “How to Evaluate and Retain Effective Teachers” (2011-2013), but this report misreads the evidence on teacher evaluation and thus distracts high-poverty states from needed educational reform. [1]

A review of the report shows it does not establishing a clear problem with teacher quality in SC and misrepresents the current body of research on teacher evaluation, particularly value added methods (VAM) of evaluation.

As a high-poverty and racially diverse state, SC is similar to many other states facing educational hurdles, but those hurdles have less to do with identifying and ranking teacher quality and more to do with the inequitable distribution of teachers. Children of color, children in poverty, English language learners, and special needs students are taught disproportionately by inexperienced and un-/under-certified teachers. SC and other high-poverty states would do well to address teacher assignment and teaching conditions before experimenting with new teacher evaluation systems.

Ultimately, this report misreads and misrepresents the current understanding of how to evaluate and determine teacher quality—specifically through test-based methods.

continue reading at Daily Kos

Educational Research, Statistics, Teacher Evaluation, Testing, VAM

NFL again a Harbinger for Failed Education Reform?

January 28, 2013 plthomasedd 3 Comments

During the impending NFL strike in 2011—the act of a union—I drew a comparison between how the public in the U.S. responds to unionization in different contexts:

“I am speaking about the possible NFL strike that hangs over this coming Super Bowl weekend: a struggle between billionaires and millionaires, which, indirectly, shines an important light on the rise of teacher and teacher union-bashing in the US. Adam Bessie, in Truthout, identifies how the myth of the bad teacher has evolved.”

Once again, the NFL is facing a situation that I believe and even hope is another harbinger of how education reform can be halted: A suit filed by the family of Junior Seau:

“The family said the league not only ‘propagated the false myth that collisions of all kinds, including brutal and ferocious collisions, many of which lead to short-term and long-term neurological damage to players, are an acceptable, desired and natural consequence of the game,’ but also that ‘the N.F.L. failed to disseminate to then-current and former N.F.L. players health information it possessed’ about the risks associated with brain trauma.”

This law suit has prompted a considerable amount of debate concerning whether or not the NFL as we currently know it could be dramatically reconfigured under the pressure of more law suits. In other words, the inherent but often ignored or concealed dangers of football are now being exposed by legal action, in much the same way as the tobacco industry was unmasked and thus the entire culture of smoking has radically changed in the last couple decades.

With the release of the Education Policy Analysis Archives (EPAA) Special Issue on “Value-Added Model (VAM) Research for Educational Policy,” a similar question should now be raised about the future of implementing high-stakes accountability policies that focus on teacher evaluation and retention through VAM-style metrics.

“High-Stakes Implementation of VAM,…Premature”

Two articles in the special issue from EPAA examines the validity and reliability of VAM-based teacher evaluation in high-stakes settings and then places these policies in the context of legal ramifications faced by districts and states for those policies.

“The Legal Consequences of Mandating High Stakes Decisions Based on Low Quality Information: Teacher Evaluation in the Race-to-the-Top Era” (Baker, Oluwole, & Green, 2013) identifies the current trend: “Spurred by the Race-to-the-Top program championed by the Obama administration and a changing political climate in favor of holding teachers accountable for the performance of their students, many states revamped their tenure laws and passed additional legislation designed to tie student performance to teacher evaluations” (p. 3). Because of the political and public momentum behind reforming teacher evaluation, Baker, Oluwole, and Green seek “to bring some urgency to the need to re-examine the current legislative models that put teachers at great risk of unfair evaluation, removal of tenure, and ultimately wrongful dismissal” (p. 5).

While Baker, Oluwole, and Green offer a detailed and evidence-based examination of the VAM-based and student growth model approaches to high-stakes teacher accountability, they ultimately place the weaknesses of reform policies in the context of potential challenges from teachers who believe they have been wrongfully evaluated or dismissed:

“In this section, we address the various legal challenges that might be brought by teachers dismissed under the rigid statutory structures outlined previously in this article. We also address how arguments on behalf of teachers might be framed differently in a context where value-added measures are used versus one where student growth percentiles are used. Where value-added measures are used, we suspect that teachers will have to show that while those measures were intended to attribute student achievement to their effectiveness, the measures failed to do so in a number of ways. That is, where value-added measures are used to assign effectiveness ratings, we suspect that the validity and reliability, as well as understandability of those measures would need to be deliberated at trial. However, where student growth percentiles are used, we would argue that the measures on their face are simply not designed for attributing responsibility to the teacher, and thus making such a leap would necessarily constitute a wrongful judgment. That is, one would not necessarily even have to vet the SGP measures for reliability or validity via any statistical analysis, because on their face they are invalid for this purpose.”

The analysis ultimately discredits both the use of narrow metrics to determine teacher quality and the high-stakes policies being implemented using those metrics, concluding with the ironic consequences of these policies: “Overly prescriptive, rigid teacher evaluation mandates, in our view, are likely to open the floodgates to new litigation over teacher due process rights. This is likely despite the fact that much of the policy impetus behind these new evaluation systems is the reduction of legal hassles involved in terminating ineffective teachers” (pp. 18-19).

In “Legal Issues in the Use of Student Test Scores and Value-added Models (VAM) to Determine Educational Quality” (Pullin, 2013), the rapid increase of VAM-based accountability is further examined in the context of “a wide array of potential legal issues [that] could arise from the implementation of these programs” (p. 2).

Pullin notes the motivation for reforming teacher evaluation:

“VAM initiatives are consistent with a highly publicized press from the business community and many politicians to make government services more like private business, data-driven to measure productivity and accountability (Kupermintz, 2003). VAM approaches are in part a response to concerns that the current system of selecting and compensating teachers based their education and credentials is insufficient for insuring teacher quality (Corcoran, 2011; Gordon, Kane & Staiger, 2006; Hanushek & Rivkin, 2012; Harris, 2011). There have been increasing expressions of concern that teacher evaluation practices are not robust and do not improve practice (Kennedy, 2010). In the contemporary public policy context, much of the support for the use of student test scores for educator evaluation comes from a concern that the current system for evaluation is ineffective and that the current legal protections for teachers are too cumbersome for schools seeking to terminate teachers (Harris, 2009, 2011).”

While a business model for addressing quality control of a work force may seem efficient, Pullin highlights that legal ramifications are likely with these new models.

Pullin’s analysis offers a detailed and useful examination of previous court cases involving the use of test scores to evaluate educators, including recent cases involving VAM, concluding that the picture is not clear on how the courts may rule in the future, but that a pattern exists of “heavy judicial deference to state and local education policymakers and the allure of using test scores to make decisions about education quality” (p. 5).

Further, Pullin notes “there are differences of perspective among social scientists about VAM and the defensibility of using it to make high-stakes decisions about educators,” further complicating the concerns of legal action (p. 9).

While raising many other complications, Pullin also notes that students and parents may enter legal battles using VAM metrics “to substantiate their own legal claims that schools are not meeting their obligations to provide education” (p. 14).

Pullin concludes with a sobering look at teacher quality reform built on VAM and implemented in high-stakes environments:

“In the broad contemporary public policy context for education reform, the desire for accountability and transparency in government, coupled with heavily financed criticisms of public school teachers and their unions, may mean that VAM initiatives will prevail. The concerns of education researchers about VAM, coupled with legal obligations for the validity and reliability of education and evaluation programs should require judges and education policymakers to take a closer look for future decision-making. At the same time, the social science research community should be generating substantial new and persuasive evidence about VAM and the validity and reliability of all of its potential uses. For public policymakers, there are strong reasons to suggest that high-stakes implementation of VAM is, at best, premature and, as a result, the potential for successful legal challenge to its use is high. The use of VAM as a policy tool for meaningful education improvement has considerable limitations, whether or not some judges might consider it legally defensible.” (p. 17)

Like the NFL, federal and state governments may soon be compelled to reform the reform movement under the threat of legal action from a variety of stakeholders since the science of teacher evaluation remains far behind the curve of implementation, particularly when teacher evaluation is high-stakes and based on VAM and other metrics linked to student test scores.

The special issue from EPAA is yet another call for political leadership to pause if not end wide-scale teacher evaluation and retention models that pose legal, statistical, and funding challenges that those leaders appear unwilling to acknowledge or address.

Teacher Evaluation, VAM

VAM: A Primer

January 19, 2013 plthomasedd 4 Comments

Education reform has existed at some policy and public levels since at least the 1890s in the U.S. The current reform movement grounded in state-based accountability began in the early 1980s with a Nation at Risk, and then was nationalized in 2001 with No Child Left Behind.

In the first decades of the recent accountability era, standards and high-stakes testing were implemented and periodically revised at the state level with the primary focus being on student and school accountability. The current cycle, however, has seen an increase in policies and practices aimed at teacher accountability and increasing teacher quality—despite a solid research base showing that teacher quality constitutes only 10-15% of measurable student outcomes (test data).

The focus on increasing teacher quality and accountability has included both experiments and policies with value added methods (VAM) of determining teacher quality in order to label, rank, sort, and retain or dismiss teachers. VAM claims to isolate teacher quality through pre- and post-testing methods that seek to identify teacher quality and isolate that from the other factors reflected in test scores.

Before policy-makers and stakeholders in education commit to reforming teacher evaluation and retention, foundational questions must be addressed, and then the current facts about VAM must be acknowledged.

First, the foundational questions:

(1) What evidence exists identifying teacher quality as a primary or significant problem facing a school or district? Where, then, does teacher quality rank as a priority for a school, district, or state in terms of cost effectiveness in committing funding to the reform?

(2) Are all elements of implementing VAM at any percentage to the revised teacher evaluation process valid for determining teacher quality? In other words, what measures are taken to account for using student test scores (designed to reflect student learning, and not designed to reflect teacher quality) as data points for teacher quality?

(3) Have the teaching and learning environments for students’ home and schools been addressed to insure equitable teaching and learning environments in which determining teacher quality becomes valid?

Next, what is the current knowledge base about VAM*?:

(1) Including VAM at any percentage in reformed teacher evaluation models is currently in the experimental phase. Data and the validity of including VAM are being tested, but almost no researchers currently claim VAM (at any percentage but certainly at high percentages such as 40-50%) to be ready for widespread implementation.

(2) VAM models for labeling teacher quality are highly unstable. Teacher rankings tend to shift with new populations of students.

(3) Researchers agree that VAM is unlikely ever to be completely stable; thus, it is possible that VAM will never be a practical or fair element in teacher evaluation, particularly at the individual teacher level or in any single year.

(4) The statistical and practical requirements to isolate teacher quality in student test scores pose tremendous costs in time and funding that also may prove to be not cost effective in the context of most school systems’ priorities. In order to implement a fair and equitable teacher evaluation system that includes VAM at any percentage, states must create and implement pre- and post-tests to all students in all teachers’ courses, creating a new and costly commitment to education funding.

(5) Decades of research on high-stakes testing have shown many negative unintended consequences to accountability measures focusing on students and schools; including VAM in high-stakes accountability policies focusing on teachers is likely to have similar unintended negative consequences such as discouraging high-quality teachers from working with high-needs populations of students.

Thus, policy-makes and stakeholders are strongly cautioned to consider education reform priorities, the experimental nature of VAM, and the current knowledge base on VAM before committing tax dollars to either field testing or implementing new teacher evaluation policies built in any way on VAM.

* See a recent review of teacher evaluation reform for links and citations to numerous research studies on VAM.

CCSS, Education, Educational Research, No Excuses Reform, Rhee, Social Context Reform, Standards, Teacher Evaluation, VAM

Assembled Pieces Reveal Disturbing Reform Picture

January 17, 2013 plthomasedd 1 Comment

Every time I write about Michelle Rhee, as I noted in a recent post, I feel like I should reenact the shower and wire-brush scene in Silkwood to purge myself of participating in the ceaseless media attention disproportionately afforded Rhee while the voices, daily efforts, and expertise of K-12 practitioners are not just ignored, but marginalized and even demonized.

So it is with a shared reservation (see Jose Vilson’s excellent post) that I once again wade into the Common Core State Standards (CCSS) debate—not to rehash my unequivocal opposition to the CCSS movement, but to offer a brief look at the picture revealed once all the pieces of the corporate/ “no excuses” reform movement puzzle are assembled. First, then, let me identify the primary pieces of that puzzle:

CCSS

National high-stakes tests built on CCSS

Reformed teacher evaluation driven by VAM-based teacher ranking

Teach for America

Charter schools

These various pieces are an effective strategy with a common thread because separately each reform element creates a focal point of debate; for an educator or researcher to challenge any one of these policies is a seemingly endless task since the reform agenda is being set by those with political and financial power. Refuting the need for new standards, much less the flaws with implementing those new standards, immediately positions educators as reactionary and allows the self-appointed reformers to characterize those challenges as being for the status quo and against reform and accountability.

For example, teachers in my home state of South Carolina who have spoken against VAM-style teacher evaluation reform have been publicly labeled by the state superintendent of education, Mick Zais, as trying to avoid being held accountable for their work.

The picture these reform pieces show is not a patchwork of evidence-based and innovative strategies for improving public education, but a carefully unified process of infusing even more deeply the power of high-stakes standardized testing into the fabric of public schools. Look beneath any of the elements listed above and find the allure of new and better tests, as Secretary of Education Arne Duncan (2010) celebrated himself:

Today is a great day! I have looked forward to this day for a long time–and so have America’s teachers, parents, students, and school leaders. Today is the day that marks the beginning of the development of a new and much-improved generation of assessments for America’s schoolchildren. Today marks the start of Assessments 2.0. And today marks one more milestone, testifying to the transformational change now taking hold in our nation’s schools under the courageous leadership and vision of state and district officials.

Duncan’s entusiasm doesn’t stop there:

This new generation of mathematics and English language arts assessments will cover all students in grades three through eight and be used at least once in high school in every state that chooses to use them. In addition, the PARCC consortium will develop optional performance tasks to inform teachers about the development of literacy and mathematics knowledge and skills in kindergarten through second grade.

I am convinced that this new generation of state assessments will be an absolute game-changer in public education. For the first time, millions of schoolchildren, parents, and teachers will know if students are on-track for colleges and careers–and if they are ready to enter college without the need for remedial instruction. Yet that fundamental shift–re-orienting K-12 education to extend beyond high school graduation to college and career-readiness–will not be the only first here.

For the first time, many teachers will have the state assessments they have longed for– tests of critical thinking skills and complex student learning that are not just fill-in-the-bubble tests of basic skills but support good teaching in the classroom.

And what provides the basis upon which Duncan makes these claims?:

Yet existing assessments are only part of the problem. An assessment system and curriculum can only be as good as the academic standards to which the assessments and curriculum are pegged. We want teachers to teach to standards–if the standards are rigorous, globally competitive, and consistent across states. Unfortunately, in the last decade, numerous states dummied down their academic standards and assessments. In effect, they lied to parents and students. They told students they were proficient and on track to college success, when they were not even close.

The Common Core standards developed by the states, coupled with the new generation of assessments, will help put an end to the insidious practice of establishing 50 different goalposts for educational success. In the years ahead, a child in Mississippi will be measured against the same standard of success as a child in Massachusetts.

Even if we account for the sort of soaring rhetoric associated with political discourse, Duncan clearly envisions policy that must include a staggering and unprecedented commitment to testing that rises to the level of parody. But for all stakeholders in public education, the results of all the policies linked to standardized testing must include a brave new world of testing that boggles the mind in terms of the amount of time and funding required to design, field test, implement, and manage pre- and post-tests aligned with CCSS for every single course and teacher year after year after year.

As Yong Zhao has detailed carefully in an exchange with Marc Tucker, commitments to education reform policy linked to CCSS and the high-stakes tests built on these new standards are not anything new, are not justified by any clearly identified problems or needs, and are not consistent with the larger democratic goals of universal public education:

[L]et me restate my main point: it is impossible, unnecessary, and harmful for a small group of individuals to predetermine and impose upon all students the same set of knowledge and skills and expect all students progress at the same pace (if the students don’t, it is the teachers’ and schools’ fault). I am not against standards per se for good standards can serve as a useful guide. What I am against is Common and Core, that is, the same standards for all students and a few subjects (currently math and English language arts) as the core of all children’s education diet. I might even love the Common Core if they were not common or core.

Classroom teachers, educational researchers, and educational historians have offered and continue to offer a clear and valid voice that Duncan’s claims and the resulting policies are deeply flawed, but as Brian Jones asks, “If all of this testing is so bad for teaching and learning, why is it spreading?” According to Jones, the answer detailed in the full picture is clear:

As the tests spread and the consequences associated with them rise, absurdities abound….

The shift toward using student data to evaluate teachers is part of a larger trend of restructuring public education to align it with the rest of the economy. As one of the last heavily unionized groups of workers in the country, teachers stand in the way of privatization. And to the extent that they are self-governed, self-motivated and enjoy professional autonomy, teachers are a ‘bad’ example for other workers.

Even though it may not make for great teaching or genuine learning, high-stakes standardized testing is spreading because it is the perfect tool for controlling and disciplining teachers–and for training the next generation to internalize the priorities of the system.

The attempt to quantify and track every aspect of an employee’s ‘performance’ is not new.

Standardized testing—the inevitable consequence of commitments to CCSS, reformed teacher evaluation, and each piece of the corporate reform puzzle—combines the veneer of objectivity with the power of perpetual control over schools, teachers, and students, what Foucault characterized as “…entering the age of infinite examination and of compulsory objectification” (p. 200):

The exercise of discipline presupposes a mechanism that coerces by means of observation; an apparatus in which the techniques that make it possible to see induce effects of power in which, conversely, the means of coercion make those on whom they are applied clearly visible….

[T]he art of punishing, in the regime of disciplinary power, is aimed neither at expiation, nor precisely at repression….It differentiates individuals from one another, in terms of the following overall rule: that the rule be made to function as a minimal threshold, as an average to be respected, or as an optimum toward which one must move. It measures in quantitative terms and hierarchizes in terms of value the abilities, the level, the ‘nature’ of individuals….The perpetual penalty that traverses all points and supervises every instant in the disciplinary institution compares, differentiates, hierachizes, homogenizes, excludes. In short, it normalizes….

The examination combines the techniques of an observing hierarchy and those of normalizing judgment. It is a normalizing gaze, a surveillance that makes possible to qualify, to classify, and to punish….

In discipline, it is the subjects who have to be seen. Their visibility assures the hold of the power that is exercised over them. It is the fact of their being constantly seen…that maintains the disciplined individual in his subjection. And the examination is the technique by which power…holds them in a mechanism of objectification. (pp. 177, 170, 197, 199)

Now, in the context of whether or not the U.S. is committed to universal public education as a central element of a commitment to democracy and individual liberty, and then whether or not education reform is seeking that foundational goal, time has come to set aside the puzzle-piece-by-puzzle-piece dismantling of the corporate reform agenda and confront directly the central flaw with the picture itself, as Jones acknowledges:

The solution to this dilemma is not to develop better tests, but to tear down the whole enterprise of high-stakes standardized assessment and replace it with authentic assessments that are organic to the process of real teaching and learning.

In sum, the attempt to quantify learning and teaching in a standardized manner is extremely expensive; takes up weeks and, in some places, months of time in school; narrows the curriculum; undermines the intrinsic joy of learning; and leads to a culture of corruption and cheating. As a measure of student learning, standardized tests are an extremely limited instrument. As a measure of teacher effectiveness, they are even more flawed.

Measuring, labeling, ranking, and then sorting students, teachers, and schools is an anti-democratic process, a dehumanizing process, and a mechanism for control. At the center of this process being antithetical to both our democracy and our faith in education is the fundamental flaw of high-stakes standardized testing.

Do many of the puzzle pieces of the corporate reform puzzle misuse standardized tests and the data drawn from those tests? Yes.

But we must not fall prey to the simplistic claim that the problem is how tests are used and not the tests themselves.

The ugly full picture of corporate reform shows that the problem is testing. Period.

No Excuses Reform, Statistics, Teacher Evaluation, Teaching, VAM

Teacher Quality Mania: Backward by Design

January 14, 2013 plthomasedd 4 Comments

Let’s return to the allegory of the river.

Throughout the Land, people discovered babies floating in the river. A few were chosen to save those babies. While many survived, too many babies perished.

Technocrats, Economists, and Statisticians gathered all the Data that they could and discovered that at least 60% of the reason the babies survived or perished in the river was due to babies being tossed in the river; about 10-15% of the reason babies survived or perished was due to the quality of those trying to save babies in the river.

So the Leaders of the Land decided to focus exclusively on increasing the quality of those trying to save the babies floating in the river, saying, “There is nothing we can do about babies being tossed in the river, and there are no excuses for not saving these babies!”

And so it goes…

•

While this altered tale above reads like a dystopian allegory, it is a fair and accurate portrayal of the current mania to address teacher quality—a mania that simply has the entire reform process backward.

First, the body of research shows a clear statistical pattern about the array of factors influencing measurable student outcomes, as summarized by Di Carlo:

But in the big picture, roughly 60 percent of achievement outcomes is explained by student and family background characteristics (most are unobserved, but likely pertain to income/poverty). Observable and unobservable schooling factors explain roughly 20 percent, most of this (10-15 percent) being teacher effects. The rest of the variation (about 20 percent) is unexplained (error). In other words, though precise estimates vary, the preponderance of evidence shows that achievement differences between students are overwhelmingly attributable to factors outside of schools and classrooms (see Hanushek et al. 1998; Rockoff 2003; Goldhaber et al. 1999; Rowan et al. 2002; Nye et al. 2004).

When educators and education researchers note that teacher quality is dwarfed by other factors, primarily out-of-school factors associated with affluence and poverty, Corporate and “No Excuses” Reformers respond with straw man arguments that quoting statistical facts is somehow saying teachers cannot have an impact on students or that quoting those facts is simply an excuse for not trying to educate all students (see Larry Ferlazzo and Anthony Cody for examples of this phenomenon in the debate over teacher quality).

To be clear, however, the problem is not that teacher quality doesn’t matter or that teachers do not want to be evaluated or held accountable. The problem is that addressing in a single-minded way teacher quality is self-defeating since (as the altered allegory above shows) it has the priorities of reform backward.

Teacher quality reform should occur, but it must come after the primary factors impacting learning and teaching conditions are addressed, thus making it possible to make valid and reliable evaluations of teacher quality. That process should be:

(1) Address first and directly the inequity of opportunity in the lives of children to create the conditions within which schools/teachers can succeed and thus school and teacher quality can be better evaluated and supported. As stated in a recent review of misleading “no excuses” and “miracle” school claims: “Addressing out-of-school factors is primary and fundamental to resolving education inequality” (Paige, 2013, January).

(2) Address next equity and opportunity within schools. Teaching conditions must be equitable in all school and for all students. Currently, affluent and successful students have the most experienced certified teachers and also sit in AP and IB classes with low student/teacher ratios while poor and struggling students have new and un-/under-certified teachers, sitting in high student/teacher ratios classes that are primarily test-prep. Inequitable teaching/learning conditions actually mask our ability to identify quality teachers.

(3) And then, once out-of-school equity is addressed and then in-school equity is addressed focusing on teaching and learning conditions, teachers must be afforded autonomy; and finally, we can gather credible evidence to begin identifying valid teacher quality metrics to inform evaluating, supporting, and retaining teachers.

The first and second priorities can be implemented simultaneously and immediately, with the third priority delayed until conditions are equitable enough to make authentic assessments of teacher impact on student learning. [And regardless, everyone involved in teaching and learning can and must continue to teach as well as possible; that is a given.]

Current arguments that only teacher quality matters are neither statistically accurate nor an effective reform priority.

Current arguments that only teacher quality matters are a frantic effort to save the babies floating in the river while ignoring the real crisis of babies being thrown in the river in the first place.

LWV, Teacher Evaluation, VAM

Review [UPDATED]: “How to Evaluate and Retain Effective Teachers” (League of Women Voters of SC)

January 3, 2013 plthomasedd 28 Comments

The League of Women Voters of South Carolina has released a report entitled “How to Evaluate and Retain Effective Teachers” (2011-2013) with the identified purpose “to examine the growing movement toward ‘results based’ evaluation nationally and in South Carolina.”

Before examining the substance of the report, several problems with the larger context of teacher evaluation and retention as they intersect with important challenges facing SC need to be identified:

• The report fails to clarify and provide evidence that teacher quality and retention are primary problems facing SC. Without a clear and evidence-based problem, solutions are rendered less credible. However, SC, like most of the US, has a teacher assignment problem that has been clearly documented: students of color, students from poverty, English language learners (ELL), and special needs students are disproportionately assigned to un-/under-certified and inexperienced teachers (Peskey & Haycock, 2006). As well, the implied problem of the report marginalizes the greatest obstacles facing SC schools, poverty and the concentration of poverty (notably the Corridor of Shame along I-95).

• The report compiles and bases claims on a selection of references that are not representative of the body of research on teacher quality and value-added methods (VAM) or performance-based systems of identifying teacher quality. As detailed below, the claims and research included in this report misrepresent the reports themselves as well as the current knowledge-base on teacher quality and retention.

In that context, the report fails the larger education reform needs facing SC as well as the stated purpose of the study.

Do Teachers Matter?

The opening claim of the report asserts “the most important school-based factor is an effective teacher,” and then cites Hanushek, among others. While the report is careful to note teacher quality is an important in-school factor, it repeatedly overstates teacher quality’s impact with terms such as “overwhelmingly” and fails to clarify that in-school factors are dwarfed by out-of-school factors. Matthew Di Carlo offers a balanced picture of the proportional impact of teacher quality, including an accurate interpretation of many of the same references (such as Hanushek):

“But in the big picture, roughly 60 percent of achievement outcomes is explained by student and family background characteristics (most are unobserved, but likely pertain to income/poverty). Observable and unobservable schooling factors explain roughly 20 percent, most of this (10-15 percent) being teacher effects. The rest of the variation (about 20 percent) is unexplained (error). In other words, though precise estimates vary, the preponderance of evidence shows that achievement differences between students are overwhelmingly attributable to factors outside of schools and classrooms (see Hanushek et al. 1998; Rockoff 2003; Goldhaber et al. 1999; Rowan et al. 2002; Nye et al. 2004).” [1]

Along with misrepresenting the impact of teacher quality on measurable student outcomes, the report lends credibility to a misrepresented and flawed study by Chetty, Friendam and Rockoff (2011):

“[T]hose using the results of this paper to argue forcefully for specific policies are drawing unsupported conclusions from otherwise very important empirical findings.” (Di Carlo)

“These are interesting findings. It’s a really cool academic study. It’s a freakin’ amazing data set! But these findings cannot be immediately translated into what the headlines have suggested – that immediate use of value-added metrics to reshape the teacher workforce can lift the economy, and increase wages across the board! The headlines and media spin have been dreadfully overstated and deceptive. Other headlines and editorial commentary has been simply ignorant and irresponsible. (No Mr. Moran, this one study did not, does not, cannot negate the vast array of concerns that have been raised about using value-added estimates as blunt, heavily weighted instruments in personnel policy in school systems.)” (Baker)

The teacher quality impact is misrepresented in this report and perpetuates popular and agenda-driven research myths such as the need for consecutive years of high-quality teachers; the claim is inaccurate and should not drive policy:

“This is important, because the ‘X consecutive teachers’ argument only carries concrete policy implications if we can accurately identify the ‘top’ teachers. In reality, though, the ability to do so is still extremely limited [emphasis added].

“So, in the context of policy debates, the argument proves almost nothing. All it really does – in a rather overblown, misleading fashion – is illustrate that teacher quality is important and should be improved, not that policies like merit pay, higher salaries, or charter schools will improve it.

“This represents a fundamental problem that I have discussed before: The conflation of the important finding that teachers matter – that they vary in their effectiveness – with the assumption that teacher effects can be measured accurately at the level of the individual teacher (see here for a quick analogy explaining this dichotomy)….

“But the ‘X consecutive teachers’ argument doesn’t help us evaluate whether this or anything else is a good idea. Using it in this fashion is both misleading and counterproductive. It makes huge promises that cannot be fulfilled, while also serving as justification for policies that it cannot justify. Teacher quality is a target, not an arrow.” (Di Carlo)

Has Traditional Teacher Evaluation Failed?

One of the implied and cited reasons for addressing teacher quality and retention rests on wide-spread criticism of traditional teacher evaluation policies and practices. This report lends a great deal of credibility to those criticisms while relying on The Widget Effect from The New Teacher Project (TNTP). [2] However, a review of this report calls into question, again, the credibility of the study’s claims as well as using it as a basis for policy decisions:

“Overall, the report portrays current practices in teacher evaluation as a broken system perpetuated by a culture that refuses to recognize and deal with incompetence and that fails to reward excellence. However, omissions in the report’s description of its methodology (e.g., sampling strategy, survey response rates) and its sample lead to questions about the generalizability of the report’s findings.”

Di Carlo has also documented that TNTP continues to misrepresent and overstate the impact of teacher quality and the effectiveness of identifying high-quality teachers:

“I just want to make one quick (and, in many respects, semantic) point about the manner in which TNTP identifies high-performing teachers, as I think it illustrates larger issues. In my view, the term ‘irreplaceable’ doesn’t apply, and I think it would have been a better analysis without it….

“Based on single-year estimates in math and reading, a full 43 percent of the NYC teachers classified as ‘irreplaceable’ in 2009 were not classified as such in 2010. (In fairness, the year-to-year stability may be a bit higher using the other district-specific definitions.)

“Such instability and misclassification are inevitable no matter how the term is defined and how much data are available – it’s all a matter of degree – but, in general, one must be cautious when interpreting single-year estimates (see here, here and here for related analyses).

“Perhaps more importantly, if you look at how they actually sorted teachers into categories, the label irreplaceable,’ at least as I interpret it, seems inappropriate no matter how much data are available.”

Performance-Based Teacher Evaluations: Is VAM Credible?

A significant portion of the report makes claims about value-added methods (VAM) of teacher evaluations in the context of performance-based approaches to identifying teacher quality. Nationally, VAM and other performance-based policies are being implemented quickly, but with little regard to the current understanding of the effectiveness and limitations of those policies; this report fails to represent the current state of research on VAM accurately and depends on research and think tank advocacy (National Council on Teacher Quality [NCTQ]) that distorts the importance of teacher quality and the effectiveness of identifying teacher quality based on measurable student outcomes.

The report remains supportive of performance-based policy recommendations, but does identify cautions about test-based teacher evaluations while also encouraging teacher evaluations include multiple measures and include teachers in the creation of a new evaluation system.

Two failures, however, of this report’s endorsement of VAM and/or performance-based teacher evaluations systems include couching that endorsement in the distorted claims about teacher quality’s impact on measurable student outcomes and depending on reports and claims made by NCTQ and the Bill and Melinda Gates Foundation. [3]

What, then, are the current patterns from the research on VAM and performance-based models and how should those patterns shape policy? [4]

• VAM and test-based evaluations for teachers remain both misleading about teacher quality and misrepresented by research, the media, and political leadership. Numerous researchers have detailed that teachers identified as high-quality or weak one year are identified differently in subsequent years: Numerous factors beyond the control of teachers remain reflected in test scores more powerfully than the individual impact of any specified teacher. The debate over teacher quality and measuring that quality, then, is highly distorted, as Di Carlo explains: “Whether or not we use these measures in teacher evaluations is an important decision, but the attention it gets seems way overblown.” This report makes that mistake.

• VAM and performance-based teacher evaluations in high-stakes settings distort teaching and learning by narrowing the focus of both teaching and learning to teaching to the test and test scores. VAM and test-based data are likely valuable for big picture patterns and in-school or in-district decision making regarding teacher assignment, but VAM and performance-based evaluations of individual teachers remain inaccurate and inappropriate for evaluation, pay, or retention.

Particularly in a state such as SC where poverty and state budget concerns burden the state and the public school system, VAM and performance-based systems that rely on extensive retooling of standards, testing, and teacher evaluation systems are simply not cost effective (Bausell, 2013): “VAM is not reliable or valid, and VAM-based polices are not cost-effective for the purpose of raising student achievement and increasing earnings by terminating large numbers of low-performing teachers” (Yeh, 2014).

And further, rejecting VAM and using significant percentages of student test scores to evaluate and retain teachers is not rejecting teacher accountability, but confronting the misuse of data. Ewing (2011) clarifies that VAM is flawed math and thus invalid as a tool in teacher evaluation:

“Of course we should hold teachers accountable, but this does not mean we have to pretend that mathematical models can do something they cannot. Of course we should rid our schools of incompetent teachers, but value-added models are an exceedingly blunt tool for this purpose. In any case, we ought to expect more from our teachers than what value-added attempts to measure.”

If SC choses to reform teacher evaluation—which remains a project far less urgent than other problems being ignored—the state would be guided better by Gabriel and Allington (2012), who have analyzed and challenged the Gates Foundations MET Project, which has prompted misguided and hasty implementation of VAM-style teacher evaluation reform:

“Although we don’t question the utility of using evidence of student learning to inform teacher development, we suggest that a better question would not assume that value-added scores are the only existing knowledge about effectiveness in teaching. Rather, a good question would build on existing research and investigate how to increase the amount and intensity of effective instruction.”

Gabriel and Allington (2012) recommend five questions to guide teacher evaluation reform, instead of VAM or other student-outcome-based initiatives:

“Do evaluation tools inspire responsive teaching or defensive conformity?…

“Do evaluation tools reflect our goals for public education?…

“Do evaluation tools encourage teachers to use text in meaningful ways?…

“Do evaluation tools spark meaningful conversations with teachers?…

“Do evaluation tools promote valuable education experiences?”

Again, these guidelines are evidence-based alternatives to discredited and experimental commitments to the misrepresented evidence in the report from LWV SC, but SC remains overburdened by issues related to equity and opportunity that outweigh the need to reform teacher evaluation at this time.

How Should SC Proceed with Teacher Quality and Retention?

On balance, this report misrepresents teacher quality and overstates the need and ability to identify high-quality teachers using VAM and other performance-based policies. The flaws in this report grow from an over-reliance on misguided and misrepresented research and advocacy while ignoring the rich and detailed evidence from the full body of research on teacher quality. Finally, the report concludes by discrediting SC’s current teacher evaluation system (ADEPT) in the context of the inaccurate and distorted claims in the report.

Ultimately, the report encourages SC to spend valuable time and resources on policies that are dwarfed by more pressing needs facing the state and its public schools—a failure of state leadership replicated in the perpetual retooling of state education standards (Common Core State Standards adoption) and high-stakes testing based on revised standards. In short, SC has a number of social and educational challenges that need addressing before the state experiments with revising teacher evaluation and retention policies, including the following:

• Identify how better to allocate state resources to address childhood and family poverty, childhood food security, children and family access to high-quality health care, and stable, well paying work for families.

• Replace current education policies based on accountability, standards, and testing with policies that address equity and opportunity for all students.

• Address immediately the greatest teacher quality issue facing SC’s public schools—inequitable distribution of teacher quality among students in greatest need (high-poverty children, children of color, ELL, and special needs students).

• Address immediately the conditions of teaching and learning in the state’s schools, including issues of student/teacher ratios, building conditions and material availability, administrative and community support of teachers, equitable school funding and teacher salaries, teacher job security and academic freedom in a right-to-work state, and school safety.

Any policy changes that further entrench the culture of testing in SC as a mechanism for evaluating students, teachers, and schools perpetuate the burden of inequity in the state and schools.

SC does not need new standards, new tests, or a new teacher evaluation system. All of these practices have been implemented in different versions with high-stakes attached for the past thirty years—with the current result being the same identified failures with public schools that were the basis of these policies.

SC, like much of the US, needs to come to terms with identifying problems first before seeking solutions. The problems are ones of equity and opportunity, and no current teacher evaluation plan is facing those realities, including this report.

[1] For additional examinations of out-of-school factors compared to teacher quality and in-school factors see Berliner, 2009, 2014; Hirsch, 2007; Rothstein, 2010; Traub, 2000.

[2] Also see Di Carlo, 2011, on TNTP.

[3] For evaluations of NCTQ and Gates see the following: (a) NCTQ: Benner, 2012; Merrill & Ingersoll, 2010; Baker, 2010, January 29; Baker, 2010, October 8; Baker, 2010, December 4, and (b) Gates: Baker, 2010, December 13; Baker, 2011, March 2; Libby, 2012; [Gates MET Project] Gabriel & Allington, 2012; Rubinstein, 2013, January 13; Rubinstein, 2013, January 9; Baker, 2013, January 9; Glass, 2013, January 14; Rothstein & Mathis, 2013

[4] For comprehensive examinations of research on VAM, see Di Carlo, 2012, November 13*; Di Carlo, 2012, April 12 (reliability); Di Carlo, 2012, April 18 (validity); Baker, 2012, May 28 (misrepresentations of VAM); Baker, et al., 2010, August 27; Ewing, 2011, May. For recent concerns about legal action and VAM-based teacher evaluation, see Baker, Oluwole, & Green, 2013; Pullin, 2013

* Di Carlo remains a proponent of including VAM in teacher evaluations.

References

Berliner, D. C. (2014). Effects of inequality and poverty vs. teachers and schooling on America’s youth. Teachers College Record, 116(1). Retrieved from http://www.tcrecord.org/content.asp?contentid=16889

Berliner, D. C. (2009). Poverty and potential: Out-of-school factors and school success. Boulder, CO and Tempe, AZ: Education and the Public Interest Center & Education Policy Research Unit. Retrieved from http://nepc.colorado.edu/publication/poverty-and-potential

Gabriel, R., & Allington, R. (2012, November). The MET Project: The wrong 45 Million dollar question. Educational Leadership, 70(3), 44-49. Retrieved from http://216.78.200.159/Documents/RandD/Teacher%20Evaluation/MET%20Project%20-%20Wrong%2045%20Millon%20Question%20-%20Gabriel.pdf

Hirsch, D. (2007, September). Experiences of poverty and educational disadvantage. York, North Yorkshire, UK: Joseph Rowntree Foundation. Retrieved from http://www.jrf.org.uk/knowledge/findings/socialpolicy/2123.asp

Peske, H. G., & Haycock, K. (2006, June). Teaching inequality: How poor and minority students are shortchanged on teacher quality. Washington DC: The Education Trust, Inc. Retrieved from http://www.edtrust.org/sites/edtrust.org/files/publications/files/TQReportJune2006.pdf

Rothstein, R. (2010, October 14). How to fix our schools. Issue Brief 286. Washington DC: Economic Policy Institute. Retrieved from http://www.epi.org/publications/entry/ib286

Thomas, P. L. (2011, December). Teacher quality and accountability: A failed debate. Daily Censored. Retrieved from http://www.dailycensored.com/teacher-quality-and-accountability-a-failed-debate/

Traub, J. (2000, January 16). What no school can do. New York Times Magazine. Retrieved from http://local.provplan.org/pp170/materials/what%20no%20school%20can%20do.htm