Category Archives: PISA

Big Lies of Education, Education, education reform, Educational Research, NAEP, PISA, Testing

Big Lies of Education: International Test Rankings and Economic Competitiveness

“Human development is an important component of determining a nation’s productivity and is measured in employee skills identified by employers as critical for success in the modern global economy,” claims Thomas A. Hemphill, adding:

The United States is obviously not getting a sufficient return on investment in elementary and secondary education, as it has mediocre scores in mathematics literacy and declining scores for science literacy for 15-year-old students surveyed in 2022. The only significant improvement for 15-year-olds is in reading, where the United States finally entered the top 10 in 2022.

Commentary: We must improve students’ math, science skills to boost US competitiveness

Hemphill reaches an unsurprising conclusion:

If these educational trends continue, the United States will not have an adequate indigenous workforce of scientists, engineers and technologists equipped to maintain scientific and technological leadership and instead will become perpetually reliant on scientifically and technologically skilled immigrants. We must demand that elementary and secondary education systems reorient efforts to significantly improve mathematical and scientific teaching expectations in the classroom.

Commentary: We must improve students’ math, science skills to boost US competitiveness

However, for decades, evidence has shown that there is no causal link between international rankings of student test scores and national economic competitiveness.

This Big Lie is purely rhetorical and relies on throwing statistical comparisons at the public while drawing hasty and unsupported causal claims from those numbers.

If you really care about the claim, see Test Scores and Economic Growth by Gerald Bracey.

Bracey offers this from researchers on the relationship between international education rankings and economic competitiveness:

Such countries [highest achieving] “do not experience substantially greater economic growth than countries that are merely average in terms of achievement.”

The researchers then lay out an interpretation of their findings that differs from the causal interpretation one usually hears:

“We venture, here, the interpretation that much of the achievement ‘effect’ is not really causal in character. It may be, rather, that nation-states with strong prodevelopment policies, and with regimes powerful enough to enforce these, produce both more economic growth and more disciplined student-achievement levels in fields (e.g., science and mathematics) perceived to be especially development related. This idea would explain the status of the Asian Tigers whose regimes have been much focused on producing both economic growth and achievement-oriented students in math and science.”

Test Scores and Economic Growth

Bracey quotes further from that research:

“From our study, the main conclusion is that the relationship between achievement in science and mathematics in schoolchildren and national economic growth is both time and case sensitive. Moreover, the relationship largely reflects the gap between the bottom third of the nations and the rest; the middle of the pack does not much differ from the rest. . . . Much of the obsession with the achievement ‘horse race’ proceeds as if beating the Asian Tigers in mathematics and science education is necessary for the economic well-being of other developed countries. Our analysis offers little support for this obsession. . . .

“Achievement indicators do not capture the extent to which schooling promotes initiative, creativity, entrepreneurship, and other strengths not sufficiently curricularized to warrant cross-national data collection and analysis. Unfortunately, the policy discourse that often follows from international achievement races involves exaggerated causal claims frequently stress- ing educational ‘silver bullets’ for economic woes. Our analyses do not offer defini- tive answers, but they raise important ques- tions about the validity of these claims. In an era that celebrates evidence-based policy formation, it behooves us to carefully weigh the evidence, rather than use it simply as a rhetorical weapon.”

Test Scores and Economic Growth

A key point to note here is Bracey is writing in 2007, and the OpEd above is March 2024. The Big Lie about international education rankings and economic competitiveness is both a lie and a lie that will not die.

I strongly recommend Tom Loveless exposing a similar problem with misrepresenting and overstating the consequences of NAEP data: Literacy and NAEP Proficient.

Bracey offers a brief but better way to understand test data and economic competitiveness: “education is critical, but among the developed nations differences in test scores are trivial.”

Instead of another Big Lie, the US would be better served if we tried new and evidence-based (not ideological) ways to reform our schools and our social/economic structures.

Big Lies of Education: Series
Education, education reform, Educational Research, literacy, PISA, reading, Science of Reading

UK PISA 2022 Results Offer Cautionary Tale for US Reading Reform

From the SAT and National Assessment of Educational Progress (NAEP) in the US to Programme for International Student Assessment (PISA), student test scores tend to prompt over-reactions in the media and among political leaders.

Even Finland, recently the Golden Country for educational outcomes, now finds itself with the release of PISA 2022 facing headlines like this: PISA 2022: Performance in Finland collapses, but remains above average.

However, the best lesson from PISA 2022 for the US is England (UK), especially in the context of the “science of reading” (SOR) movement driving state-level reading legislation across the country:

First, for context, England implemented phonics-centered reading policy in 2006 that rejected balanced literacy and parallels in most ways the elements of SOR-based reading legislation in the US. Research on those policies shows that students have received systematic phonics for almost two decades now, but that outcomes have not produced the promises of that reform.

UK PISA scores in reading, none the less, rank below the US along with Australia, New Zealand, Canada, and Ireland; as well, the UK suffered a large score decrease in the wake of Covid:

While how reading is taught, what reading theory is embraced, and what reading programs are being implemented vary significantly across these countries, they all have one thing in common—perpetual reading crisis.

In fact, since A Nation at Risk, the US has endured several cycles of education crisis and reform that often hyper-focuses on reading (notably the impact of the National Reading Panel in No Child Left Behind).

Yet, none of these reforms have fulfilled promised outcomes, just as we are witnessing in the UK.

PISA 2022 and England’s reading results, then, offer some lessons we have refused to acknowledge about reading and education reform:

  • Education reform that exclusively implements in-school-only reform is guaranteed to fail. The value-added methods (VAM) era under Obama revealed that teacher impact on measurable student achievement is only 1-14%, and NAEP data show that the most successful schools in the country are Department of Defense (DoDEA) schools where the inequity of children’s lives are mitigated by access to healthcare, stable housing, food security, etc.
  • Media coverage and political rhetoric proclaiming “crisis” have repeatedly misidentified the barriers to reading proficiency and teaching reading, and thus, have resulted in policy that doesn’t address those roots causes. As the UK shows, making promises that are not fulfilled only further fuels cycles of crisis and reform. Currently in the US, SOR advocates are promising 90% of students will be at grade level proficiency; however, a recent working paper on reading reform in California show only about 1/3 of students as proficient after SOR implementation, the same percentage that media has used to claim a reading crisis and a far cry from 90%.
  • PISA is a measure of 15-years-old students (see also NAEP LTT data), which offers a way to evaluate misleading short-term score gains that are more mirage than miracle. In the US, states such as Florida and Mississippi demonstrate that while grade 4 scores in reading can be inflated, that learning isn’t real, and by middle and high school, the improvements have vanished. Thus, there are no miracles and no silver bullets in education reform.

Whether NAEP or PISA, standardized test scores are less about learning and teaching but more about the incredible inequity in the lives and schooling of children.

That inequity is no crisis; that inequity is a historical fact, especially in the US where we do not have the political will to confront both the out-of-school and in-school influences on teaching and learning.

If we are concerned about the disproportional vulnerable populations of students who routinely test well below the many different criteria for reading proficiency—and we should—reading and education reform must start with addressing inequity in the lives and education of all children.

Reading theories and reading programs are neither the problems nor the solutions to increasing student reading proficiency.

The real reading crisis in the US and internationally is the refusal by those with power to read what test scores actually show us—and the failure to act in ways that can and will make attainable gains in the best interest of children, teachers, and each nation.