Yesterday, I spent an hour on the phone with the producer of a national news series.
I realized afterward that much of the conversation reminded me of dozens of similar conversations with journalists throughout my 40-year career as an educator because I had to carefully and repeatedly clarify what standardized tests do and mean.
Annually for more than the first half of my career, I had to watch as the US slipped into Education Crisis mode when SAT scores were released.
Throughout the past five decades, I have been strongly anti-testing and anti-grades, but most of my public and scholarly work challenging testing addressed the many problems with the SAT—and notably how the media, public, and politicians misunderstand and misuse SAT data.
Over many years of critically analyzing SAT data as well as the media/public/political responses to the college entrance exam, many key lessons emerged that include the following:
Lesson: Populations being tested impact data drawn from tests. The SAT originally served the needs of elite students, often those seeking Ivey League educations. However, over the twentieth century, increasingly many students began taking the SAT for a variety of reasons (scholarships and athletics, for example). The shift in population of students being tested from an elite subset (the upper end of the normal curve) to a more statistically “normal” population necessarily drove the average down (a statistical fact that has nothing to do with school or student quality). While statistically valid, dropping SAT scores because of population shifts created media problems (see below); therefore, the College Board recentered the scoring of the SAT.
Lesson: Ranking by test data must account for population differences among students tested. Reporting in the media of average SAT scores for the nation and by states created a misleading narrative about school quality. Part of that messaging was grounded in the SAT reporting average SAT scores by ranking states, and then, media reporting SAT average scores as a valid assessment of state educational quality. The College Board eventually issued a caution: “Educators, the media and others should…not rank or rate teachers, educational institutions, districts or states solely on the basis of aggregate scores derived from tests that are intended primarily as a measure of individual students.” However, the media continued to rank states using SAT average scores. SAT data has always been strongly correlated with parental income, parental level of education, and characteristics of students such as gender and race. But a significant driver of average SAT scores also included rates of participation among states. See for example a comparison I did among SC, NC, and MS (the latter having a higher poverty rate and higher average SAT because of a much lower participation rate, including mostly elite students):
Lesson: Conclusions drawn from test data must acknowledge purpose of test being used (see Gerald Bracey). The SAT has one very narrow purpose—predicting first-year college grades; and the SAT has primarily one use—a data point for college admission based on its sole purpose. However, historically, media/public/political responses to the SAT have used the data to evaluate state educational quality and the longitudinal progress of US students in general. In short, SAT data has been routinely misused because most people misunderstand its purpose.
Recently, the significance of the SAT has declined, students taking the ACT at a higher rate and more colleges going test-optional, but the nation has shifted to panicking over NAEP data instead.
The problem now is that media/public/political responses to NAEP mimic the exact mistakes during the hyper-focus on the SAT.
NAEP, like the SAT, then, needs a moment of reckoning also.
Instead of helping public and political messaging about education and education reform, NAEP has perpetuated the very worst stories about educational crisis. That is in part because there is no standard for “proficiency” and because NAEP was designed to provide a check against state assessments that could set cut scores and levels of achievement as they wanted:
Since states have different content standards and use different tests and different methods for setting cut scores, obviously the meaning of proficient varies among the states. Under NCLB, states are free to set their own standards for proficiency, which is one reason why AYP school failure rates vary so widely across the states. It’s a lot harder for students to achieve proficiency in a state that has set that standard at a high level than it is in a state that has set it lower. Indeed, even if students in two schools in two different states have exactly the same achievement, one school could find itself on a failed-AYP list simply because it is located in the state whose standard for proficient is higher than the other state’s….
Under NCLB all states must administer NAEP every other year in reading and mathematics in grades 4 and 8, starting in 2003. The idea is to use NAEP as a “check” on states’ assessment results under NCLB or as a benchmark for judging states’ definitions of proficient. If, for example, a state reports a very high percentage of proficient students on its state math test but its performance on math NAEP reveals a low percentage of proficient students, the inference would be that this state has set a relatively easy standard for math proficiency and is trying to “game” NCLB.
In other words, NAEP was designed as a federal oversight of state assessments and not an evaluation tool to standardize “proficient” or to support education reform, instruction, or learning.
As a result, NAEP, as the SAT/ACT has done for years, feeds a constant education crisis cycle that also fuels concurrent cycles of education reform and education legislation that has become increasingly authoritarian (mandating specific practices and programs as well as banning practices and programs).
With the lessons from the SAT above, then, NAEP reform should include the following:
Ending state rankings and comparisons based on NAEP average scores.
Changing testing population of students by age level instead of grade level (addressing impact of grade retention, which is a form of state’s “gaming the system” that NAEP sought to correct). NAEP testing should include children in an annual band of birth months/years regardless of grade level.
Providing better explanations and guidance for reporting and understanding NAEP scores in the context of longitudinal data.
Developing a collaborative relationship between federal and state education departments and among state education departments.
While I remain a strong skeptic of the value of standardized testing, and I recognize that we over-test students in the US, I urge NAEP reform and that we have a NAEP reckoning for the sake of students, teachers, and public education.
Similar to ExcelinEd, Ohio Excels has entered the grade retention advocacy movement as part of the larger disaster reformreading policy movement occurring in the US for about a decade.
There is a pattern emerging in grade retention advocacy that contrasts with decades of research showing that grade retention, on balance, disproportionately impacts marginalized populations of students without improving academic achievement but correlating strongly with students dropping out of high school. [1]
The key aspects of the new advocacy reports include the following:
Funding and support by conservative think tanks.
An emphasis on early test score increases (grades 3 and 4) and claims of no negative impacts on students.
One problem is that these grade retention reports are often promoted in the media in incomplete and misleading ways, fitting into a similar pattern of education journalism.
The omissions, what is not reported, are the most important aspects of this advocacy, however.
Just as ExcelinEd uses one or two reports to endorse grade retention (again, see here for why that is misleading), this report connected to OSU has some key elements and one fatal flaw.
First, as is true about almost all grade retention, the reality of retention in OH is that it disproportionately impacts vulnerable populations of students:
The retained students were between 2.7% to 4.0% of all students subject to the retention policy. Numerically the largest group were retained in 2017 (4,590) and the smallest in 2016 (2,892).4 Overall, some 55% of retained students were male (versus 50% of not retained students), and 91% were economically disadvantaged (versus 50% of not retained students). Of the 20,870 retained some 17% had a disability (versus 10% of not retained students). In terms of race and ethnic characteristics, the largest fraction (48%) of students retained were African American (versus 14.3% of not retained students), 34% were White, Non-Hispanic (versus 72% of not retained students), 11% were Hispanic (versus 6% of not retained students), and 7% were Multiracial or Other Races (versus 5% of not retained students).
This report then concludes positive academic growth in math and reading for retained students. However, as with other recent grade retention advocacy reports, these positive academic gains remain linked to grade-level performance, and not age-level performance.
In short, retained students are always performing academically at an older age that non-retained students (note that this report carefully compares retained to nonretained students without controlling for age).
This is a key problem since even one month of age difference correlates strongly with phonics checks (and early literacy assessments tend to focus heavily on decoding and not comprehension):
Therefore, none of the recent grade retention advocacy reports show a causal relationship between retention and academic achievement. In fact, there is no evidence that the retained students’ gains are not simply being a year older.
These advocacy reports depend on the public confusing correlation and causation, and media fails to make that scientific distinction.
Decades of research as well have shown great emotional harm in grade retention; the grade retention advocacy reports simply ignore the personal and emotional consequences of grade retention by hyper-focusing on narrow measures of academic gain.
Grade retention is a punitive policy that disproportionately impacts Black and brown children, poor children, special needs children, and multi-lingual learners. [2]
Endorsing grade retention is ideological, neither scientific nor ethical.
The rise is grade retention advocacy reports are failing by omission and children are suffering the consequences of using reading legislation for political gain.
From CNN: How long you breastfeed may impact your child’s test scores later, study shows.
This sounds really compelling; it fits into a cultural narrative that breast feeding is superior to using baby formula.
This sounds really compelling until about ten paragraphs in and then:
“Though the results are certainly interesting, you have to bear in mind the limitations that inevitably arise in research using observational data from major cohort studies,” McConway added….
The fact that the study was observational means it followed people’s behavior rather than randomly assigning the behavior in question, McConway noted.
Consequently, the results only show a correlation between breastfeeding and test scores — not causation.
“It’s not possible to be certain about what’s causing what,” he said.
Few people will read that far, and even most who do will likely take away a careless claim that the research doesn’t justify.
Therefore, this article should never have been written—similar to many articles about educational research.
One enduring example of media repeating a misunderstanding of educational research is the word gap myth. Media repeat that number of words in children’s vocabulary is connected to economic status (again, this sounds right to most people).
Yet, the Hart and Risley study this myth is based on has been debunked often, and the word gap myth itself is based on flawed logic about literacy [1].
Media has ben shown, in fact, to cover education quite badly, typically overemphasizing think tank research versus university-based research (the former far less credible than the latter) and featuring the voices of non-educators (reformers and innovators) over educators:
Malin, J. R., & Lubienski, C. (2015). Educational Expertise, Advocacy, and Media Influence. Education Policy Analysis Archives, 23, 6. https://doi.org/10.14507/epaa.v23.1706
Currently, the misinformation campaign, ironically, related to education is the “science of reading” (SOR) movement that repeatedly misrepresents NAEP data, makes claims that have no scientific evidence (relying on anecdote [2]), and repeatedly relies on think tank “reports” (NCTQ, for example) that are also not scientific [3].
A subset of the SOR movement is also grade retention. High-profile coverage of Mississippi has made the exact breast feeding mistake from above: “’It’s not possible to be certain about what’s causing what,’ he said.”
Recently in the NYT, a think-tank funded report on MS grade retention is cited; however, the report itself notes that outcomes cannot be linked to grade retention itself [3].
In short, the report proves nothing about retention—just as the study on breast feeding proves nothing about student achievement.
The breast feeding story, the word gap myth, and the SOR story are all compelling because they sound true, but they are all false narratives that fails educational research—and public education.
[2] Hoffman, J.V., Hikida, M., & Sailors, M. (2020). Contesting science that silences: Amplifying equity, agency, and design research in literacy teacher preparation. Reading Research Quarterly, 55(S1), S255–S266. Retrieved July 26, 2022, from https://doi.org/10.1002/rrq.353
Thomas, P.L. (2022). The Science of Reading movement: The never-ending debate and the need for a different approach to reading instruction. Boulder, CO: National Education Policy Center. http://nepc.colorado.edu/publication/science-of-reading
The National Children’s Book and Literacy Alliance has a dire message about children and reading in the US:
The 2013 National Assessment of Education Progress (NAEP) reading test results demonstrate that far too many young people continue to read below grade level. Sixty five percent of all U.S. fourth graders scored “below proficient,” which means that they are not reading at grade level. Only 35 percent of fourth graders are reading at or above grade level. In addition, 64 percent of eighth graders are reading below grade level, whereas 36 percent are reading at or above grade level.
Doubt this claim? Well, try the NYT and Nicholas Kristof: Two-Thirds of Kids Struggle to Read, and We Know How to Fix It.
Kristof laments: “One of the most bearish statistics for the future of the United States is this: Two-thirds of fourth graders in the United States are not proficient in reading.”
That 2/3 of US children are below grade level reading is a staggering and damning statistic, nearly hard to believe until you do a little simple Googling:
Well, that Googling is a heap of trouble actually. But the message has a consistent core—the use of NAEP reading data to claim that 2/3 of students are not reading on grade level.
But I imagine there is another simple Google search that organizations, parents, journalists, and politicians are not doing:
NAEP student achievement levels are performance standards that describe what students should know and be able to do. Results are reported as percentages of students performing at or above three NAEP achievement levels (NAEP Basic, NAEP Proficient, and NAEP Advanced). Students performing at or above the NAEP Proficient level on NAEP assessments demonstrate solid academic performance and competency over challenging subject matter. It should be noted that the NAEP Proficient achievement level does not represent grade level proficiency as determined by other assessment standards (e.g., state or district assessments). See short descriptions of NAEP achievement levels for each assessment subject.
While still a complicated statistic and claim, the reality is that if we use NAEP data as evidence, about 2/3 of students in the US read at or above grade level.
That is much different than what has become common knowledge among reading crisis advocates.
As I have explained, grade-level reading proficiency is a problematic statistic that is more grounded in textbook and testing concerns than in supporting student learning or effective teaching.
Especially in the high-stakes accountability era since the 1980s, statistical efforts to evaluate and reform education have made grade-level proficiency increasingly important and a high-stakes aspect of how we treat children in our schools.
One of the most harmful consequences has been the rise in grade retention at grade 3 based on testing data—a policy associated with the Florida Model but currently celebrated in Mississippi.
Grade retention, in fact, has made grade-level proficiency an even more complicated and problematic statistic since retention removes the lowest scoring students from the testing population and then reintroduces them into a grade level one year older than their peers.
States with high levels of retained students often see test scores increase, but those increases may be from students being older and not from any identifiable academic gain (note that many states with increased grade 3 or 4 scores, the gains disappear by middle school).
In fact, test scores are highly correlated with birth months:
Therefore, for testing and especially teaching and learning, grade-proficiency should likely be replaced with age-level proficiency.
Along with shifting from grade-level to age-level proficiency, as the NAEP explanation above reveals, we need to establish a normalized standard for “proficiency.”
And next, we must stop using proficiency levels to punish students:
Grade retention, the practice of holding students back to repeat a grade, does more harm than good:
• retaining students who have not met proficiency levels with the intent of repeating instruction is punitive, socially inappropriate, and educationally ineffective;
• basing retention on high-stakes tests will disproportionately and negatively impact children of color, impoverished children, English Language Learners, and special needs students; and
• retaining students is strongly correlated with behavior problems and increased drop-out rates.
Finally, once we move from grade-level to age-level reading proficiency, create a standard for measuring that proficiency, and stop using proficiency in punitive policy, we have to come to terms with the threshold for proficiency.
One central confusion that persists with testing is the expectation for students meeting or exceeding proficiency at any designated point (grade or age).
One of the greatest flaws in NCLB was the mandate that 100% of students achieve proficiency by 2014. That impossible goal in many ways brought an end to the NCLB era.
However, we are confronted with two problems with setting age-level reading proficiency.
First, if we want age-level proficiency to be attainable by all students at any given age, that standard may be so low as to be pointless.
Next, as is more statistically valid, if we establish age-level reading proficiency where most students can be at or above proficiency, we must come to a social agreement on what percentage below proficiency is acceptable.
Historically with NAEP, that has been about 1/3 of students, which may be evidence that this is normal, even as it causes discomfort in the context of equally compelling and misleading beliefs about the urgency of third grade reading proficiency.
Our current reading crisis is doubling down on the worst aspects of all the previous reading crises as well as the worst elements of education reform.
One way to rise above these historical failures is to understand and then reform reading proficiency as one tool for supporting student achievement and teacher effectiveness.
UPDATE
In an embarrassing promoted Tweet, the University of Florida makes the same false claim about a reading crisis found in media:
More than 60% of students in our nation's K-12 schools can’t read at their grade levels, and pandemic-related challenges have exacerbated the decline. The University of Florida is leading pro-literacy efforts to combat this crisis.
Seems hard to trust a program at a university that doesn’t understand the data being used to make their claims.
CONTEXT
The “science of reading” movement is mostly journalism, anecdotes, and politics. Note that the movement has its roots in Emily Hanford’s Hard Words, which established misinformation and misunderstanding NAEP, whole language, and balanced literacy.
The evidence used for crisis shows a flat to improving trend line and Hanford’s uses anti-BL sources, Moates and Seidenberg, who have finanacial interests in SOR:
They like to get you in a compromising position They like to get you there and smile in your face They think, they’re so cute when they got you in that condition Well I think, it’s a total disgrace… I fight authority, authority always wins
As an educator for 40 years who doesn’t grade or test, I hate to do this, but let’s start with a pop quiz (and the worst possible kind, multiple choice):
In the US, where are children being indoctrinated?
public schools
their homes
their churches
all of the above
Let’s add another just for fun:
When children are indoctrinated, what ideology is being imposed on those children in the US?
Liberal
Conservative
Both
Neither
I’ll let you ponder those while you read, now, because the point here is to work our way to these answers.
I want to start with a few stories of my life and time as a public school English teacher in my small hometown that is very conservative and mostly fundamentalist Christians.
As I have written often, my childhood was nearly as idealistic as I recall. My parents were fun and doting—lots of play initiated by my parents and lots of formative engagement with my parents that lay the foundation of my becoming an academic, an avid reader, and a writer.
But, well into my late teens, I lived under the possibility of physical violence and anger from my father—although what I call “violence” was pretty mild compared to the beatings that were seen as normal in the South throughout the 1960s and 1970s.
My point is that I was raised in a household where the authoritarian/patriarchal norm was supported by corporal punishment.
By adolescence, however, I had recognized in myself a strong aversion to authority. My father’s credo, “Do as I say, not as I do,” taught me the opposite of his intended lesson.
I came to loath hypocrisy and authority-for-authority’s-sake.
In the privacy of my room, I listened to George Carlin and Richard Pryor for hours and began to read voraciously. The result was that by college, I had become a completely different person than my parents, than almost everyone in my hometown.
I toyed with rejecting religion in high school (even as I was elected president of the Fellowship of Christian Athletes my senior year), and then, I did the embarrassingly aggressive atheist routine for my first few years of college.
Ironically, although I rejected much of what my parents taught me directly and indirectly—their heavy smoking, their subtle and not-so-subtle racism, etc.—college taught me a lesson similar to my father but completely unlike my father’s mandates; in college I learned to shut my mouth an listen.
I was well into my thirties before I could recognize the distinction between authoritarian and authoritative (Paulo Freire), but that was my journey away from conservative ideologies (authoritarian) and toward liberal ideologies (authoritative) grounded in the sanctity of the human mind and the glorious possibilities of ideas most often found in books.
By the end of college, I had dedicated myself to being a teacher and a writer, always reading multiple books at a time.
My missionary zeal, then, worked in a different way than what I had witnessed growing up in a small rural town in South Carolina; I took a position in that hometown high school determined to give my students the opportunity to find their own minds, their own intellect, and not embarrass themselves as I had if and when they went off to college.
Being a teacher in a conservative small town in the South introduced me to the indoctrination paradox, in fact.
Nearly daily, I was the one being accused of indoctrination even as I had chosen to teach directly as a rejection of indoctrination. Of course, those most adamant that I was indoctrinating were the most fundamentalist people in town who were terrified of a diversity of ideas, who were the first to try to ban books, who were convinced of their own certainty in a way that was terrifying.
I have hundreds of examples, but one situation stands out to me to this day.
A beautiful part of teaching literature is that novels open the door to ideas and class discussion.
Having students read Kurt Vonnegut or Margaret Atwood, among many others, often led to discussions of free will, but when I would note to students that it defied logic to assert that there is an all-knowing god and human free will, many of my most conservative and religious students would have melt downs in class (this occurred also when we read The Scarlet Letter and confronted Original Sin).
For many of my students, my class was their first experience with questioning ideas and coming to their own understanding as opposed to simply accepting the authority of what their parents or churches told them was the Truth.
Increasingly over my nearly two decades as a public school teacher, I had homeschooled students transfer into our public school, and as a college professor at a selective liberal arts college, I teach a significant number of homeschooled students.
The subset of homeschooled students who often fit inside very conservative and fundamentalist Christian ideology was similar to those experiences while I was a K-12 teacher, but often even more pronounced.
I enter my year 40 as an educator this coming fall. For my entire career (and what I have explored as a historian of education), universal public education, books, and independent thought have always been under attack—especially from conservatives and Christians.
However, the most recent wave of book bans and curriculum gag orders focusing on CRT and chilling charges that LGBTQ+ materials are grooming children is a level of ugliness I never really expected.
The indoctrination paradox is gaining momentum because the end game of Christian conservatives is not to eradicate indoctrination or grooming from public schools, but to have complete control of indoctrination and grooming.
If you want to know what that end game looks like, take a peak inside the world of Christian conservative homeschooling: The revolt of the Christian home-schoolers by Peter Jamison.
These are the chilling highlights but you’d do yourself a service to read the entire piece, carefully:
Corporal punishment, aversion to different ideas, a fear of books, and, most chilling of all, “to reshape America according to biblical principles.”
This is the America being built in Florida, where the governor claims that book bans are not book bans.
So here is the disturbing answers from the opening pop quiz.
The first answer is “4. all of the above,” and the second answer is “2. Conservative.”
dr. p.l. (paul) thomas 