Value-Added Modeling (VAM) is pseudoscience, but profitable pseudosciences persist

"While value-added models are intended estimate teacher effects on student achievement growth, they fail to do so in any accurate or precise way. — Professor Bruce Baker"

Back in early April I penned a piece for K12NN on The American Statistical Association's (ASA) paper on Value Added Methodologies [1]. In it I asserted that the "document provides strong support to those who oppose this wrongheaded use of statistics to make high stakes decisions effecting the lives of students, educators, and our school communities." This week I noticed a trackback to a reprint of the article. What caught my eye was the title, which seemingly was entirely out of keeping with the spirit of the ASA's stance: Advocating for a robust value-added implementation.

I read through the VAM cheerleading piece and was gobsmacked by the deliberate manipulation of the ASA document's tenor and tone. This VAM apologetic read less like a legitimate blog posting and more like a corporate press release. Without doing much more research, I typed the following comments:

This quotation from the ASA document sums up the entire issue best: "The majority of the variation in test scores is attributable to factors outside of the teacher’s control". To, as the author above has, try and frame ASA's position as supportive of VAM phrenology takes mendaciousness to breathtaking heights. Rather than considering students as empty receptacles for "knowledge" deposited by a method that can be "measured," perhaps we can start talking about students as agents in their own pedagogical experiences—something that doesn't exist in the current regime of the profitable testing-industrial-complex.

The author's response displayed the same blatant avoidance of issues as the original piece. In fact, it stated some of the long discredited claims of the VAM camp, including Professor Bruce Baker's favorite trope about how "more sophisticated" VAMs address the issues people have with VAMs.

Thanks for your comment. The ASA statement seems to discuss primary drivers of student test scores, not student growth. It is well known that there is a strong relationship between students’ achievement (or test scores) and their socioeconomic/demographic background. However, there is typically little or no relationship between students’ growth and their socioeconomic/demographic background.

Another way to see this is that the most important factor of “current” test scores is prior tests scores and, once enough prior test scores are included in the model, the socioeconomic/demographic factors become relatively small or even non-significant, despite enormous sample sizes.

That said and to your concern about considering students in the context of their own experiences, more sophisticated value-added/growth models, like EVAAS, can follow the progress of individual students over time, so that each student serves as his or her own control.

Aside from being patently wrong, the whole thing smacked of being boilerplate text written in the bowels of a corporate public relations department. That's when I decided to look into this dubious Jennifer Facciolini was and who she wrote for. I should have done that in the first place.

As of 2012, Statistical Analysis System (SAS) Institute Inc., is one of the largest privately owned software companies in the world with revenue $3.02 billion USD (2013). They are the developers of the wildly inaccurate, but highly profitable Education Value-Added Assessment System, (AKA SAS EVAAS) — a VAM implementation used in many districts. Money chasing SAS is all about big fish government contracts, to wit an excerpt from a recent Businessweek piece:

SAS Institute Inc. won a $6,479,583.96 federal contract from the Defense Information Systems Agency, Scott Air Force Base, Illinois, for Statistical Analysis System software licenses and support renewal.

When a huge firm like SAS pulling down big dollar defense contracts makes the education "market" their priority, you can believe that they won't let things like facts and evidence discrediting VAM get in the way of them selling and supporting their EVAAS phrenology kit to any and all districts infected by the neoliberal corporate reform virus.

Hiring several white, well educated former-teachers like Nadja Young and Jennifer Facciolini to shill for your product is a smart public relations investment for the VAM behemoth. Given that they were teachers in the south, the chances are that they are making a great deal more money at SAS by simply selling out their former profession to corporate interests. Whether the boilerplate prose in their gushing blog posts is written by them or not isn't all that important. What's important is the appearance that professional teachers actually might think that phrenology and VAM are legitimate sciences. I can only hope that that the prose in these mindless corporate posts isn't written by these former teachers. Students should never be exposed to such nonsensical drivel. A selection of some of their titles should serve to numb the mind of any sentient being:

• Data-driven education books make great holiday gifts for educators. Yes, really.
• "March madness" of student course enrollment gets assist from value-added assessment
• Beyond value-added: Teachers need diagnostic data to improve their practice
• Student growth measures can be the bridge to new assessments

I'll spare readers further torment. As the preponderance of evidence against VAM pseudoscience like the watershed ASA paper grows, expect profit hungry firms like SAS to keep doubling down on the duplicity and deception. Having an army of former teachers shilling for your defective product is a small expense compared to losing those highly profitable contracts with districts.

---

NOTES

[1] In its various permutations we've seen the "M" in VAM stand for Modeling, Measures, Methodologies, and others. The only honest word for last member of the acronym would be "Mendaciousness," since phrenology by any other name is…

K12NN: American Statistical Association has just released a very important document on Value Added Methodologies

First published April 9, 2014 on K-12 News Network

---

"The President of the United States and his Secretary of Education are violating one of the most fundamental principles concerning test use: Tests should be used only for the purpose for which they were developed. If they are to be used for some other purpose, then careful attention must be paid to whether or not this purpose is appropriate" — Gerald Bracey, PhD

The American Statistical Association (ASA) released their ASA Statement on Using Value-Added Models for Educational Assessment today. While their spokesperson explicitly said they neither support, nor oppose the use of so-called "Value Added" methodologies, the actual document provides strong support to those who oppose this wrongheaded use of statistics to make high stakes decisions effecting the lives of students, educators, and our school communities. Too bad the amateur statisticians at the Los Angeles Times were able to commit their egregious acts several years ago before this document was released. It's also too bad that LAUSD recently implemented one of these seriously flawed models, one that will abjectly harm students' education and further undermine the morale of our professional educators for years to come.

Some important excerpts from the document (all emphasis mine):

Estimates from VAMs should always be accompanied by measures of precision and a discussion of the assumptions and possible limitations of the model. These limitations are particularly relevant if VAMs are used for high-stakes purposes. (1)

VAMs should be viewed within the context of quality improvement, which distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools. Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores, and that the majority of opportunities for quality improvement are found in the system-level conditions. Ranking teachers by their VAM scores can have unintended consequences that reduce quality. (2)

In practice, no test meets this stringent standard, and it needs to be recognized that, at best, most VAMs predict only performance on the test and not necessarily long-range learning outcomes. Other student outcomes are predicted only to the extent that they are correlated with test scores. A teacher’s efforts to encourage students’ creativity or help colleagues improve their instruction, for example, are not explicitly recognized in VAMs. (4)

Attaching too much importance to a single item of quantitative information is counter-productive—in fact, it can be detrimental to the goal of improving quality. In particular, making changes in response to aspects of quantitative information that are actually random variation can increase the overall variability of the system. (5)

The quality of education is not one event but a system of many interacting components. (6)

A decision to use VAMs for teacher evaluations might change the way the tests are viewed and lead to changes in the school environment. For example, more classroom time might be spent on test preparation and on specific content from the test at the exclusion of content that may lead to better long-term learning gains or motivation for students. (6)

Overreliance on VAM scores may foster a competitive environment, discouraging collaboration  and efforts to improve the educational system as a whole. (6)

The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences. (7)

The VAM scores themselves have large standard errors, even when calculated using several years of data. These large standard errors make rankings unstable, even under the best scenarios for modeling. (7)

A VAM score may provide teachers and administrators with information on their students’ performance and identify areas where improvement is needed, but it does not provide information on how to improve the teaching (7)

All in all, the document is an academic condemnation of the VAM/AGT pseudosciences that have been ushered in by neoliberal corporate education reform project. While the ASA is populated with actual scientists and statisticians, we can be sure that the corporate reform crowd will be quick to try to refute the document. Here the tag-line of a recent article in Salon by Paul Rosenberg is apropos: 'Like global warming deniers, "education reformers" have nothing to  lose and everything to gain by sowing confusion'.

For a copy of the ASA Statement on Using Value-Added Models for Educational Assessment see http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf. For additional information, please visit the ASA website at www.amstat.org.

"The majority of the variation in test scores is attributable to factors outside of the teacher’s control"—ASA #LAUSD http://t.co/Ihydoazv4K

— Robert D. Skeels (@rdsathene) April 8, 2014

Refocusing LAUSD on reading and learning instead of testing

First published on Robert D. Skeels for School Board on November 24, 2012.

---

It's difficult to explain exactly what being poor is all about, or why access to books and ideas might be as important as a free breakfast. — Walter Dean Myers

Hope Is an Open Book, an op-ed piece by author Walter Dean Myers, was tweeted this morning by educator Susan Ohanian. While written in 2005, Myers' message about access to books is profound and even more urgent today with canned corporate education solutions that narrow curriculum dominating policy. Sadly, Los Angeles Unified School District (LAUSD) is run by a Superintendent who neither values books, nor libraries. This has to change, and change quickly. We need to shutdown LAUSD's testing-industrial-complex and reopen both our school and classroom libraries. Reopening libraries also means rehiring credentialed librarian-educators. We can pay for that by ditching discredited and expensive attempts to tie teacher evaluations to test scores (VAM/AGT) and use the millions of squandered dollars associated with them. Read with your children, read in front of your children, and let them choose their own reading materials. It's a proven formula for fostering authentic life-long learning.

Free Voluntary Reading (FVR) is a well researched methodology in which students are allowed to choose their own reading materials. Professor Stephen Krashen and his colleagues have found that "[r]ecreational reading or reading for pleasure is the major source of our reading competence, our vocabulary, and our ability to handle complex grammatical constructions." The Power of Reading, Second Edition: Insights from the Research is an excellent text to familiarize oneself with the concepts and research behind FVR.

How the VAM/AGT pseudoscience worm turns

While value-added models are intended estimate teacher effects on student achievement growth, they fail to do so in any accurate or precise way — Dr. Bruce D. Baker

Maybe it's karma, but Kyle Hunsberger of the Gates Foundation backed astroturf TeachPlus, and one of Los Angeles Unified School District's biggest cheerleaders of the highly discredited VAM/AGT pseudosciences, is now a victim of that selfsame modern phrenology.

This quote is pretty amazing coming from a bona fides member of the "no excuses" camp.

"I have to be reassured that I don't have to lobby for honors students," Hunsberger said. "I have to know that I have a shot at a good evaluation if I teach lower-performing kids."

Maybe he'll be a former member of that camp from here out, now that he's experienced the practical application of his faulty theoretical framework. That framework incorrectly posits that "effective" teaching "overcomes" any other factor including poverty, student motivation, or even English Language Learner status.

Call me cynical, but I doubt Michael Stryer and James Encinas, Hunsberger's TeachPlus coauthors of this fact-free Op-Ed shilling for value added measures will stand by him now that the worm has turned. Corporate reformers are just like that, they're snakes and most venomous to anyone who break ranks and start telling the truth.

Let's stop testing children and start teaching them. Standardized tests are a perversion. NCLB/RTTT/CCSS are the real "status quo" and an abject failure at that.

Evaluate This! Shows the way forward for authentic education reform

"I am heading for a four-year college. The classes that most helped me prepare for this were NOT test prep classes. They were the ones where I got to explore stuff and do projects." — Tiffany Tuggle (CEJ Member and Senior at UCLA Community School)

Jessica Kochick teaches at the Miguel Contreras Learning Complex in the Westlake District of Los Angeles near my home. Jess and I come out of the same political tradition, and over the years we've worked together on several struggles fighting the racist, classist, neoliberal policies of Los Angeles Unified School District (LAUSD). A year or two ago, Jess was discussing standardized testing and Value Added Measures (VAM) — or Average Growth Over Time (AGT) in LAUSD parlance — with several of us and suggested an event demonstrating how the vast majority of real pedagogy can't be measured on standardized tests or quantified in any similar fashion.

Her vision eventually led to Evaluate This!, an education fair showcasing projects at various schools sites throughout LAUSD. With an emphasis on project based learning, the fair featured some absolutely amazing forms of pedagogy. Even though I arrived to catch the very end of the event, it exceeded expectations in every possible way. Even the virulently anti-public education Los Angeles Times ran a blog [1] piece entitled L.A. students and teachers display learning beyond tests.

Evaluate This! was sponsored by Progressive Educators for Action (PEAC), United Teachers Los Angeles (UTLA) North Area, UTLA Raza Education Committee, and Coalition for Educational Justice (CEJ). The same groups also arranged for Professor Linda Darling-Hammond to speak to educators about authentic teacher evaluations at the Historic Coconut Grove Auditorium in the Robert F. Kennedy School Complex a week later.

A notable display included Grand View Boulevard Elementary School's Dual Language Immersion program. A bystander informed me that they heard it had been in place for 21 years. Thinking immersion programs had only been around for about a decade, I wrote my good friend and fellow neighborhood activist Cheryl Ortega, who is also Director of Bilingual Education at UTLA. She confirmed the program has been around that long, and pointed out that it has flourished in an affluent neighborhood, but our task is to convince inner city schools to adopt these programs. Cheryl and I have discussed many times in the past how it is often difficult to convince immigrant parents to enroll their children in dual language immersion programs given the racist nature of U.S. society that both discourages impoverished peoples from retaining their own cultural identities, combined with the pressures on schools from No Child Left Behind (NCLB) to force English proficiency on students.

Cheryl had the following to add in terms of her experience from the education fair:

The entire event was wonderful. I loved the mix of elementary, middle and high school exhibits. I was particularly impressed by the project on violence against women done by 3 high school boys. Well done!

There were amazing robotics experiments from Dahlia Heights ES, a garden snail project from MacArthur Park Primary Center, and even books published by classes of students at various schools. One that caught my eye was Sheep can't fly by students of Garfield High School, with a forward by Father Greg Boyle, S.J.. Several years ago the Garfield community successfully fought off LAUSD privatizer Yolie Flores' attempt to give their school away to the Green Dot Charter Corporation run by capricious corporate CEO Marco Petruzzi.

My neighborhood was well represented with a biology project from Logan Street Elementary School headed up by the widely regarded Julie Van Winkle. Many of the CEJ Student Activists that had made the Social Justice Schools Conference so dynamic were also in attendance, and many of them had displays at the fair.

I was able to get two of the participants to submit brief write ups of their projects. The second of the write ups is in relation to a poignant display from an amazing educator who through the sheer budget mismanagement of LAUSD, like many of our newer teachers, receives a reduction in force (RIF) letter on an anual basis. To add insult to injury, many of these RIFs could be averted if the district wasn't squandering hundreds of millions of dollars on discredited and disproven Value Added Measures (VAM).

Esperanza Elementary School first grade garden project at Evaluate This!

Our garden exhibit at Evaluate This! On 5/18/12 at Miguel Contreras Learning Center.

Our display at Evaluate This! documented with photos, student writing, drawings, graphs, crafts and realia how students in four first-grade classes at Esperanza Elementary School have embarked on a quest to learn to grow healthy food in an urban asphalt environment. Students began by planting lettuce, radishes and tomatoes in containers and a small raised bed in a concrete courtyard. They've cared for the plants, documented their growth in writing, graphs and illustrations, started over with new plants when some died, captured green caterpillars who came to dine on the plants and later watched them turn into butterflies, released ladybugs to eat aphids that threatened their plants, and enjoyed eating the lettuce and radishes they grew. (The tomatoes aren't ready yet!) Students have read and discussed books about plants, learned about the needs of all living things, and all have written booklets about their gardening adventures. Some conducted experiments to see if varying the amount of water affects plant growth, and graphed their results. Many have adopted healthier eating habits, snacking at school on carrots, apples, bananas, tangerines and sugar peas still in the pods. Students now want to try growing a wider variety of vegetables, and some have already started growing them in containers in the apartments where they live. School gardening offers a venue for engaging real-life learning about social studies, language arts, math and science that goes far beyond anything measurable on a scantron. Our gardening project is evolving as it proceeds, and we hope to expand it next year to include more classes. The teachers are Angie Low, Alicia Rodriguez, Lynda Ayala and Anne Zerrien-Lee.

How Standardize Tests and Value Added Measures Undermine Learning

My name is Larry Shoham and I teach 8th grade English and 7th grade World History at Westside Global Awareness Magnet.

We are here to challenge the notion that our students are not engaged in meaningful learning experiences. Every day I go to work I am engaged in educational reform. My colleagues and I are constantly reflecting on our practices, trying figure out ways to make our instruction more meaningful and engaging to the students we teach. We know our students and we understand their needs. Innovation isn't achieved through top-down mandates and relentless onslaughts of bubble tests. It's achieved through collaboration, academic freedom, and equitable resources.

Unfortunately, so many rich learning experiences will be lost once we succumb to these new evaluations. There will be incredible pressure on us to teach to the tests and abandon the active, project-based learning that does not directly lead to measurable gains on multiple-choice tests. Instead of speaking, writing, and questioning, our students will be subjected to mindless exercises in an attempt to micromanage testable skills. In fact, it's already happening. Last year, I read two novels with my students. This year, we barely got through one. Why? In an attempt to juke the stats and squeeze our school out of program improvement status, we agreed, as a staff, to commit more time toward test prep--teaching directly to the State's test release questions and reviewing, ad nauseam, the District's periodic assessments. My students were bored out of their minds. I was bored out of my mind. Sadly, this is the game we have been intimidated into playing.

Value-Added Measurement (what LAUSD calls "AGT") is problematic in so many ways. This past fall, LAUSD sent me a confidential AGT report evaluating me as an English teacher. It was a number. It did not tell me anything useful that I could apply in the classroom. The fact that our precious public money is being spent on LAUSD's AGT made me angry. These state tests are meant to assess students on their grade-level mastery of ELA content standards. Having large populations of English Language Learners and students who are far below grade level puts so many of our teachers at an immediate disadvantage. An 8th grade student who is reading at a 5th grade level may improve to a 6th or 7th grade level, but her gains will never be recognized because she is being assessed on her mastery of an 8th grade reading level. This is not right. Yet the LA Times publishes this horrendously flawed data. The public is being fed misinformation about our schools and our teachers.

AGT does nothing to help us improve our instructional practices. We envision something that relates more closely to the NBPTS (National Board of Professional Teaching Standards) certification process. Check out the plan we propose on www.utla.net. Just like the evaluations we give our students, evaluations for our teacher are useless if they are only punitive, if they can't be used as a tool for learning and improvement. I am a National Board Certified teacher. I learned so much by going through the process. We need an evaluation system will provide us with meaningful feedback and promote professional growth among all teachers in our District.

The press release from the organizers listed four framing principles:

• Standardized tests are discriminatory to students.
• Evaluating teachers based on standardized tests will harm education and harm students with the greatest needs.
• Value added measures (VAMs) are inaccurate, unreliable and wasteful.
• Academic freedom is necessary if teachers are going to be able to engage students in curriculum that will help them reach both their academic and human potential.

In the wake of Pearson's Pineapplegate and the ominous limiting of learning that will inevitably result from Corporate Core Curriculum, there's never been a more critical time to hold these types of events all around the country throughout the year. It's not that students are clamoring for more testing, and many parents are beginning to catch on to the testing industrial complex. It is my strong belief that events like Evaluate This! make it politically untenable for the privatizers to continue to push their agenda of eliminating critical thinking skills from public schools. If it can be answered on a scantron, it probably isn't worth learning!

_____
NOTES

[1] Naturally they didn't see fit to print anything depicting public education in a positive light in their actual paper.

Guest Post: LAUSD Should Fund Schools in Need, Not District Offices

By David Lyell

First published in the Huffington Post

Dear LAUSD School Board Members, Superintendent Deasy, Secretary Duncan, and President Obama,

We all want to provide the educational opportunities for children and our communities. Please help me receive clarity on the following:

Instead of sending Title I, II, and III money to school sites, as is intended under these programs, LAUSD senior management has chosen to keep this money at central district offices in order to fund unproven, costly initiatives such as the Teaching and Learning Initiative, commonly known as the Value-Added Teacher Evaluation model.

Due to a lack of transparency (the dust hasn't yet settled), it's not even clear yet how much will be spent centrally on these unproven programs, but it appears to be well over $175 million.

Is this legal?

I am in no way questioning the integrity of LAUSD senior district management. Perhaps cutting these programs so we can hire more consultants was a mere oversight.

Isn't this decision a clear violation of the 2011 Voluntary Resolution Agreement between LAUSD and the U.S. Department of Education's Office of Civil Rights? If not a clear violation, and I think it very well may be, at the very least, does it not violate the spirit of the agreement?

LAUSD's English learners and African-American students disproportionally suffer when money is spent at LAUSD's central administrative offices rather than at schools in our most under-served communities. Schools with well-funded PTAs that have deep pockets will be okay, but schools that can not provide these essential supports will continue to go without libraries, nurses, and counselors, among other services.

Is this really what we want for our communities? When children in affluent communities have all the best supports as they grow and learn, and children in our most economically depressed neighborhoods aren't afforded the same opportunities -- we can not even begin to approach using words to describe such an injustice.

I've taught in schools in our most under-served communities, and I've taught in schools where children have every kind of support service at their disposal, and the difference in opportunities provided is unconscionable.

It is simply disingenuous at best to suggest that we can bridge the achievement gap by simply raising expectations. Yes, expectations should be high, for students, teachers, administrators, and parents, but such an argument diverts attention away from the vast inequality in services afforded students.

A school is not a spreadsheet. A school is not data we can examine on a page so that a six-figure consultant who doesn't even live or work in the neighborhood can make snap judgments about what the community most needs.

A school is a community where relationships form, and through these bonds, children, teachers, counselors, nurses, librarians, parents and administrators establish trust, and nurture, foster, and create an ever-changing, constantly growing, always tenuous environment where mistakes become opportunities, and the insurmountable becomes possible.

These relationships only form and grow when schools foster an environment where children who otherwise would drop-out have a reason to stay in school.

Children need libraries, nurses, counselors, arts programs, access to adult education opportunities, vocational classes, early childhood education, music, dance, band, and sports programs. For 20 years, leaders across the country have been saying that the first five years of life are vital to child development, yet right now, today, the LAUSD school board is poised to decimate early childhood education.

Instead of fostering and growing the above programs, LAUSD is proposing to cut all of the above programs, either entirely eliminating them or decimating their funding to roughly 10% of their previous levels.

They say they don't have the money, but they recently found private foundation money to hire a social media director at a cost of $93,000 per year.

The argument that they don't have money would make a little more sense if the district wasn't proposing to spend $175 million at central district offices rather than providing this money as intended under Title I, II, and III to students in our most underserved communities.

Please tell me this was simply an administrative error. I'll repeat the question once more:

Isn't the decision to spend Title I, II, and III money at LAUSD central district offices rather than at school sites in our most under-served communities a clear violation of the 2011 Voluntary Resolution Agreement between LAUSD and the U.S. Department of Education's Office of Civil Rights?

David Lyell is a 13-year veteran LAUSD teacher, now serving as Secretary of United Teachers Los Angeles, after being elected March 29, 2011. He blogs at Intersections South LA, LA Progressive, and huffingtonpost.com.