Teacher Evaluations Based on Test Scores Will Requires Dozens of New Tests

The argument for longer school days and school years is getting a big boost in recent weeks, as states contemplate the dozens of new tests that will be needed for teacher evaluation plans based on test scores. Kindergarten or 12th grade,
P. E. or computer programming, history or home ec, everyone will have to be tested, and more than once a year.  Remember, this is about added value.  Remember?

When Broadie (rhymes with toady) Peter Gorman (Class of '04) rolled out the Oligarchs' year-round testing plan in Charlotte recently, non-corporate ed reformers noted that it didn't make much sense to spend $2 million on 52 new, unnecesssary tests for kindergarten through grade 12 that parents do not want, especially when the school system is laying off teachers and aides to make payroll.  But, then, we have come to learn from the Broadies that it is not the needs of adults that schools should consider--it is about the needs of corporations.  How could anyone, even Bill Gates, make the case to parents that this year-round testing to evaluate teachers can be in the interests of children?  Anyone, anyone?

Well, now another Broadie, Bloomberg's school CAO Shael Polakow-Suransky (Class of '08) has marched out the plan in New York City, this one with a price tag of over $60,000,000 for testing companies to develop tests for grades 3 through 12 (twice a year) in math, science, English, and social studies.

These tests come on top of the annual NCLB testing and the end-of-course Regents' Exams.  Initial implementation is planned for this Fall, even though no community or teacher input has been sought or received, and no validity or reliability has been established for these tests that the testing industry is still cooking up.  And what happened to the new Chancellor?  The last time we saw him, Mr. Wolcott was walking his grandson to school and preparing pink slips for 4,500 NYC teachers.

From the Times:

. . . .Elementary school students would most likely take at least one or two additional tests every year, beginning in the third grade. High school students could take up to eight additional tests a year, and middle school students would also have extra tests. These would be in addition to the state English, math and Regents exams that students already take.

The exams, which would begin rolling out as early as next academic year, are being created as part of a statewide overhaul of how teachers are evaluated. Under a law passed last year that helped the state win $700 million in a federal grant competition, known as Race to the Top, each school district must find a way to evaluate teachers on a scale from “ineffective” to “highly effective,” with teachers facing potential firing if they are rated ineffective for two years in a row.

Under the law, 40 percent of a teacher’s grade will be based on standardized tests or other “rigorous, comparable” measures of student performance. Half of that should be based on state tests, and half on measures selected by local districts. The remaining 60 percent is to be based on more subjective measures, including principal observations.

Most districts will not create their own standardized tests, an expensive process that requires considerable expertise. The state does not require them to do so, instead permitting districts to set academic goals for teachers, broadly defined.

But New York City, which has made standardized tests a centerpiece of its school reform efforts, is pushing ahead. The city schools system is planning to use up to one-quarter of its $256 million share of the federal grant money for as many as 16 new standardized exams to cover science, math, social studies and English in the 3rd through 12th grades.. . . .

From McGraw-Hill's Kelly Girls to Pearson's No Girls at All

Four years ago today I posted on the developing scandal involving McGraw-Hill's hiring of Kelly part-timers and under-qualified scorers for Florida's FCAT.  Well, now that Pearson has the $254,000,000 contract with the State of Florida, they seem to cutting corners as well. 

Reading the memo that Valerie Strauss posted from Chancellor Haithcock to public school superintendents, it is clear that Pearson does not have enough people on the job, which is the reason for the unprecedented delays in the test scores that shape everything educational in Florida.  I have clipped the relevant section from Valerie's posting:

. . . despite confidence in this year’s results, I am sure you remain curious about the cause of the delays we have been experiencing. These difficulties are directly related to Pearson’s database technology, which handles student demographic information.

Florida’s assessment process involves a great number of validation procedures, beyond the validation of students’ scores, which are designed to ensure that information in each student’s assessment record is complete and accurate. During the scanning and scoring process, database files are created by the contractor from the tests received from school districts. These files are matched against a Pre-Identification file, which contains student information reported to the contractor directly from districts prior to the testing administration. The student information contained in the test results files must match up perfectly with each district’s Pre-Identification file so that you can seamlessly integrate the test results into your student information database. It is in this file matching that our delays are occurring. Each apparent mismatch must be researched, resolved and rechecked in order to ensure accuracy. While this checking and rechecking process is clearly outlined in Pearson’s scope of work, internal systems issues within their database are causing delays in the resolution of these matches.

Additionally, I realize that you may have particular concerns with this year’s FCAT Writing results as they have been delayed the longest. The extended delay on these scores is the result of the additional validation review by Buros and the database issues mentioned above. Again, we remain fully confident in the results of this FCAT Writing administration.

I am also aware that some of you remain apprehensive about how the FCAT writing results will translate into the school grading formula. Due to the testing administration change from two essay scorers to one, a score of 3.5 is no longer possible, which necessitates a slight change in grade calculations.. . .

Not enough staff to handle the mismatches and a reduction in force from two scorers to one.  That's what $254 million gets you from the corporate swindlers.

From a Scorer with 15 Years Experience Inside the Testing Industry

From an op-ed (ht to Monty Neill) in today's Christian Science Monitor by Todd Farley, author of Making the Grade: My Misadventures in the Standardized Testing Industry:

NEW YORK - President Obama's "Race to the Top" initiative promotes an accountability in education that links teacher evaluations with student test scores. While many will point to the potential problems this poses for the classroom, there's something else to think about. This position exhibits considerable confidence in the results returned to students by the standardized testing industry.

That's not a faith I share.The 15 years I spent scoring student essays and written responses to state K-12 tests revealed a process that was less "scientifically based research" (a heady phrase dropped more than 100 times in No Child Left Behind) than a theater of the absurd.

On one of the first tests I scored, I was hired (after barely flashing my college diploma at a human resources representative) to assess high school writing. It was a job granted with the caveat that as temporary workers, we could keep the position only if we passed a "qualifying" test.

We'd have to score enough student essays accurately during a training session to prove we could do the job in a standardized way, a process that allowed the testing company that hired us to claim just how capable, consistent, and qualified its employees were.

It was a nerve-racking experience to know our jobs hinged on those "qualifying" tests, but in the end we part-timers needn't have worried so much.

After two days of training, nearly half the 100 people applying for the job failed the tests and were fired. Our unemployment lasted only about 12 hours. The next morning nearly every one of us flunkies was hired right back, an employment rebirth that occurred just as soon as the testing company realized it was short on personnel.

I asked the HR representative about the 70 percent accuracy on the qualifying test we were required to get in order to keep the job. To which she informed me that they had decided 60 percent was good enough after all.

During my time in testing I was witness to (or party to) every conceivable manner of corporate chicanery and statistical tomfoolery, any shortcut imaginable that would help those for-profit companies get tests scored, deadlines met, and money made.

On one scoring project I managed, for instance, the government agency in charge passed down an edict stating that all scorers had to go through a remedial retraining (group discussions with their peers about scoring rubrics and training papers) after any work stoppage of 30 minutes or more, including their scheduled half-hour lunch break. The government agency in charge said such retrainings would help ensure the student responses were scored within the proper context of "psychometric rigor."

Those retrainings, however, also invariably devolved into time-consuming and contentious debates among the scorers about what score Anchor Paper No. 3 or Practice Paper No. 9 really deserved, theoretical arguments that completely interfered with the project's real goal of slapping scores on student tests.

Because the testing company that hired us was more concerned with scoring tests than achieving any sort of "psychometric rigor," they solved the problem of any extra retraining simply by changing the work schedule, eliminating from our workday any 30-minute breaks.

"From now on," I told the group of temps I was supervising, "we'll take a 29-minute lunch. Enjoy."

Obviously I can't just be pointing fingers here at big, bad corporate America because I am guilty, too, and during my time in testing there were few corners I didn't cut and few rules I didn't bend. I fudged the numbers like everyone else. Much of my career managing scoring projects was spent manipulating statistics to give off the appearance of a test-scoring process that was consistent and standardized even when it clearly was not.

And while I don't mean to imply such statistical skullduggery was the testing industry's official stance, neither could it have been unknown that such things regularly occurred.

Just before I left, for instance, I saw a temporary supervisor chastised and demoted for too obviously cheating on his scoring group's accountability stats, although that harsh reprimand barely slowed down the accused in his charge right up the company ladder. Within a month that same guy – the one whose blatant manipulation of test stats had earned him both a pay cut and a verbal redressing from company management – was given a full-time job as a project manager.

While accountability in education may be an important goal, it's critical to realize how difficult that might be to pin down. The lesson of my career should be that trusting massive corporations that answer to a bottom line to make decisions about American schools is a whole lot different than trusting those men and women who stand every day at the front of the classroom.

Too often in my career the test results we returned had to be viewed not as exemplars of educational progress, but rather as numbers produced in a mad rush to get things done, statistics best viewed solely through the prism of profit. Caveat emptor, Mr. President.

Todd Farley is the author of "Making the Grades: My Misadventures in the Standardized Testing Industry."