Flaws In The New School Tests
by Andrew Beveridge, Gotham Gazette, June, 2004

High-stakes testing of schoolchildren has been causing high anxiety not just among students and their parents, but also for teachers, principals, superintendents, even public officials - all of whose fates, in a very real way, are now determined in part by these tests. Some of the latest headlines warned that, thanks to the mayor's new policy to end "social promotion," the results of the tests for third graders in New York City public schools indicated that some 10,000 of them may be forced to repeat third grade next year. As more provisions of the No Child Left Behind Act take effect, more and more such testing will occur. Is it any wonder that the press has started to publish charts that track these scores in every single school?

But what do these tests actually measure? The truth is something that those who produce the data from the tests admit only privately: The entire process is seriously flawed. Even sustained increases over time may tell us much less about the academic performance of a student or a school than about the testing itself.

Educational researchers know that there are better ways to assess the progress of both individual students and entire school systems than the process now mandated by New York.

The current testing process is fundamentally flawed in at least three ways.

UNRELIABLE TEST RESULTS

Fundamental to the use of these tests is the assumption that they measure the same set of skills and knowledge from year to year, and for the same sort of students. But in fact a score from one year may not be equivalent to the same score the following year. It is impossible to know whether a change in a test score from one year to the next is due to a change in actual student performance or to an unintended change in the make-up of the test. Obviously, one cannot use the same questions from year to year, and even though much effort goes into making sure that the test given in 2004 yields the same standards as the test form in 2003, it is a difficult task to be absolutely sure this is so.

TEACHING TO THE TEST

No one who is working in any of the hard pressed urban schools in New York State can doubt the advantages of getting higher test scores on the English and Language Arts (ELA) and mathematics test. If a school does not improve, it may find itself on one of the several "bad school" lists prepared by the State Education Department. The school may be reconfigured; and the principal may lose his or her job or be reassigned.

So, once the new test was established in 1999, it is not surprising that districts and schools realigned their curriculum. However, as with any standardized test, there are specific things about the test itself that one needs to learn to do well on it. James Traub's article "The Test Mess," in the April 7, 2002 New York Times Magazine follows the efforts (or lack of efforts) of three districts to prepare for the exam. Such test prep can work, but it costs time, effort and resources for drill and practice. Now with the No Child Left Behind Act, there are complaints that schools are sacrificing additional parts of the curriculum to prepare the students for the test material only.

Companies now market test prep approaches to school districts. State grants, in part based upon federal money are now available for supplementary educational services that include test preparation. No matter how effective such programs are, they certainly must displace time and resources that could be used for other classroom activities, so improved scores may come at the expense of other learning.

PLAYING GAMES WITH TEST SCORES

Some commentators noted that Mayor Michael Bloomberg's efforts to end social promotion starting with third graders should lead immediately to an improvement in fourth grade test results next year, since many of the low-performing students will no longer be slated to take the test. Such "gaming" of high stakes test standards is rampant throughout the US. (See my August Column on counting dropouts)

A host of such approaches are being used to make the results look better, from suggesting the poor-achieving students stay home, to reclassifying them as special education or English As A Second Language students. One year New York City suppressed the test results for seventh grade, when they indicated a drop, after trumpeting them the year before when they indicated an increase. New York City is not alone. A national study found evidence of just this sort of Enron-style accounting among school systems throughout the United States.

BETTER METHODS OF ASSESSMENT AND TRACKING

To assess educational progress at the state or large city level, the United States Department of Education developed the National Assessment of Educational Process. It is a test based upon knowledge determined to be needed at various grade levels. This is not a high stakes test, since it is only given to a sample of students, and the results are used to track overall educational performance change and have no consequences for a specific school or student.

One does not learn anything about the effect of schools on individual students from the current testing regime, nor would one from a test like the National Assessment of Education Process. An alternative approach that overcomes many of these difficulties is the so-called "value added" approach. One tracks the gains for each particular student, not the raw test score averages for each school, district, county or state.

In this way one can relate the student gains to specific schools, teachers and educational practices. This testing approach was mandated as a remedy to civil rights violations in a Tennessee court case and is being used in some other states as well. The value added method goes well beyond simply grading schools on whether the percent of students who passed a state-mandated exam increased or decreased from one year to next. Rather it takes into account where the student starts and the extent to which growth occurs while the student is enrolled in a specific school.

New York State and New York City it seems, rely upon an inferior and invalid method of tracking student progress even though they use the information it provides for many decisions affecting schools and students. Perhaps, given the brouhaha over the recent release of test scores, more realistic methods of tracking and assessment will be adopted - especially if parents and educators demand them.

Andrew A. Beveridge has taught sociology at Queens College since 1981, done demographic analyses for the New York Times since 1993, and provides expert testimony on a range of cases, including housing discrimination. The opinions expressed are his alone.
---------------------------------------------------------------------------------------------------
Ending 'Social Promotion':Will Third Graders Learn More Now?
by Marcia Biederman, Gotham Gazette, April, 2004

A theater adage warns adults they'll be upstaged if they agree to perform with animals or small children. It has proven true for Diana Lam, the schools' former Deputy Chancellor.

All eyes are now on the 15,000 third graders potentially affected by Mayor Michael Bloomberg's harsh new school promotion policy. Who can think about Lam, forced out after a report raised questions about her role in helping her husband find work in the schools, when eight-year-olds are in the klieg lights?

But let's not forget the results Lam promised from the so-called uniform citywide curriculum, of which she was chief architect. It didn't stop 31,700 third graders from falling into the "promotions-in-doubt" category in January.

Now crates of supplementary reading materials are being shipped to elementary classes for last-minute "interventions," that the mayor says will shore up many of the stragglers. This is the latest of the one-size-fits-all prescriptions that are supposed to work uniformly well at every school in the city.

In reality, some city schools have the teachers, space and professionalism to use the materials with small groups of no more than seven struggling readers, as their publisher directs. Other schools are understaffed and overwhelmed, still puzzling over Lam's curriculum as they take delivery of the newest magic bullets.

Bloomberg tacitly recognized these shortcomings recently, when he demanded that the state boost its educational aid to the city by $5.3 billion a year in order to halt the "neglect of our public school children." But, wasting no time waiting for a check that may never arrive, he expects immediate great strides from the victims of that neglect.

At one well-run school, PS 69 in Jackson Heights, counselors were dispatched to third grade classrooms after Mayor Michael Bloomberg replaced dissidents on the Panel for Educational Policy to ram through passage of his new promotion policy. Children scoring far below grade level on reading or math tests later this month will get another chance after summer school – make that "Summer Success Academy" -- or it's back to third grade with them.

It's good to know that schools offer counseling, though headed by a mayor who trivialized children's feelings in saying that those failing on the basis of a single test might "cry a bit."

That tough-love stance dates back to his mayoral campaign, when he said failing kids must learn that free weekends and vacations are a "luxury earned."

It's a lesson they're learning now. As a million other New York schoolchildren enjoy spring break, third graders shaky in reading or math stay in school, attending hastily organized academic boot camps.

However, the mayor softened toward them, even turning spiritual, in his March 19 radio show as he described the child who scores at Level 1 on standardized tests – the bottom rung and now a dealbreaker for promotion -- as a "child who's been dealt a bad hand by God."

Certainly some city schoolchildren have been dealt worse schools than others. A flip through the school report cards on the Department of Education web site shows that Level 1 scores are a rare occurrence at some elementary schools but routine at others.

That brings us back to Diana Lam. If her ouster is all but forgotten, how about the uniform citywide curriculum, which was supposed to help all schools teach children to read?

Even supporters of the approach say it isn't designed for the struggling reader. The method, in use in K-8 classrooms around the city, allots large blocks of time to "independent reading" of classic and popular children's books. The teacher presents a quick lesson on a topic that, according to standards set by the state, competent readers need to know. In lower grades, this might be identifying main ideas or discussing characters.

Classroom libraries are sorted by reading level, and children are told to read a book coded for their skill levels and respond to the topic under discussion. Thus, a skilled third grade reader might be reading something like "Harry Potter" for characterization while a struggling classmate supposedly does likewise with a book on the first-grade level, something with short chapters and lots of pictures.

Or perhaps, as a teacher in one Crown Heights elementary school packed with Level 1's suggested, "It works best with children who can already read. The kids who can't read are just turning the pages."

Students in grades three to eight also get a daily half-hour dose of word study, the more traditional fare of reading classes, including phonics, vocabulary and spelling. Here Lam's suggestions stirred the most controversy. She had recommended a program called Month by Month phonics, which has materials for various skill levels and intended for use by an entire class.

Then President George W. Bush's top reading adviser threatened New York City with the loss of millions in federal education funds on the grounds that Lam's choice, Month by Month, wasn't backed by research. Overruling his deputy, Schools Chancellor Joel Klein ordered a federally sanctioned phonics program, Voyager Passport, developed especially for New York City by Voyager Expanded Learning of Dallas.

Now Bloomberg says the city will spend $41 million in immediate aid to flailing third graders.

Apparently, a substantial chunk of that is heading to the company in Dallas. Defending his boss's policy, Chancellor Klein told WNYC's Brian Lehrer that Voyager Passport was coming to the rescue. The materials also figure prominently in the Department of Education's 20-page paper on the new promotion policy [In PDF Format] , where they're described as the most common "intervention" now used in the city's third grades for substandard readers.

Yet right now, as fresh shipments of Voyager Passport materials arrive in third grade classrooms, it's still unclear how effective they are. As the Daily News has suggested, their GOP stamp of approval might have more to do with pedigree than pedagogy. Their executives include Jim Nelson, who served as the Texas education commissioner under Gov. George W. Bush.

For Voyager Expanded Learning, founded in 1996, the Passport product is a first foray into materials aimed solely at struggling learners. It has provided summer school materials here for years, said a spokeswoman, Deborah Nugent, but those products were for use by an entire class. She said Passport is meant for use with about a half-dozen students, either pulled out and taken to a supplementary instructor, or working with their teacher in a classroom where other children are busy with separate tasks.

Certain schools here are awash in student teachers, parent volunteers and artists-in-residence who can make such things happen. Others are lurching from day to day.

The Department of Education asked 600 schools what support they provide for their weakest third graders, and 97.5 percent said they use some kind of small group instruction. But that could mean anything at all, including letting some kids gather around a computer; it is hard to imagine a school saying it gave no help at all.


A smaller portion, 72 percent, said they've been using Voyager Passport materials. Those were shipped in September, and there are "early indications of promising results," says the education department's promotion policy paper, because of 16,978 third enrolled in Voyager Passport programs "almost 30 percent (29.6 percent) did not receive Promotion-in-Doubt letters."

Which leaves 70 percent that did.

Still, the assistant principal of a Long Island City elementary school said she was happy to see the new Passport shipments. She pointed out that the promotion policy – far from being the take-no-prisoners document that the public imagines – will allow students to be promoted despite Level 1 scores if the school can prove every possible intervention has been tried and will continue.

"After all," said the assistant principal, "we can't have fifth graders who shave."

Marcia Biederman writes about city public schools for InsideSchools.org and contributes regularly to The New York Times.