More Reasons for Value-Added Assessments
By Lance T. Izumi, SacUnion, August 30, 2004

LINK

Two recent events underscore the importance of improving the way we measure student achievement. The first was a front-page New York Times story that cited an American Federation of Teachers (AFT) study that said that charter school students often performed worse than comparable students in regular public schools. The second was the release of California's 2004 student test scores.

Charter schools are deregulated public schools that promise better results in exchange for their greater freedom from bureaucratic red tape. There are more than 400 such schools in California. Because charters emphasize higher student achievement, the AFT finding that charter-school fourth graders performed half a year behind other public-school students on national reading and math tests is disturbing. The AFT study, though, has a crucial Achilles heel.

The study's conclusions were based on test results from a single year. In other words, it was a snapshot of student performance at one point in time. This static picture fails to show any achievement growth trends. This is especially important given that many charter-school students performed poorly at their previous schools and thus start at a very low level of achievement. A charter school may improve a student's performance, but that improvement won't be picked up if only a snapshot is used. The same problem was also on display in the release of California's 2004 test scores.

The test scores showed that only 30 percent of third graders were proficient in English language arts. However, 45 percent of fifth graders were proficient in English. The Los Angeles Times reported that, "Educators were at a loss to explain the robust fifth-grade results, saying they expected the second and third grades to do better because of smaller class sizes and easier material to master." This befuddlement arises because a static picture is being used to compare the wrong things.

Instead of comparing today's third-grade scores with today's fifth-grade scores, which compares the performance of different groups of students, the scores of individual fifth graders should be compared with their own third-grade scores. That comparison would allow policymakers to see whether a student was growing in achievement.

Because of the snapshot problem, California should adopt a value-added assessment system that collects individual student test results over time and uses the results to determine how much a school or even an individual teacher has contributed to the improvement or decline in a student's performance. The state has taken a first step in this direction by assigning identification numbers to students so their performance can be tracked.

The Pacific Research Institute recently proposed a new value-added model that calculates a rate of expected academic change, or REACH, using an individual student's test scores to come up with an annual individual improvement target for that student. In other words, given a student's current location on the ability scale, the REACH model tells teachers, principals, parents and officials how much a student's achievement needs to grow each year in order to be proficient in a subject area by the time each leaves school. Schools can then be judged on whether students hit those targets or not.

If the top goal of our education system is to improve student achievement, then we need to use a measurement device that gauges that improvement. It's time to create a value-added assessment system.

A REACH score (Rate of Expected Academic Change) enables educators and policymakers to see the rate at which individual students must progress in order to reach an state standards by their last year in school.

LINK

Using this indicator, schools and teachers can be judged on the basis of not just whether they are making an above or below average rate of progress but on whether their students are making the progress necessary to reach an acceptable minimum outcome.

Darling-Hammond comments on how requiring schools to meet multiple targets for each student subgroup results in a "statistical gauntlet that penalizes schools serving the most diverse populations".
Critics Float 'No Child' Revisions
By Lynn Olson, Education Week, August 11, 2004

LINK

Critics of the No Child Left Behind Act have begun to circulate proposals for fixing what they view as major flaws in its accountability provisions.

Although the law is not slated for reauthorization until 2007, they are hoping for amendments as early as next year, in part to address the large number of schools and districts that may not meet its performance targets.

Many of the proposed changes were discussed at a recent meeting here sponsored by the Center on Education Policy, a Washington think tank. "We want to push the debate beyond criticism to discussing alternatives," said Jack Jennings, the group's director and a former top Democratic congressional aide on education, "and to subject those alternatives to criticism."

But Ross Wiener, the policy director for the Washington-based Education Trust, which strongly supports the law, maintained that the papers "presented no compelling evidence" that its accountability requirements need big changes. In large measure, Mr. Wiener said, he came to that conclusion because many of the projections that large number of schools will be identified for improvement in the future presume that schools will perform as they always have.

In general, the alternatives put forward focus on addressing what many describe as the law's unrealistic expectations for schools. A number of models also would give schools credit for growth, rather than just the percent of students at or above the "proficient" level on state tests. Some also urge the federal government to move beyond a concentration on test scores to include other measures of school and student performance.

The federal law, a reauthorization of the Elementary and Secondary Education Act, requires that 100 percent of students score at state-defined proficiency levels on reading and math tests by 2013-14. To make "adequate yearly progress," or AYP, schools and districts must meet annual targets for the percent of students at or above the proficient level, with those targets rising over time. They also must test at least 95 percent of their students and do well on at least one other academic indicator-graduation rates for high schools and, typically, attendance rates at the elementary and middle levels.

Schools and districts must meet the annual targets both for their total student populations and for subgroups of students who are poor, speak limited English, have disabilities, or come from racial- or ethnic-minority backgrounds.

Critics have charged that the rate of improvement expected of schools is unreasonable and sets them up for failure, particularly given the large number of subgroup targets. Projections from states such as California and Connecticut, for instance, indicate that, eventually, most schools will fail to make AYP unless the targets are changed. State definitions of "proficient" also vary widely, making it hard to compare results across states.

Rethinking Targets
"The most important modification is to set performance targets for judging adequate yearly progress that are more reasonable and for which there is a realistic hope that they might be achieved given sufficient effort," argues Robert L. Linn, a professor of education at the University of Colorado at Boulder, in a paper presented at the Center on Education Policy meeting, held July 28.

One alternative, he suggested, is to use the median score on state tests in 2002-the year President Bush signed the measure into law-as a state's baseline. The expected annual increase for schools could be based on the rate of improvement in the state's best-performing schools averaged over the past five years. Some experts countered that assuming schools can make only the same rate of improvement that they've made in the past presumes they won't do anything differently. But Mr. Linn said evidence would have to be available that the amount of change expected was possible.

W. James Popham, the author of America's 'Failing' Schools: How Parents and Teachers Can Cope With No Child Left Behind, also advocates abandoning the use of "proficiency" as a target. Instead, he suggests, states should set more realistic "grade level" expectations for students. Schools and districts would then have to make annual improvements in the percent of students at or above grade level, but those targets would vary based on where the school started, and they would "reek of realism," he said in an interview.

For the system to work, he added, states would need to select tests that were more sensitive to instruction; for example, by focusing on fewer and more fundamental content standards and reporting results in ways that inform classroom practice.

Growth and Absolute Goals
Detractors of the law also suggest that focusing only on the percent of students at or above the proficient level ignores the growth of students well below or above that bar. For that reason, some states, such as Massachusetts and New York, are using a performance index to evaluate schools that gives them credit for moving student achievement up, even when youngsters do not yet score at the proficient level.

A number of the alternatives recommend taking that method one step further by tracking the growth of individual students over time.

A "hybrid success model," proposed by the Northwest Evaluation Association, based in Portland, Ore., would identify an annual-growth target for each student in a school based on where the student began, so that he or she reached proficiency within a specified time. Schools would be judged by the extent to which each child met his or her growth target. Schools in which students continued to progress after reaching the proficient level would be rewarded.

Similarly, Harold C. Doran, a senior research scientist at the Washington-based American Institutes of Research, has suggested a model known as REACH, for Rate of Expected Academic Change, that would judge schools based on whether individual students were making enough progress annually to reach the proficient level on state tests by the time they finished the highest grade in a school.

"The addition of an individual-growth approach to current AYP models is the important factor," argued Allan Olson, the president of the Northwest association, a nonprofit group that provides assessment services to districts, "not the particular approach that is added."

In a March letter to U.S. Secretary of Education Rod Paige, 16 chief state school officers championed the use of growth models to meet the law's AYP requirements.

But Lauress L. Wise, the president of the Human Resources Research Organization, in Alexandria, Va., cautioned that psychometric issues are involved in building such value-added systems. Linking results across grades is difficult, he said, because each grade tests and teaches a different set of knowledge and skills. Others pointed out that many states now lack the data systems and assessments needed to do such value-added analyses.

Gavin Payne, the chief deputy superintendent of the California education department, said one solution would be to empower states to devise their own accountability systems as long as they could show significant improvement in the percent of students at the proficient level and in closing achievement gaps between various subgroups. What constitutes "significant improvement" and "closing the gap" in a particular state should be informed by the best research in educational measurement, he contended, not by federal mandate.

"What has occurred is the de facto federal pre-emption of what were once state prerogatives in the area of educational accountability," Mr. Payne said.

But others cautioned that Congress passed the No Child Left Behind Act, in part, because states had been lax in their own accountability efforts under the previous reauthorization of the ESEA, passed in 1994.

"There's almost no reference to the massive failure of the states," said William L. Taylor, the chairman of the Washington-based Citizens' Commission on Civil Rights, arguing that the papers presented at the conference lacked historic context.

Statistical Hurdles
While agreeing on the importance of closing achievement gaps, critics charge that requiring schools to meet multiple targets for each subgroup results in a "statistical gauntlet that penalizes schools serving the most diverse populations," in the words of Linda Darling-Hammond, a professor of education at Stanford University.

Mr. Linn suggests that states could reduce the instability in disaggregated results by averaging scores across a number of years.

The National Education Association has proposed that schools be identified for improvement or corrective action only when the same subgroup fails to meet performance targets in the same subject for two or more consecutive years, a proposal that so far has been rejected by federal officials as contrary to the law. The union also would limit the provision of tutoring and school choice to the particular subgroup that fails to make adequate progress, rather than providing such options to all students in a school, as the law now requires.

Mr. Popham supports basing the status of schools and districts on the performance of their overall student populations, while continuing to report disaggregated results publicly. To ensure that educators maintain their focus on subgroup performance, he recommends that local citizen-review panels scrutinize test scores and other relevant data on each school and release public evaluation reports yearly.

But Mr. Wiener of the Education Trust contends that "accountability based on disaggregated data is here to stay." His organization has highlighted numerous examples of schools that appear to be successful based on schoolwide averages that mask the poor performance of individual subgroups.

The strongest critics of the No Child Left Behind Act are pushing to broaden the evaluation of schools far beyond test scores. The NEA, for example, wants to permit states to incorporate additional measures-such as the percent of students taking honors or Advanced Placement courses-into an accountability index to evaluate school performance.

And in a book published this summer, Many Children Left Behind: How the No Child Left Behind Act Is Damaging Our Children and Our Schools, Ms. Darling-Hammond and others outline alternatives to test-based accountability that rely on multiple measures.

For instance, Monty Neill, the executive director of the National Center for Fair & Open Testing, or FairTest, a watchdog group based in Cambridge, Mass., proposes that teachers gather evidence of students' progress in the classroom that could then be aggregated up to the school, district, and even state levels.

Limited standardized testing in reading and mathematics would be used solely to check up on school-level information and investigate discrepancies. In addition, each school would report annually to the public on a range of quantitative and qualitative data. On-site reviews of a school's teaching and learning environment every five years would supplement those efforts.

Accountable Knowledge: Innovation, Improvement, and the Quest For Professional Accountability