TEA wants full list of suspect schools
But expanded inquiry into TAKS cheating not certain, state says
11:54 PM CDT on Tuesday, July 25, 2006
By HOLLY K. HACKER and JOSHUA BENTON / The Dallas Morning News

LINK

Reversing course, the Texas Education Agency said Tuesday that it wants a complete list of schools with suspicious scores on last year's state exams. But officials made no promises to investigate those additional campuses.

Officials said Tuesday they have asked for the names of all schools that were flagged as suspect by Caveon, a Utah test-security company. The agency hired Caveon to look for evidence of possible cheating on the 2005 Texas Assessment of Knowledge and Skills.

On Sunday, The Dallas Morning News reported that the TEA's list of suspected cheaters left off at least 167 schools that Caveon had flagged. Neither the TEA nor the schools knew which campuses they were.

Last week, agency officials said they did not ask Caveon for the names of the additional schools because they did not consider them worthy of investigation. That's because Caveon used a different type of analysis to identify the additional schools.

"I think that over the weekend, people thought about the situation and just realized we need the complete list," said Debbie Graves Ratcliffe, a TEA spokeswoman.

"And whether we take further action – we'll have to decide once we see that list."

Schools' needs
Jim Nelson, outgoing superintendent of the Richardson Independent School District, said schools need to know whether they're on Caveon's list. They also need to understand why they were flagged as having potential problems.

"I think if we're going to be able to conduct investigations locally, we have to have all the information used to make their findings," Mr. Nelson said.

He said more data would be particularly important for the difficult task of investigating alleged cheating from over a year ago. "The only way you can check is to have as much data as you can," Mr. Nelson said. "If you're looking for specific acts from that long ago, I think it's going to be very difficult."

The analysis
In its analysis, Caveon looked for unusual test scores at the classroom and school levels. While schoolwide anomalies involve larger numbers of students, they can also be triggered by a smaller percentage of students with suspect scores.

Robert Scott, the TEA's chief deputy commissioner, said he believed the schoolwide problems were less egregious than the classroom-level anomalies Caveon found.

"It's like looking for a problem citywide or doing it by looking in a specific ZIP code," he said. "The classroom-based list is more focused."

Mr. Scott said the agency is determining how best to investigate some of the schools Caveon identified. A plan should be announced by the end of this week, he said.

State incentives
The state's first priority will be 14 schools that are supposed to receive $60,000 to $220,000 each for improved test scores under a new incentive plan of Gov. Rick Perry's. Other schools will also be investigated, although the agency has not announced how many.

The agency still has not requested detailed information on what suspicious behavior Caveon found in each of the schools it flagged. "I don't think we're far enough in our investigation to warrant that," Mr. Scott said.

Caveon's two analyses found 609 schools and 702 classrooms with unusual scores. State officials originally did not plan to identify the schools or tell them they had suspicious scores.

Shifting response
But the TEA released a list in early June, after The News and other newspapers requested it. That list identified only schools with suspicious classroom scores.

Once the list came out, the agency said it planned only a limited investigation into some of the flagged schools where other testing violations had already been reported.

But after media reported the TEA's plan, the agency announced it would do a more thorough investigation.

TEA officials said Tuesday that the search for score anomalies is a new kind of analysis for the agency, so they're learning along the way.

"We are having to feel our way along on this," Ms. Ratcliffe said. "The more we think about it, the more questions that arise."

Not all suspect schools on TAKS list
Exclusive: Firm flagged 167 more, but state not looking into it
12:16 AM CDT on Sunday, July 23, 2006
By JOSHUA BENTON and HOLLY K. HACKER / The Dallas Morning News

LINK

The list of schools suspected of cheating is longer than Texas education officials have reported – and those officials say they aren't interested in tracking down the latest suspects.

A Dallas Morning News analysis has found that at least 167 unidentified schools were flagged as potential cheaters by Caveon, the company Texas hired to hunt for TAKS cheaters. That's in addition to the 442 schools named by state officials. None of the other schools have been notified that they are on the list.

Texas Education Agency officials say they don't know which schools they are – and they have no plans to find out.

"The only list of schools we have is the list that has been made public," said TEA spokeswoman Suzanne Marchman. "That's the list we plan to work with."

Superintendents with schools that have been named have complained that the TEA hasn't given them all the information they need to investigate Caveon's findings. But at least they know their scores are suspicious.

"That is so grossly unfair," said Gayle Fallon, president of the Houston Federation of Teachers. "If you're going to accuse someone of cheating, look them in the eye and do it."

Caveon, a Utah-based data-analysis company, was hired by Texas officials last year to examine the students' 2005 scores on the Texas Assessment of Knowledge and Skills.

That followed a series of stories by The News that found evidence of cheating on the TAKS in schools throughout the state.

The News discovered the missing schools when analyzing data in one of the appendixes of Caveon's May report to the TEA. Caveon withholds many of the details of how it performs its analysis, citing proprietary reasons. But in the appendix, the company outlines what it found in one high school where it suspects cheating on the math TAKS.

The report doesn't identify the school by name and lists its students only as anonymous ID numbers. But The News was able to determine that the school is Westbury High School in Houston by matching the student scores to state data. Westbury is the only school in Texas that had student scores matching the data in the appendix.

Houston school officials declined to comment, as did a Caveon spokesman.

Westbury had 1,431 students take the math TAKS in 2005. Of those, 185 had answer sheets Caveon considered suspiciously similar to at least one other student's. Caveon said the chance of that happening at random was less than 1 in 4 million million billion billion billion. That's a 1 with 40 zeroes after it.

The analysis also found several groups of Westbury students who had identical answer sheets – even getting all their unlikely wrong answers wrong in exactly the same way. Caveon also found an unusually high level of erasures at Westbury.

One Westbury junior – known only as No. 3561511 in the report – made a seemingly miraculous gain. As a sophomore, she performed very poorly on the math TAKS, outscoring only about 20 percent of the state's test takers. But as a junior, her score zoomed up – beating that of about 73 percent of Texas students.

No. 3561511 was helped by the 23 wrong answers on her answer sheet that were erased and replaced with correct answers. She ended up with an answer sheet identical to those of three of her peers and almost identical to three others.

In its report, Caveon is confident that there was wrongdoing at Westbury. "It appears likely that there are instances of testing irregularities at this school," it states. Caveon also writes that of all the Texas schools where it found math irregularities, Westbury was the seventh-most suspicious.

But the TEA never told Houston officials that Westbury's math scores were suspicious. That's because TEA officials didn't know themselves.

Dual analyses

The core of the confusion is that Caveon actually performed two different but complementary analyses of the state's test scores. One looked for suspicious test scores in each classroom. The other looked for problems throughout a school.

Both analyses examined the same scores. But they had different standards for how much suspicious activity it took for a classroom or school to be flagged.

Because classrooms have fewer students than whole schools, it takes a higher incidence of suspicious activity for a classroom to be flagged than for an entire school.

For example, imagine a classroom with only 10 students. If two of those students had scores Caveon considered suspect – 20 percent of the total – that probably wouldn't be enough for the classroom to be flagged as suspicious. That's because, when dealing with such small numbers, two strange test scores could result from random chance or "noise" in the data.

But imagine a school with 1,000 students. If 200 of those students had suspicious scores – still 20 percent of the total – that would in many cases be enough for Caveon to declare the school's performance suspect. Statistically, it is less likely that 200 strange scores would be attributable to chance.

As a result, Caveon was more likely to flag an entire school with strange scores than a classroom. Of the 73,793 classrooms whose scores it analyzed, the company flagged 702 – about 1 percent. But Caveon flagged 609 of the 7,112 schools it analyzed – more than 8 percent.

Short list explained

The problem is that the list of 442 suspect schools that the TEA distributed to districts includes only the schools that had classrooms flagged – not those flagged as an entire campus.

There is probably some overlap between the two lists. For example, in the case of Westbury High School, Houston officials have been told that there was potential cheating on the science and social studies tests in 11th-grade classrooms. But the TEA never informed them about the problems Caveon found schoolwide in the math test results.

At a minimum, 167 schools were flagged by Caveon as possible cheaters and still have no idea. According to the Caveon report, the number of those schools could be as high as 394.

Lisa Chandler, the state's director of assessment, responded to questions about the missing list via e-mail. She said the state didn't obtain the list of schools Caveon considered suspicious because "the list based on the classrooms seemed to be the most useful for districts to use in following up the results."

She cites a section of the Caveon report that suggests the suspicious classroom scores may be a good place to begin investigations. "That is, those schools where exceptions were detected in multiple classrooms might be investigated first," the report states.

But the TEA has already committed to investigate a number of schools regardless of how many classrooms were flagged. Last month, it announced plans to investigate 14 schools on the Caveon list that were also due cash bonuses from the state for their outstanding test scores.

The agency doesn't know whether any other schools on the bonus list might have been flagged by Caveon's schoolwide analysis. And it has no plans to find out which schools are on the list or how egregious their possible cheating might have been.

That means that even schools with what Caveon considers highly suspicious scores won't be identified or investigated.

"The schools that TEA intends to address would be the ones where classrooms were flagged in the list that's been provided," Ms. Marchman said.

E-mail jbenton@dallasnews.com
and hhacker@dallasnews.com

State seems to have had right to see cheat data
Contract with Caveon appears to grant info superintendents want
11:48 PM CDT on Saturday, July 22, 2006
By JOSHUA BENTON / The Dallas Morning News

LINK

Some Texas superintendents have complained that state officials haven't given them enough information to investigate their schools' appearance on Caveon's list of schools with suspicious TAKS scores.

State officials say that's because they don't have the information themselves.

But Caveon's contract to analyze Texas' test scores seems to give state officials substantial access to the data necessary to investigate potential cheating.

"Consultant agrees that all Works are, upon creation, works made for hire and the sole property of TEA," states the contract, obtained by The Dallas Morning News. "Consultant hereby assigns to TEA all worldwide ownership rights, including the Intellectual Property Rights, in the Works, without the necessity of any further consideration."

Caveon has said its methods and algorithms are proprietary and can't be shared with the TEA or the public. But the contract pledges to provide the TEA with substantially more data than Caveon has apparently turned over.

It promises "summary and detailed results" from its analysis, including detailed information on "the incidence of test fraud/theft by classroom and school" and "cheating and piracy activity in individual examinees."

When asked about the contract language, the state's director of assessment, Lisa Chandler, responded via e-mail. She referred The News to a legal filing by attorneys for Pearson – the company that runs Texas' testing programs – in an open-records dispute before the state attorney general's office. That document argues that the "TEA has no carte blanche right to the disputed information." It is unclear whether the TEA agrees with that assessment.

To date, the TEA has been provided with only a partial list of the classrooms and schools where Caveon found potential problems. The agency has no detailed information on what, exactly, was suspicious about those classrooms. It also has no data on individual students.

But it may be hard to obtain that information now. Caveon's contract to analyze the state's test scores expired in February, according to documents. Earlier this month, a Caveon vice president said his company had completed all of its obligations to the TEA and that, should the agency want more data, a new contract would likely be necessary.

In the version of the contract released to The News, the cost of Caveon's analysis is redacted. A note from Pearson states that the "Payment Terms are Confidential and Proprietary to NCS Pearson, Inc."

The TEA previously stated that it paid more than $500,000 for Caveon's analysis. At this point, spokeswoman Suzanne Marchman said, it is unclear how much of the agency's money was spent on it.

E-mail jbenton@dallasnews.com

Texas Education Agency Hires Contractor Caveon to Find TAKS Cheaters- Caveon Won't Tell Them Who the Schools Are
Caveon
2006 State Education Test Security Survey Results
Tracking down cheaters