Being honest about generative AI

 The New York Times this weekend covers students 'fending off accusations of cheating' using AI  [gift link there]:

 In interviews, high school, college and graduate students described persistent anxiety about being accused of using A.I. on work they had completed themselves — and facing potentially devastating academic consequences.

In response, many students have imposed methods of self-surveillance that they say feel more like self-preservation. Some record their screens for hours at a time as they do their schoolwork. Others make a point of composing class papers using only word processors that track their keystrokes closely enough to produce a detailed edit history.

The next time Ms. Burrell had to submit an assignment for the class in which she had been accused of using A.I., she uploaded a 93-minute YouTube video documenting her writing process. It was annoying, she said, but necessary for her peace of mind.

Of course, as we learned in April, also in the Times: 

We have the integrity to note that what students are doing is cheating, but if teachers outsource their intellectual work--and yes, creating lesson plans and quizzes is intellectual work--then somehow that is not? 
Give me a break.

I appreciate the discussion that Marcus Luther at "The Broken Copier" had with his sophomores on this topic.

I spent part of last week with school business officials, which of course is another front in the "AI can do it!" battle. As I shared earlier this month, the latest testing has demonstrated no more than a 50% accuracy rate on finances.

The interview of Colin Fraser that Benjamin Riley shared on 'Cognitive Resonance' perhaps demonstrates why: 

...what you hope for is that the model is able to take a prompt such as “what is two plus two?” and find a symbolic representation which corresponds to two plus two, and then carry out the addition, two plus two equals four. The end. That’s what you hope happens.

But whether or not that is what’s happening is just a really complicated question. It’s very rare nowadays that a production model is not going to tell you two plus two is four, and in fact they can go pretty high in terms of addition.

Yet this actually breaks down once you get to a certain size of number. I've tested this out pretty extensively. Even the best production models start to break down around 40 digits of addition. And that’s a big number to add, for sure. But at the same time, if you know how to add, you can do it. Even a kid can add 40 digit numbers—it'll take them a long time, but they can do it.

If a model gives me an answer to a 40 digit sum and it’s wrong, that means it wasn’t adding.

When I tell you that I find the prospective of generative AI in school finance terrifying, I am not exaggerating.  

 

Remember the Atlanta cheating scandal?

NPR reporter Anya Kamenetz followed up to see what has happened with students.

more to read on the Atlanta verdict

You've no doubt caught that 11 of 12 Atlanta teachers and administrators were found guilty on various charges--mostly rackteering--around the cheating scandal. Note that the superintendent involved, Beverley Hall, died of cancer last month. Those found guilty, with the exception of one woman due to deliver a baby shortly, were immediately taken into custody.
Also note that larger questions of why teachers might find themselves in such a situation largely have not been part of the national conversation (possibly except for this New Yorker story from last July).

There's a whole lot going on here, so I'd recommend P.L. Thomas on the burden of the impossible, Jason Linkins about accountability in America, Richard Rothstein (likewise) on accountability but also numbers, Nancy Flanagan on power and control in test-cheating scandals, and José Vilson who pulls the identities of the teachers into the larger conversation around what our schools are facing.

In depth coverage of the Atlanta test cheating scandal

I'd highly recommend reading Rachel Aviv's in depth New Yorker article about the Atlanta cheating scandal. It has implications far, far beyond Atlanta:

To explain the improvement in scores, Hall told the investigators that “an effective teacher three years in a row will completely close the gap between a child born in poverty and a child born to a middle-income family.” This theory, in its earliest form, derives from a study by William L. Sanders, a statistician formerly at the University of Tennessee, but the findings, which have contributed to a nationwide effort to rate teachers rigorously, have been overstated to the point of becoming a myth. According to a recent statement by the American Statistical Association, most studies show that teachers account for between one and fourteen per cent of variability in test scores. 

John Ewing, who served as the executive director of the American Mathematical Society for fifteen years, told me that he is perplexed by educators’ “infatuation with data,” their faith that it is more authoritative than using their own judgment. He explains the problem in terms of Campbell’s law, a principle that describes the risks of using a single indicator to measure complex social phenomena: the greater the value placed on a quantitative measure, like test scores, the more likely it is that the people using it and the process it measures will be corrupted. “The end goal of education isn’t to get students to answer the right number of questions,” he said. “The goal is to have curious and creative students who can function in life.” In a 2011 paper in Notices of the American Mathematical Society, he warned that policymakers were using mathematics “to intimidate—to preëmpt debate about the goals of education and measures of success.”

Mark of a good leader

He checks the bathrooms.
Also:

When Dr. Hall was the superintendent, she covered one wall in her office with bar graphs showing the test results for all 100 city schools.

After Mr. Davis became superintendent, he took the test scores down and replaced them with large color photographs of Atlanta schoolchildren.

Remember the principal in Dallas

who pushed her teachers to only teach reading and math?
She was fired in Dallas. She's just been hired in D.C. 

Federal government seeking ways to stop cheating on standardized tests

The federal Department of Education is seeking information on preventing cheating on standardized tests.
From the Federal Register:

For these reasons, this RFI seeks solutions; advice; technical information; legal, regulatory, and policy approaches; and other input from the public regarding best practices for the prevention, detection, and investigation of alleged or actual testing irregularities. Through this RFI, the Department also seeks to gather information and suggestions for SEAs and LEAs on how they can address these issues.

(RFI= Request for Information; SEA=State Education Agency; LEA=Local Education Agency)
Comments are due February 16. The specific questions they'd like answered are here, though you do not have to answer any or all of them.  You can submit them electronically here, or send them in to:

Mr. Carlos Martinez
Office of Elementary and Secondary Education
Attention: Testing Integrity RFI
U.S. Department of Education
400 Maryland Avenue SW., room 3W104
Washington, DC 20202-6132

And do remember that public comments in response to an RFI become part of the public record, so make sure you write only what you'd be comfortable seeing in the public media (and speaking from personal experience, commenting on in the press).

Another one

The state investigation into Dougherty County, Georgia around cheating on exams is back. This is the same county that, after its own investigation, found no evidence of cheating. From the report (via Get Schooled):

Hundreds of school children were harmed by extensive cheating in the Dougherty County School System. In 11 schools, 18 educators admitted to cheating. We found cheating on the 2009 CRCT in all of the schools we examined. A total of 49 educators were involved in some form of misconduct or failure to perform their duty with regard to this test.
While we did not find that Superintendent Sally Whatley or her senior staff knew that crimes or other misconduct were occurring, they should have known and were ultimately responsible for accurately testing and assessing students in this system. In that duty, they failed. The 2009 erasure analysis, and other evidence, suggests that there were far more educators involved in cheating, but a fair analysis of the facts did not allow us to sufficiently establish the identity of every participant.
The statistics, and the individual student data, leave little room for any other reasonable explanation, save for cheating. For example, the percentage of flagged classrooms for DCSS is ten times higher than the state average.

A few things of note: the state investigation team was not from the education department. It was a former state attorney general, a former DA, and a special investigator. They were granted broad powers by the state. The intention of the state from the beginning was to make the results public (as was done in Atlanta), as well as hold accountable those adults who so remarkably failed in the public trust.
Not how it's been handled everywhere, but a credit to the state of Georgia.

Cheating in NYC

In the ongoing tracking of erasure/overcoaching/cheating scandals, here's the report from New York City.

See also this report on the possible motivation behind cheating.

reports coming in from Atlanta

..on their cheating scandal:

... the AJC is reporting that an educators’ ethics committee looking at the APS cheating scandal this morning voted to  yank the certification of 11 Atlanta teachers and administrators implicated in the cheating investigation. Final approval for the actions must come from the Professional Standards Commission.

And it'll be public when it does.

Who's cheating who?

Glen Ford with a report on who's cheating who in the high-stakes testing scandals.

As LA joins the cheating scandal list...

As Los Angeles joins the cheating scandal list, the Answer Sheet has a good summary of what's going on with cheating nationwide.

Get those graduation rates up! they were told

so they did.

I should point out that Worcester Public Schools do have credit recovery. From what I've seen, though, it's being carefully used.

two more states investigating cheating on state tests

Connecticut and New Jersey

I have no idea what this means

The big news here is that Bloomberg is now going after teacher tenure in the city entirely (speaking as a former teacher for whom the First Amendment was not going to be enough, yes, I see the value of tenure!), but the quote from the NYC Chancellor of Ed on whistleblowing is mystifying:

“I’m a person who wants to make sure we have a system that is honest, which it is,” he said. “And so there’s no public call where I want you to out your principal, I want you to out your assistant principal. I want to be very clear about that. It’s up to all of us to make sure that any cheating that’s taken place at any level, that we know about it because we will follow up.”

The best I can do is you don't need to call them because they're going to...magically, somehow...know? that something is wrong, and they'll fix it.
How...novel.

Philly Superintendent on her way out?

EdWeek reports that it appears that the superintendent of Philadelphia's public schools, Arlene Ackerman, may be on her way out.

With Philadelphia superintendent Arlene Ackerman facing a potential cheating scandal, a $629 million budget deficit, and evidence of contracts being steered toward unqualified companies, lawmakers in the city are reportedly trying to negotiate her departure.

Unfortunately, it may cost the district $1.5 million. (Not that keeping her in office is a bargain.)
Yes, there's a Broad Academy connection: she was Superintendent in residence before she came to Philly, and she is a past member of their board.

And speaking of cheating scandals...

There are 89 schools under investigation in Pennsylvania now.

"I ran out of synonyms for cheating"

Great commentary by Michael Winerip on the Atlanta Public Schools cheating scandal:

Investigators described how Dr. Hall had humiliated principals who didn’t reach their targets. Every year she gathered the entire district staff at the Georgia Dome. Those from schools with top scores were seated on the Dome floor; the better the scores, the closer they sat to Dr. Hall. Those with low scores were relegated to sitting in the stands.

Principals, in turn, humiliated teachers. At Fain Elementary, the principal, Marcus Stallworth, had teachers with low test scores crawl under a table, according to the report. At Parks Middle School, teachers who refused to join “changing parties” that were organized by the principal, Christopher Waller, to doctor answer sheets were isolated or let go, the report said.

It's Never Sunny...

sent in by an alert reader. Thanks!
Philadelphia also has a developing scandal around possible cheating on the 2009 PSSA. Philadelphia blog The Notebook reports that according to analysis by Data Recognition Corporation:

The analysis, prepared for the Pennsylvania Department of Education (PDE) in July 2009, highlights roughly 60 schools with suspicious results due to multiple statistical irregularities, including 22 District schools and six seven charters in Philadelphia...

Porter stressed that statistical analysis alone, without witnesses or confessions, cannot definitively prove that there was cheating. But he added that the report "describes a reasonable approach to identifying schools where there may have been cheating."
Nevertheless, it appears that the state never followed up with any further investigations. The forensic data analysis was discontinued in 2010, although PDE spokesperson Timothy Eller said it would be reinstated this year.

Both Philadelphia, where 44% of the schools of concern are, and Pennsylvania officials are now scrambling to figure out what happened to the report.
It appears that nothing was done with it at the time.
We can thus add Philly to the list of cities that are dealing with allegations or proven examples of widespread cheating on standardized tests.

Reputation follows you

Notes from Atlanta on administrators with new jobs in other districts and questions about their role in the cheating scandal back in Atlanta.