WTF: AI avatar busted as "lawyer" in New York court

Artificial intelligence (AI) no longer stops at the justice sector. However, the introduction of automation technology into the legal world is not going smoothly. This was demonstrated on March 26 in the First Judicial Department of the Appellate Division of the Supreme Court of the State of New York. Under a light-flooded glass dome, a panel of judges wanted to hear a US citizen named Jerome Dewald. He is suing the insurance company MassMutual in an employment dispute over an alleged arbitration clause in his employment contract.

The plaintiff apparently wanted to save himself a lawyer and still make a lasting impression. This strategy only partially worked. "The appellant submitted a video for his argument," Judge Sallie Manzanet-Daniels said, according to a recording of the hearing published on YouTube, initially open in principle to an audiovisual interlude. "OK. We're going to listen to this video now."

But the show only lasts a few seconds. A smiling, almost suspiciously youthful-looking male figure appears on the screen with a well-fitting hairstyle, shirt and sweater over it. The figure greets the court politely and introduces himself as a "modest representative" on his own behalf "before a panel of five distinguished judges". But despite the flattery, the interruption comes at the drop of a hat, because the presiding judge smells the AI-generated roast. "Ah, ah," Manzanet-Daniels is briefly surprised at first. "Stop that!" she orders and asks: "Is that the lawyer for the case?"

Judges feel they are being led around by the nose

"I created that. That's not a real person," Dewald says, stuttering like a convicted schoolboy in a bad sketch. From then on, it's hard to miss: The complainant has sent an avatar into the race with the help of AI. "I don't appreciate being misled," the judge clarifies. He had to make it clear whether or not he was suffering from an illness and needed assistance. "Don't use the courtroom to present a business idea," Manzanet-Daniels tells the perplexed man. Then she gets loud: "Turn that off."

The judge gives the complainant five minutes to present his side of the story. The elderly gentleman searches for a solution on his smartphone. He keeps typing on his phone and stammering a few words. Eventually he catches himself and presents his case more or less fluently, without, of course, wowing the audience with rhetorical fireworks.

"They really beat me up"

One thing is clear: According to the current legal situation in Germany, an AI is not allowed to represent a citizen in court. The obligation to have a lawyer, a lack of qualifications and responsibilities, frequent hallucinations with generative chatbots and the need for personal interaction all speak against this. In the USA, the situation is more complex and still in flux. Generally speaking, however, AI is not currently allowed to speak for citizens in court without further ado. In many cases, technology already acts as a tool for human lawyers, but not as a replacement for them. In 2023, two US lawyers already had to pay a fine of 5,000 euros each in New York because they relied on fictitious precedents that ChatGPT had invented.

Dewald told the Associated Press that he had first applied to the court for permission to play a recorded video. He then used a solution from a San Francisco-based tech company to create the avatar. It was supposed to look like him, but he was unable to do so before the hearing. The complainant is now aware that he did not have a brilliant idea: "The court was very annoyed about it. They gave me quite a battering." A new verdict in his case was not issued by the end of the week.

(nen)