Commonwealth Short Story Prize: Uncertainty after AI doubts about winning text

Following the announcement of the regional winners of the Commonwealth Short Story Prize, readers have expressed suspicions that one of the stories was generated, at least in part, by artificial intelligence (AI). The British literary publisher Granta published the text “The Serpent in the Grove” by Trinidadian author Jamir Nazir online, which led to discussions on social media about typical AI phrasing. The incident now confronts the jury and publishers with the question of how to deal with potential AI texts.

The Commonwealth Foundation awards the literary prize annually to promote unknown authors worldwide. The literary magazine Granta, which traditionally publishes works by renowned writers, presents the winning texts on its website. In a statement, however, the publisher clarified that its editors are not involved in the jury's selection or the determination of the winning stories.

Readers recognized various text structures in the story that frequently occur with machine language models. For example, users on social networks noticed an excessive use of comparisons and unusual idioms. Additionally, the text contained so-called negative parallelisms in the pattern “not X, but Y,” which are considered typical stylistic features of AI-generated texts.

Publisher questions AI model

Sigrid Rausing, editor of Granta, confronted the language model Claude from the tech company Anthropic with the questionable short story after the accusations became known. The model concluded that the text was “almost certainly not produced unaided by a human” , but it did identify core passages that were presumably written by humans. Rausing pointed out the irony that an artificial intelligence itself is the most efficient tool for detecting machine-generated texts.

The director of the Commonwealth Foundation, Razmi Farook, defended the foundation's selection process to The New York Times (NYT). She warned organizations against responding with “a kneejerk reaction to the general hysteria around these issues ”. The foundation must also fulfill its duty of care to the author, who has so far been little published, for whom unfounded accusations could have serious personal consequences.

Limits of detection software

Some readers attempted to substantiate their suspicions with detection software like Pangram, which reportedly classified the text as entirely machine-generated. However, The New York Times quotes Nicholas Andrews, a computer scientist at Johns Hopkins University, saying, “AI text detectors make lots of mistakes, particularly on creative writing that uses unusual constructions” . These literary peculiarities are often not included in the training data of detection systems, he says.

Jack Grieve, a linguistics professor at the University of Birmingham, also emphasized the risks of such programs to the NYT. Without considering variations in dialect, topic, or genre, the analysis results are unreliable. After reading the short story himself, he found it neither obviously AI-generated nor obviously human-written.

The problem of machine text detection has been occupying the AI industry for some time and goes beyond this individual case. Major providers like OpenAI have had to discontinue their AI detectors due to inaccurate results in the past. Until the Commonwealth Foundation makes an official decision, Granta will leave the short story online with an appropriate editorial note.

(wpl)