BBC warns of erroneous AI news summaries

AI tools such as ChatGPT or Copilot can make life easier, for example by creating summaries of messages. However, the BBC warns that the results should be treated with caution.

For a study, the BBC had the ChatGPT, Copilot, Gemini and Perplexity AI systems evaluate messages from its website. The systems were then asked questions. Journalists who were familiar with the respective topic then evaluated the answers.

According to the BBC, more than half of the answers (51 percent) showed "significant problems of some kind". The study authors found factual errors, such as incorrect factual claims, figures and data, in one in five responses (19 percent) that referred to the BBC.

Sunak and Sturgeon are still in office

These included, for example, that both ChatGPT and Copilot incorrectly believed that former Prime Minister Rishi Sunak and former Scottish Government leader Nicola Sturgeon were still in office. Gemini made the false statement that the NHS does not recommend vaping as a smoking cessation aid.

The AI systems performed differently in the study: According to the BBC, Microsoft's Copilot system and Gemini from Google delivered more errors than ChatGPT from OpenAI or Perplexity.

"AI assistants cannot currently be relied upon to deliver accurate messages, and there is a risk that they will mislead audiences," the authors write. "The extent and scope of errors and biases in trustworthy content are unknown." However, the study can "only scratch the surface of the problem".

The BBC normally excludes AI chatbots from its website. It had lifted the exclusion for the duration of the study.

In a blog post, Deborah Turness, head of BBC News and Current Affairs, called on providers such as Apple to switch off their AI-generated news summaries. At the end of last year , the BBC complained to Apple about a false report from its Apple Intelligence service. Apple has since deactivated the summaries.

"We live in troubled times," Turness writes, "and how long will it be before an AI-distorted headline causes damage in the real world?"

(wpl)