BBC: “Largest Study of Its Kind Shows AI Assistants Misrepresent News Content 45% of the Time – Regardless of Language or Territory”

From the BBC:

New research coordinated by the European Broadcasting Union (EBU) and led by the BBC has found that AI assistants – already a daily information gateway for millions of people – routinely misrepresent news content no matter which language, territory, or AI platform is tested.

The intensive international study of unprecedented scope and scale was launched at the EBU News Assembly, in Naples. Involving 22 public service media (PSM) organizations in 18 countries working in 14 languages, it identified multiple systemic issues across four leading AI tools.

[Clip]

Professional journalists from participating PSM evaluated more than 3,000 responses from ChatGPT, Copilot, Gemini, and Perplexity against key criteria, including accuracy, sourcing, distinguishing opinion from fact, and providing context.

Key findings:

• 45% of all AI answers had at least one significant issue.
• 31% of responses showed serious sourcing problems – missing, misleading, or incorrect attributions.
• 20% contained major accuracy issues, including hallucinated details and outdated information.
• Gemini performed worst with significant issues in 76% of responses, more than double the other assistants, largely due to its poor sourcing performance.
• Comparison between the BBC’s results earlier this year and this study show some improvements but still high levels of errors.

The research team have also released a News Integrity in AI Assistants Toolkit, to help develop solutions to the issues uncovered in the report. It includes improving AI assistant responses and media literacy among users. Building on the extensive insights and examples identified in the current research, the Toolkit addresses two main questions: “What makes a good AI assistant response to a news question?” and “What are the problems that need to be fixed?”

In addition, the EBU and its Members are pressing EU and national regulators to enforce existing laws on information integrity, digital services, and media pluralism. And they stress that ongoing independent monitoring of AI assistants is essential, given the fast pace of AI development, and are seeking options for continuing the research on a rolling basis.

Read the Complete Release

Resources

+ Complete Report: News Integrity in AI Assistants An International PSM Study

+ News Integrity in AI Assistants Toolkit

See Also

+ Coverage From DW: AI Chatbots Fail at Accurate News, Major Study Reveals

+ Audience Use and Perceptions of AI Assistants for News (via BBC)

Filed under: Associations and Organizations, News, Patrons and Users