AI chatbots unable to accurately summarise news, BBC finds

A recent investigation by the BBC has revealed that four leading artificial intelligence chatbots are struggling to accurately summarize news articles. The chatbots in question are OpenAI's ChatGPT, Microsoft's Copilot, Google's Gemini, and Perplexity. The BBC provided these AI systems with content from its own website and subsequently posed questions regarding the news. The findings indicated that the responses generated by these chatbots contained numerous inaccuracies and misleading information. Deborah Turness, the CEO of BBC News and Current Affairs, emphasized that while AI technology presents 'endless opportunities', the companies developing these tools are 'playing with fire'. She raised concerns about the potential consequences of AI-generated headlines that could lead to significant harm in the real world. The BBC has reached out to the companies behind these chatbots for their comments on the study's results. In the course of the investigation, the BBC tasked the chatbots with summarizing 100 different news stories and evaluated the quality of their responses. Journalists with expertise in the relevant subjects were enlisted to assess the answers provided by the AI assistants. The results were alarming, revealing that 51% of all AI-generated responses contained significant issues. Furthermore, 19% of the answers that referenced BBC content included factual inaccuracies, such as incorrect statements, erroneous numbers, and wrong dates. In her blog, Ms. Turness expressed the BBC's desire to initiate a dialogue with AI technology providers to collaborate on finding solutions to these pressing issues. She urged tech companies to 'pull back' their AI news summaries, particularly in light of instances where Apple Intelligence misrepresented news stories. Some specific examples of inaccuracies identified by the BBC included: Gemini incorrectly asserting that the NHS does not recommend vaping as a method to quit smoking. ChatGPT and Copilot mistakenly claimed that Rishi Sunak and Nicola Sturgeon were still in office, despite the fact that they had already stepped down. Perplexity misquoted BBC News in a report about the Middle East, stating that Iran initially exhibited 'restraint' and labeled Israel's actions as 'aggressive'. Overall, Microsoft's Copilot and Google's Gemini were found to have more significant issues compared to OpenAI's ChatGPT and Perplexity, which counts Jeff Bezos among its investors. Typically, the BBC restricts its content from being accessed by AI chatbots, but it made an exception for this study in December 2024. The report highlighted that, in addition to factual inaccuracies, the chatbots struggled to distinguish between opinion and fact, often editorializing and failing to provide essential context. Pete Archer, the BBC's Programme Director for Generative AI, stated that publishers should maintain control over how their content is utilized and that AI companies must demonstrate how their assistants process news, along with the extent and nature of the errors and inaccuracies they produce.