Nov 16, 3:02 PM

AI Chatbots Fail Reliability Tests for Climate Policy Development

A study presented at COP31 reveals that artificial intelligence models abandon scientific facts when challenged by users.

By

AI Chatbots Fail Reliability Tests for Climate Policy Development

A joint study by UNESCO and University College London presented at the COP31 climate summit in Antalya, Türkiye, indicates that artificial intelligence chatbots abandon scientific facts when challenged by users. According to the report, the models consistently prioritize user agreement over factual accuracy, rendering them unreliable for shaping environmental regulations.

Researchers tested up to 33 leading AI models across 13 languages using 656 climate policy questions, which were verified against the En-ROADS simulation tool. While the systems answered roughly 90 percent of neutral questions correctly, they abandoned factual positions in eight out of ten instances when encountering mild user disagreement. During these tests, the chatbots confirmed false statements, including claims that carbon dioxide is not the primary driver of climate change.

The report highlighted significant performance failures in complex policy evaluations. When tasked with assessing how different environmental policies interact with one another, the systems achieved an average accuracy rate of 33.8 percent, barely exceeding the 33.3 percent expected from random guessing. When evaluating whether new evidence supported established climate science, accuracy averaged 75 percent. Leona Verdadero, UNESCO's lead author of the study, stated that models have a design flaw – they are incentivised to agree with their user. Senior author Maria Perez-Ortiz of University College London noted that the fluency of these systems creates a false sense of their underlying capabilities.

Accuracy rates also varied depending on the language used. Questions posed in English yielded an 88.6 percent accuracy rate, with similar results recorded in Spanish, French, and Portuguese. However, accuracy fell to 83.6 percent in Hindi and Bengali, 81.4 percent in Urdu, and 78.3 percent in Swahili. The authors noted that multilingual systems reflect the volume and quality of training data available in each language rather than reasoning before translating.

The findings were prepared as part of the Global Initiative for Information Integrity on Climate Change, launched in November 2024 by the United Nations, UNESCO, and Brazil. UNESCO reported that climate disinformation has shifted toward asserting that climate change is not caused by human activity or that mitigation measures are too costly, rather than outright denial.

Written by Martina Kirchner martina.kirchner@alpineweekly.com