AI models struggle with facts when given conflicting external data

Image credit: Search Engine Journal

New research indicates large language models face significant challenges retaining accurate information when presented with contradictory data from external tools, impacting AI reliability and transparency.

These findings highlight a critical hurdle in developing trustworthy artificial intelligence systems, particularly concerning their ability to discern and prioritize correct facts over conflicting input, according to recent papers published on arXiv.

One study, dubbed MemToC, investigated scenarios where a language model’s internally correct answer clashed with incorrect information provided by an external tool. The research found models exhibited low retention of their own correct answers, ranging from 6.5 percent to 17.1 percent.

Across 120 instances where models received incorrect information from external tools, none of the five models tested explicitly acknowledged the disagreement, according to the MemToC study. This lack of explicit conflict flagging makes it difficult to ascertain if models are even capable of identifying such discrepancies.

Another paper, titled ‘Empty Shelves or Lost Keys?’, differentiated between a model’s ability to encode a fact with strong contextual cues and its capacity for reliable recall across varied question formats. While models like GPT-5 and Gemini-3 showed high encoding rates of 95-98 percent, their reliable recall proved significantly weaker.

Further research, ‘From Parameters to Answers,’ explored the internal mechanisms of models, specifically regarding country-continent questions. This analysis suggested conclusions about how models retrieve facts are heavily dependent on the specific signal measured and how it changes, indicating no universal internal diagram for factual retrieval.

Google AI researchers noted the inherent difficulty in explaining the behavior of AI models, particularly when they encounter conflicting information or are missing crucial facts. This complexity extends beyond simple performance metrics, demanding a deeper understanding of their internal processes.

The collective research underscores a growing need for more sophisticated methods to understand and improve how AI models manage conflicting data and ensure factual accuracy, moving beyond superficial evaluations to address the core challenges of AI visibility and explainability.

Source: Search Engine Journal