SINGAPORE — A new research paper found that OpenAI’s API and Google search results only partially matched the citations provided by ChatGPT’s web application, indicating significant discrepancies in source reproduction.
The study, conducted by researchers from the National University of Singapore and Xi’an Jiaotong University, revealed that a combined search using Google and OpenAI’s Responses API could only locate 26.2 percent of the specific pages cited by ChatGPT and 48.7 percent of the sites in tests involving GPT-5.5 Instant.
The research involved 80 questions, evenly split between English and Chinese, with each question run 20 times on both GPT-5.3 Instant and GPT-5.5 Instant models. This methodology aimed to assess the consistency and accuracy of citation retrieval.
A notable finding was that ChatGPT never used the exact wording of the original question when performing searches in visible runs of GPT-5.3 Instant and GPT-5.5 Instant. This behavior contrasts with 5.5 percent of GPT-5.2 Instant runs where the original query was used, according to the arXiv preprint.
For Chinese language questions, the study observed that both GPT-5.3 Instant and GPT-5.5 Instant models exclusively utilized English language searches 67.0 percent and 69.6 percent of the time, respectively. This suggests a potential limitation or bias in how the models process non-English queries for sourcing.
The researchers, who also collaborated with SerpAPI and Peec AI, recommended several measures to improve the reliability of AI-generated content. These suggestions include monitoring citations at both the webpage and site levels, running prompts multiple times to check for consistency, and accounting for instances where ChatGPT rewrites user prompts before conducting searches.
The paper, an arXiv preprint, underscores the need for ongoing scrutiny of AI model sourcing mechanisms as large language models become more integrated into information retrieval processes.
Source: Search Engine Journal
