A new global study by Pierre Sauvé found that brands known by large language models (LLMs) like ChatGPT and Gemini have a low correlation with those actually cited in AI-generated responses, challenging current brand visibility audit methods.
The research indicated a correlation of just 0.1 on a scale of 0 to 1 between a brand’s internal presence within a model’s training data and its likelihood of being cited, suggesting significant instability in how AI models reference brands.
Sauvé, from the firm Qwairy, distinguished between a brand’s ‘presence’ — what an AI model has learned and stored in its memory from training data — and its ‘citability’ — what the model provides through live web searches at the time of generating a response.
Repeated queries to OpenAI’s ChatGPT yielded inconsistent lists of brands, with only one brand, Symbial, appearing in two out of three responses, and no brand appearing in all three, according to the study.
The findings showed that on ChatGPT, 42 percent of brands were cited only once across three passages, while 37 percent appeared in all three. For Google’s Gemini, 55 percent of brands were seen only once.
Sauvé utilized log-probability (logprob) to measure a brand’s ‘presence’ within the models, focusing on whether the AI recognized a brand rather than its sentiment.
The study also noted that blocking GPTBot, OpenAI’s web crawler, does not prevent a brand from being included in a model’s training data if other websites mention it, but it does transfer control over the brand’s description to third parties.
Companies such as Moosend, Dealabs, Ouest-France, and Le Figaro were among those mentioned in the research concerning brand visibility.
Source: Abondance
