AI brand recommendations vanish after single buyer question: study

Image credit: Search Engine Journal

OSLO — Sixty-two percent of initial AI brand recommendations vanished after a single follow-up question, indicating significant instability in AI models’ responses to buyer specifics, a new study by Clovion AI reported Thursday.

The findings underscore the high responsiveness of AI to detailed inquiries and highlight a potential vulnerability for brands relying on AI for initial customer engagement, according to the research firm.

Clovion AI’s study, titled “Surviving the AI Funnel,” analyzed 69,120 multi-turn conversations across leading AI platforms including Claude, ChatGPT and Gemini.

These conversations spanned 36 categories within B2B software and fintech, examining how AI models adapt recommendations when buyer details, such as “for a small team,” were added.

Zahir Hasan, a researcher at Clovion AI, stated that the disappearance of initial recommendations after one clarifying question suggests that “AI models are highly sensitive to new information, which can drastically alter their output.”

The study also revealed significant discrepancies in factual claims made by different AI assistants regarding brand features.

Initially, the report cited 33 verified contradictions in brand facts between AI assistants, but this figure was later corrected to 330 due to a layout error, according to Clovion AI.

Corrected data showed Claude underclaimed brand features 160 times compared to 10 overclaims.

ChatGPT underclaimed 70 times with no overclaims, while Gemini overclaimed 80 times against 30 underclaims.

Frederick Vallaeys, another researcher involved in the study, theorized that Gemini’s tendency to overclaim might stem from its reliance on marketing materials.

Conversely, Claude and ChatGPT, which use documentation, tended to underclaim features not explicitly detailed in their training data, Vallaeys reported.

The researchers emphasized the importance of human oversight in verifying data, citing their own “dropped zero” error as an example of how a small mistake could significantly alter study conclusions and confidence levels.

The findings suggest that brands need to ensure their content is consistent across all available documentation and marketing materials to improve AI accuracy.

Source: Search Engine Journal