An experiment demonstrated that different artificial intelligence models consistently chose diverse products from Shopify’s catalog when given identical search requests, underscoring potential inconsistencies in agentic commerce.
The findings indicate that as AI agents gain more autonomy in purchasing decisions, their inherent biases or decision-making algorithms could significantly impact e-commerce sales and product visibility for merchants.
Three Claude AI models from Anthropic — Haiku 4.5, Sonnet 5, and Opus 5 — were tasked with selecting products from a curated list of 20 Shopify items for six distinct search queries, including “coffee beans” and “running shoes.” The models never agreed on a single product choice across any of the six searches.
The experiment was designed to isolate the raw responses of the AI models, excluding external factors such as budget constraints, user taste profiles, shopper history, or agent-specific instructions.
Shopify launched Shop MCP on October 13, 2026, enabling AI agents keyless access to its catalog of more than 1 billion products for searching, building shopping carts, and completing transactions via Shop Pay.
One model, Opus 5, consistently favored a medium roast coffee, selecting it in 48 out of 50 trials. The model reportedly reasoned that a medium roast was the “safest crowd-pleaser” in the absence of stated preferences.
While the models showed internal consistency in their individual choices, their overall agreement rates on product picks remained at zero. Sonnet demonstrated 94 percent consistency with its own selections, Opus 89 percent, and Haiku 67 percent on average.
The products used in the experiment included items from brands such as nobu, Owala, POPFLEX, Liforme, Brooks, Fit2Run, Adidas, and Commonwealth Running Co.
The variations in AI model selections suggest that businesses developing AI agents for commerce may need to consider the specific model’s inherent decision-making patterns to ensure desired outcomes for consumers and retailers.
Source: Search Engine Journal
