LLMs struggle with strategic tasks, produce undifferentiated outputs

Image credit: Search Engine Journal

Large Language Models (LLMs) demonstrate significant limitations in strategic tasks, particularly marketing, due to their predictive nature and tendency to produce convergent, undifferentiated outputs rather than true reasoning, according to recent analysis.

Experts highlighted that these AI systems function as statistical machines predicting the next token, lacking human-like thought or genuine reasoning capabilities, which poses challenges when novel or distinct solutions are required.

Research from Apple, including a paper titled ‘The Illusion of Thinking,’ illustrated how LLM accuracy diminishes sharply with increasing puzzle complexity, alongside issues such as the ‘reversal curse’ and ‘compositional collapse.’

Mark Williams-Cook, a digital marketing expert, noted that LLMs often appear to reason by recalling consensus answers from their extensive training data, a method that becomes problematic when innovation or differentiation is necessary.

A specific prompt, known as the ‘car wash’ scenario, exposed LLMs’ inability to comprehend novel situations; most models initially suggested walking to the car wash instead of driving the car to be washed, indicating a lack of contextual understanding.

The ‘convergence problem’ is a significant concern, especially in marketing, where LLMs trained on similar vast datasets tend to generate outputs that trend towards an average, making it difficult for businesses to achieve unique positioning.

Jeremy Daly, a prominent Substack writer on AI, summarized this convergence as a result of shared data, common incentives, and rapid iteration among AI models.

Initial responses from models like ChatGPT, Claude, and Google‘s Gemini to the ‘car wash’ prompt were incorrect, advising users to walk to the car wash.

Google’s Gemini and Grok eventually provided correct answers to the car wash prompt only after the appropriate solution became widely disseminated and incorporated into their training data, rather than through an improvement in their intrinsic reasoning abilities.

This reliance on existing data for ‘reasoning’ means that LLMs are more likely to recite established answers than to formulate truly original or strategic insights, limiting their utility in fields demanding creativity and distinctiveness.

Source: Search Engine Journal