Google: LLMs.txt not for AI discovery, has major flaws

Image credit: Search Engine Journal

Google’s John Mueller clarified Thursday that the LLMs.txt standard was never designed to facilitate content discovery by search engines or large language model (LLM) systems, contradicting widespread assumptions among site owners.

Mueller emphasized the protocol’s original intent was to assist LLMs already familiar with a website, not to serve as a mechanism for initial content identification or ranking.

Many website administrators have been misapplying LLMs.txt for discovery and ranking purposes, which directly conflicts with its fundamental design, according to Mueller.

He highlighted that LLMs.txt is inherently unreliable for discovery because it relies on site owners’ self-declarations, which may not accurately reflect the actual HTML content available on their pages.

Discovery, an initial phase in the architecture of search engines, is not addressed by the proposed LLMs.txt standard, Mueller stated. Traditional web crawling and the analysis of HTML pages remain central to how search engines identify and rank content.

Standards such as the Web Model Context Protocol (WebMCP) are better suited for empowering AI agents with actionable capabilities on websites, particularly in e-commerce scenarios, unlike LLMs.txt.

Google has maintained that the core processes of content discovery and ranking continue to be tied to the structure of HTML pages and established web crawling techniques.

Source: Search Engine Journal