Google: llms.txt Files Ineffective for LLM Website Discovery

Image credit: Search Engine Journal

Google’s John Mueller confirmed Tuesday that llms.txt files are not effective for large language model systems to differentiate between websites for discovery, citing their self-reported nature.

Mueller, a senior Google search analyst, stated that these files cannot be trusted for distinguishing one website from another because all sites could make similar self-serving claims, diminishing their value for LLM-based discovery and ranking.

Speaking on Google’s Search Off the Record podcast, Mueller emphasized that traditional HTML and internal linking structures remain fundamental for website crawling and discovery by search engines.

He explained that the design of llms.txt files prevents them from serving as reliable signals for differentiation, a limitation that could be architectural or a problem of signal dilution when claims become universal.

Mueller suggested that while llms.txt files are unsuitable for initial website discovery, they could prove useful for on-site navigation, such as guiding an agent through a purchase process once it has already landed on a specific website.

For instance, an agent could use information within an llms.txt file to complete a task like adding items to a cart or navigating specific sections of a site, according to Mueller.

He anticipated that it could take a significant period, potentially six months to a year or even longer, for agentic systems to standardize on a format that effectively supports such site navigation.

The discussion involved Mueller, Martin Splitt, and Roger Montti, highlighting Google’s official stance on the limited utility of llms.txt files for LLM website discovery and ranking.

Industry publications like SEJ and SE Ranking have previously covered speculation regarding the potential uses of llms.txt files for AI and SEO.

Source: Search Engine Journal