Ahrefs: 97% of LLMs.txt Files See No Requests from AI Bots

Image credit: Search Engine Journal

Global web analysis by Ahrefs reported that 97 percent of LLMs.txt files received no requests, significantly challenging assumptions about their adoption and use by artificial intelligence bots.

The study, which analyzed 137,000 domains, found that only about 1,100 of 38,000 valid LLMs.txt files registered any traffic, indicating minimal engagement with the protocol designed to guide AI crawlers.

Of the files that did receive requests, 96 percent originated from bots, with non-AI bots dominating the traffic, according to Ahrefs. AI retrieval bots, such as ChatGPT and Perplexity, accounted for a mere 1 percent of these requests.

SEO audit tools were the leading source of requests, making up 21 percent of the total. Unidentified bots followed at 14 percent, with general web crawlers contributing 13 percent and tech profiling tools like BuiltWith accounting for 11 percent.

Across four distinct categories, AI bots collectively generated 19 percent of all requests. Coding agents, including Claude-Code and GPTBot, were the primary AI users, responsible for 10 percent of requests, while AI training crawlers contributed 5 percent.

John Mueller of Google previously stated that LLMs.txt is “not done for search” and serves as a “temporary crutch” for AI coding tools. This perspective aligns with Ahrefs’ findings, which suggest the primary audience for these files is coding agents and training crawlers, rather than AI search or retrieval bots.

Other significant bot types observed included Slackbot and PerplexityBot, with SEO tools like SE Ranking and image search platforms such as Shutterstock also generating requests.

SEO consultant Lily Ray had previously highlighted the anticipation surrounding LLMs.txt files as a new standard for AI bot interaction. However, the Ahrefs data indicates that this anticipated widespread adoption by AI retrieval systems has not materialized.

Source: Search Engine Journal