OpenAI asserted Monday that robots.txt directives may not apply to its ChatGPT-User page-fetching bot when a human user initiates the request, prompting concerns over widespread bypasses of website access rules.
This stance challenges traditional web crawling protocols and contributed to ChatGPT-User reaching more disallowed pages than any other AI bot of its kind in the first half of 2026, according to a recent report.
The TollBit report for the first half of 2026 indicated that approximately 15 percent of identified AI page-fetchers in Europe accessed URLs that were explicitly disallowed. ChatGPT-User, Bytespider, and Youbot were significant contributors to this trend.
While many newer page-fetching agents maintained low disallow rates, ChatGPT-User stood out as an exception. The bot was disallowed by 26 percent of North American sites and a substantial number of European sites.
OpenAI argued that the human initiation of ChatGPT-User’s actions differentiates it from automated crawlers, potentially exempting it from standard robots.txt compliance.
Perplexity-User also generally ignored robots.txt for reasons it attributed to user initiation, the report stated. In contrast, Anthropic confirmed that all its bots consistently respected robots.txt directives.
OAI-SearchBot, not ChatGPT-User, determines if a site appears in ChatGPT search results. Blocking ChatGPT-User therefore might not prevent a site’s visibility within AI search functions.
Cloudflare is implementing updates to its crawler controls to manage compliance at the network layer. The company announced it would block training and agent crawlers by default on new domains displaying advertisements starting Sept. 15, while search crawlers would remain allowed.
Source: Search Engine Journal
