OpenAI’s ChatGPT Bots Access Forbidden Web Pages in Europe

Image credit: Abondance

OpenAI’s ChatGPT-User bots are accessing web pages in Europe that are explicitly marked as forbidden by robots.txt directives, with OpenAI defending the behavior as user-initiated.

Approximately 15 percent of identified AI ‘fetcher’ bots in Europe accessed restricted URLs, according to an industry report, highlighting a growing conflict between AI developers and web publishers.

The TollBit ‘State of the Bots’ report for the first half of 2026 indicated that ChatGPT-User, Bytespider, and Youbot were the primary culprits, bypassing blocking rules on nearly half of European websites that attempted to exclude them.

OpenAI justified ChatGPT-User’s actions by stating that the bot operates at the direct request of a user, distinguishing its function from traditional automated web crawls, the report noted.

This stance means that blocking ChatGPT-User does not prevent OAI-SearchBot from accessing a site, as OAI-SearchBot is responsible for determining a site’s visibility within ChatGPT search results.

The report also revealed significant regional differences in blocking rates for various AI agents. Claude-User, for instance, was blocked by 9 percent of European sites but by 26 percent of sites in North America.

Web service provider Cloudflare announced plans to alter its default blocking methods starting Sept. 15. The company will begin blocking training crawlers and agents by default on new domains that contain advertising.

The practice of AI bots bypassing robots.txt raises concerns among web publishers regarding control over their content and potential unauthorized data scraping for AI model training.

Companies like Perplexity and Anthropic also operate AI bots, contributing to the complex ecosystem of automated web access and content indexing.

The ongoing debate centers on whether AI agents acting on behalf of users should adhere to the same automated crawling protocols as traditional search engine bots or if their user-driven nature warrants different rules.

Source: Abondance