Cloudflare AI bot blocking reportedly hinders Googlebot indexing

Image credit: Search Engine Journal

Cloudflare’s new artificial intelligence (AI) bot blocking feature is reportedly preventing legitimate Googlebot and Bingbot crawlers from indexing websites, despite an official implementation date of Sept. 15, 2026.

The issue, which could disrupt website visibility on major search engines, appears to be affecting sites that have enabled Cloudflare‘s ‘AI Training = Block’ setting.

A user on Reddit reported that their website, protected by Cloudflare, served HTTP 403 forbidden responses to both Googlebot and Bingbot when the bots attempted to fetch sitemaps. This indicated that the legitimate search engine crawlers were being blocked.

Cloudflare now categorizes these mixed-usage bots, which collect data for both search indexing and AI training, as ‘Search + Training’ bots, according to its official documentation. Consequently, enabling the AI training block configuration affects their access.

The company’s documentation specifies that beginning Sept. 15, 2026, such mixed-usage bots will be blocked by default on new domains and by any configurations designed to block AI training data collection.

The current reports of blocking occurring well before the announced 2026 date have raised questions among web administrators regarding a potential premature rollout or an unforeseen technical anomaly.

Google’s Search Liaison, John Mueller, acknowledged the reports and offered to investigate the matter further, according to statements made online.

The premature blocking could lead to significant challenges for website owners relying on search engine visibility, as it could prevent their content from being discovered and ranked by Google and Bing.

Cloudflare, a major web infrastructure and security company, offers services to millions of websites globally, including protection against malicious bots and denial-of-service attacks.

The company has not yet issued an official statement addressing the reported early blocking of search engine crawlers.

Source: Search Engine Journal