Cloudflare AI Rules May Block Googlebot, Impacting Search Visibility

Image credit: Search Engine Journal

Cloudflare’s updated AI crawler management, effective September 15, may inadvertently block Googlebot and other major search engine crawlers for websites attempting to prevent AI training.

The changes could significantly affect search engine visibility for new Cloudflare customers and free users who do not adjust their default settings, the company said.

Cloudflare is revising its methods for identifying and blocking AI crawlers, categorizing bots into three types based on their behavior: Search, Agent, and Training. New default settings will automatically block Training and Agent crawlers on pages displaying advertisements.

The company stated that multi-purpose crawlers, such as Googlebot, Applebot, and Bingbot, will be subjected to the strictest blocking rules if they are identified as performing both Search and Training functions.

Cloudflare also introduced new content-use signals for robots.txt files, which include immediate, reference, and full directives, and updated its definitions for ‘Verified’ bots.

Enterprise Bot Management users now have access to BotBase, a new searchable directory designed to help manage various bot activities.

The company reported a significant increase in AI training requests on its network, which now constitute the majority of all crawler requests. This marks a substantial rise from approximately 20 percent in spring 2025, according to Cloudflare.

Source: Search Engine Journal