Cloudflare to block AI bots from sites opting out of data collection

Image credit: Search Engine Journal

Cloudflare will begin blocking Googlebot and Bingbot from websites that have enabled AI training data blocking, effective September 15, 2026, marking a significant shift in how AI bots access web content.

This change, alongside updates to AI search indexing and new campaign benchmarking features in Google Analytics, signals a period of substantial evolution for SEO professionals regarding data collection and content accessibility.

OpenAI’s in-house search index currently processes sites without specific content deals similarly to licensed partners, utilizing titles and approximately 200 characters of content, according to reports.

Google also refiled a DMCA complaint against SerpApi, incorporating content licensing terms after its initial claims were dismissed, intensifying legal challenges surrounding data scraping.

Radu Stoian of Resoneo highlighted that access to crawling, indexing, and data collection is undergoing considerable changes, directly impacting SEO strategies.

Google Analytics’ new Ask Advisor feature will allow users to benchmark campaign performance against anonymized averages from comparable businesses, providing new insights into marketing effectiveness.

Maryam Safari of Enhance Media and PubliCare noted that these developments underscore a broader trend of tightening controls and evolving legal frameworks around web data.

The move by Cloudflare to block major search engine bots from sites opting out of AI training data collection indicates a growing industry response to concerns over intellectual property and data usage.

These collective updates are expected to reshape how digital content is discovered, indexed, and leveraged by both AI models and traditional search engines.

Source: Search Engine Journal