Cloudflare announced Tuesday a new setting enabling websites to prevent their content from being used for artificial intelligence training while maintaining visibility to major search engine crawlers.
The new “Disallow AI Training” feature allows site operators to add a no-training preference to their robots.txt files, specifically targeting AI models without impeding the indexing functions of Googlebot, Applebot, and Bingbot.
Previously, website administrators who wished to block AI training often inadvertently blocked mixed-use crawlers like Googlebot entirely, impacting their search engine optimization. Cloudflare stated that a separate “Block” option will now completely stop these crawlers for all purposes, distinguishing it from the new AI training preference.
Existing selections for blocking AI training, such as “Block” or “Block on pages with ads,” will automatically migrate to the new “Disallow AI Training” setting, according to Cloudflare.
Mixed-use crawlers will only retain search access under the “Disallow AI Training” setting if Cloudflare designates them as “Accountable,” a status requiring operators to meet specific criteria.
Cloudflare identified Google, Apple, and Microsoft as meeting these accountability requirements due to their commitments. Additionally, Amazon, Anthropic, Meta, and OpenAI meet the criteria for their distinct AI-specific crawlers.
The new setting functions through specific directives like “Google-Extended” for Google and “Applebot-Extended” for Apple. Microsoft’s support for the robots.txt no-training preference is anticipated by early 2027.
Cloudflare also indicated plans to introduce a unified setting for controlling content within AI summaries early next year, further expanding options for content creators.
Source: Search Engine Journal
