Google, Microsoft, and OpenAI announced they will adopt a unified robots.txt directive, allowing website owners to control content usage for artificial intelligence training across their services.
This collaboration establishes a standardized method for websites to opt out of AI training, streamlining content management for publishers facing numerous AI providers.
The shared line utilizes a __usage_ai-training policy token within robots.txt, proposed by Anthropic’s Kevin Kelley to the Internet Engineering Task Force (IETF) AI Preferences working group, according to statements from the companies.
Google plans to interpret the shared opt-out as a Google-Extended block when specific Google-Extended directives are absent, affecting its Gemini models and associated applications.
Google aims to support this feature by early 2027, according to Gary Illyes and John Mueller of Google Search.
Microsoft confirmed it will implement the shared line for training purposes and introduce a separate AI Grounding preference.
The company stated it would replace its older NoArchive and NoCache tags with these new preferences by March 1, 2027.
OpenAI intends to honor the new signal within the next few months, according to Krishna Madhavan.
Anthropic, which initiated the proposal through Kevin Kelley, has already begun development to integrate support for the feature across its own services.
The move represents a significant step toward industry-wide standardization for AI content governance, addressing concerns from content creators about unauthorized use of their data for AI model development.
Source: Search Engine Journal
