Google clarified the limitations of the robots.txt protocol regarding artificial intelligence content usage and introduced a new token, Google-Extended, to control content training for its Gemini AI models.
The technology industry is establishing clear guidelines for how web content is utilized by AI systems, prompting discussions around new web standards.
Robots.txt, defined by RFC 9309, functions as an access preference protocol, dictating which URLs a web crawler may request, according to Google. It does not, however, govern how content is subsequently used after it has been crawled.
To prevent a specific AI crawler from accessing content, its user-agent token must be explicitly targeted within the robots.txt file, Google stated. This means that merely blocking a general crawler does not inherently restrict AI training.
Google-Extended is a new robots.txt token that allows publishers to specify whether their crawled content can be used to train Gemini models and for grounding in Gemini Apps and Vertex AI, the company confirmed.
An Internet-Draft co-authored by Google’s Gary Illyes, Mozilla’s Martin Thomson, and others proposes a ‘Content-Usage’ rule for robots.txt to address the broader issue of AI content usage. This draft, currently under consideration by the Internet Engineering Task Force (IETF), aims to define preferences for how crawled content should be utilized, including a ‘train-ai=n’ field.
There is currently no universally accepted standard for expressing AI training preferences independently of bot access, according to the IETF draft. The proposed ‘Content-Usage’ rule seeks to fill this gap, offering a more granular control mechanism for webmasters.
Cherry Prommawin, a webmaster expert, recommended using a percent-encoded form for non-English rule paths in robots.txt. This approach ensures broader compatibility across various crawlers, despite Google’s parser being capable of handling raw UTF-8 characters.
The proposed IETF standard represents an ongoing effort to evolve web protocols to meet the challenges posed by advanced AI technologies, moving beyond simple access control to address content application.
Source: Search Engine Journal
