Google AI agent ignores robots.txt, uses crypto ID

Image credit: Search Engine Journal

Google added ‘Google-Agent’ to its list of web fetchers on March 20, 2026, introducing a new user-triggered AI browsing tool that disregards traditional robots.txt directives and experiments with cryptographic identity.

This new agent operates on Google’s infrastructure, browsing websites on behalf of users, and marks a significant shift in how automated systems interact with web content and how site owners manage access, the company said.

Google-Agent functions as a user agent string for AI systems, with Project Mariner identified as the first product to utilize it. Unlike Googlebot, which is a traditional crawler, Google-Agent is activated by user requests.

Its behavior of generally ignoring robots.txt rules mirrors that of a human browsing directly, according to information released by Google.

This approach differs from other user-triggered fetchers like OpenAI‘s ChatGPT-User and Anthropic‘s Claude-User, both of which continue to respect robots.txt directives.

The introduction of Google-Agent establishes a new three-tier model for web visitors, encompassing humans, traditional crawlers, and these new AI agents, Google said.

Google-Agent is also undergoing experimentation with the Internet Engineering Task Force (IETF) draft standard for the web-bot-auth protocol. This protocol aims to provide verifiable bot identity through digital signatures, enhancing security and transparency.

Website owners will now need to implement server-side authentication or more robust access controls to restrict Google-Agent, as the conventional robots.txt file will no longer be sufficient for managing its access, Google indicated.

This development poses new challenges for companies like Akamai, Cloudflare, and Amazon, which provide web security and content delivery services, as they adapt to a changing environment of bot identification and access management.

Source: Search Engine Journal