Google published a guide on November 17, 2026, detailing how website administrators can configure their content delivery networks, web application firewalls, and bot detection services to permit traffic from its AI agents, known as Google-Agent.
The global technology company’s new document, titled ‘Allow Google-Agent traffic,’ outlines methods for identifying these AI-driven requests and troubleshooting common blocking issues.
Google-Agent requests are identifiable through their specific user-agent strings, IP addresses listed in an official user-triggered-agents.json file, and in some cases, an experimental Web Bot Auth cryptographic signature, according to the guide.
The company emphasized that authorizing Google-Agent traffic does not guarantee improved search rankings or increased website traffic. However, technical teams should ensure that critical user journeys, such as search functions, filters, shopping carts, and booking processes, remain accessible to these agents.
Barry Schwartz, writing for Search Engine Roundtable and PPC Land, reported that this guidance suggests Google’s AI agents operate differently from traditional crawlers.
Unlike typical web crawlers, Google-Agent requests, which function as user-triggered fetchers, generally disregard robots.txt directives. This necessitates access control management at the CDN, firewall, or bot detection system level, the guide stated.
The guide also addresses six common reasons why Google-Agent requests might still be blocked even after initial authorization. These include incorrect rule priority settings, the presentation of interactive CAPTCHAs, uncovered API calls, and IP-based rate limiting.
Webmasters are advised to review their security configurations to ensure seamless interaction for Google’s AI agents, which are designed to simulate user behavior for various AI-driven applications.
Source: Abondance
