Google has issued guidance for global websites migrating to a Content Delivery Network (CDN) to manage significant surges in Googlebot crawl activity and origin server load, which can lead to performance issues and de-indexing.
This increased traffic results from Googlebot’s enhanced crawl capacity for faster-responding sites and a CDN’s initially empty cache, which directs all initial requests to the origin server, Google reported.
When a site shifts to a new CDN, its cache starts empty, meaning every request for a URL initially results in a cache miss, forcing the origin server to fulfill the request, Google explained.
Unlike human visitors who typically access a limited range of popular pages, bots systematically crawl all available URLs, exacerbating the impact of cache misses on the origin server, according to Google’s Cherry Prommawin.
Implementing global bot protection measures, such as CAPTCHA challenges or blanket 403 Forbidden errors, can inadvertently block Googlebot, potentially leading to pages being removed from Google’s index, Prommawin said.
Google recommends several strategies to manage this increased traffic, including pre-filling CDN caches with site content before migration to reduce initial origin server hits.
The company also advises allowing validated bots, such as Googlebot, to access content without obstruction, while using specific HTTP status codes like 429 (Too Many Requests) or 503 (Service Unavailable) for temporary overload situations.
These status codes signal to search engines that the issue is temporary, preventing de-indexing while the server recovers, Google reported.
Source: Abondance
