Google crawlers use diverse HTTP methods beyond GET/POST

Image credit: Abondance

Google’s web crawlers use a broader array of HTTP request methods beyond GET and POST, triggered by JavaScript rendering on web pages, a company official confirmed.

Gary Illyes, a Google analyst, said these additional request types, including HEAD, OPTIONS, PUT, PATCH and DELETE, constitute a marginal fraction of the total requests sent by Google’s crawling infrastructure.

Illyes confirmed the information on LinkedIn in response to inquiries regarding crawler behavior, according to reports by Search Engine Roundtable.

These varied requests account for less than 1.5 percent of all HTTP requests initiated by Google crawlers, Illyes said.

The origin of these less common request methods is JavaScript code embedded within web pages, which is executed during Google’s page rendering process.

Illyes clarified that this behavior is not indicative of a new or deliberate crawling practice being implemented by Google.

The company views the occurrence of these diverse HTTP requests as a byproduct of its existing rendering capabilities rather than a strategic shift in how it indexes content.

Web developers and SEO professionals have long focused on optimizing for GET and POST requests, which are traditionally associated with retrieving and submitting data on the web.

The confirmation from Google suggests that while these other methods are present, their impact on overall crawling and indexing remains minimal.

Source: Abondance