Google tests Web Bot Auth to verify automated web traffic

Image credit: Search Engine Journal

Google is experimenting with Web Bot Auth, a new protocol designed to cryptographically verify automated web traffic and enhance bot identification, the company announced Tuesday.

The experimental standard, also known as HTTP Message Signatures Directory, aims to automate trust between web services by allowing websites to confirm the genuine identity of bots and services interacting with them.

Web Bot Auth utilizes cryptographic verification, making it more difficult for malicious entities to impersonate legitimate bots compared to existing methods that often rely on easily faked user-agent strings.

The protocol establishes a three-step discovery process to replace manual setup between websites and bots, according to the Web Bot Auth Working Group. This process involves standardized JSON Web Key Sets (JWKS) files, well-known addresses (/.well-known/), and self-identifying requests via a Signature-Agent header.

Google stated that the protocol does not determine whether a bot is inherently “good” or “bad.” Instead, it provides a stronger, verifiable signal about a bot’s true identity, thereby improving overall web security.

The company advised users to continue employing traditional verification methods, such as IP address checks and reverse DNS lookups, during this experimental phase. Not all Google user agents currently utilize Web Bot Auth, and Google is not yet signing every request.

The initiative seeks to change how web services authenticate automated traffic, offering a stronger defense against rogue bot impersonation and improving the reliability of interactions across the internet.

Source: Search Engine Journal