Google Tests Cryptographic Protocol for Bot Identity Verification

Image credit: Abondance

Google is globally testing an experimental cryptographic protocol called Web Bot Auth to allow websites to reliably verify the identity of automated bots, differentiating legitimate crawlers from malicious imposters.

The initiative aims to address a long-standing challenge for website owners by providing a verifiable method to confirm bot origins, moving beyond less secure traditional identification methods.

Web Bot Auth, technically named HTTP Message Signatures Directory, is a proposal currently under consideration within the Internet Engineering Task Force (IETF), according to Google.

The protocol utilizes digital signatures that malicious bots cannot reproduce, thereby enhancing security and observability for website administrators.

It functions by storing public keys in a standardized JSON Web Key Set (JWKS) at a predictable location, specifically at the “/.well-known/” path on the service’s domain.

Each bot request then includes a “Signature-Agent” header, which websites can use to cryptographically verify the sender’s identity.

This system replaces declarative identity, such as user-agent strings, with a verifiable identity, making it significantly more reliable than current methods that rely on user-agent, IP address or reverse DNS lookups.

Google stated that while Web Bot Auth enhances security and provides reliable signals, it is not designed to be an automatic whitelist or blacklist system.

Instead, it offers administrators more trustworthy information to make informed decisions regarding bot interactions.

The company confirmed it is currently in an experimental phase, with only a subset of its agents, such as Google-Agent, utilizing the new protocol.

This means that traditional bot verification methods remain essential and must be used in parallel with Web Bot Auth during this testing period.

The IETF’s involvement indicates a potential for industry-wide adoption if the protocol proves successful and gains broader consensus.

Source: Abondance