Cloudflare, AWS Launch Pay-Per-Crawl Systems for AI Bots

Image credit: Search Engine Journal

Cloudflare and Amazon Web Services implemented new pay-per-crawl systems for artificial intelligence bots, altering how AI agents access and use web content globally.

This shift introduces a monetized, consent-driven model for AI access, empowering publishers to control and monetize their content rather than relying on free indexing for traffic, according to statements from the companies.

Cloudflare launched its pay-per-crawl feature on July 1, 2025, utilizing the HTTP 402 “Payment Required” status code to charge AI crawlers for access.

AWS integrated similar capabilities into its Web Application Firewall on June 15, 2026. The AWS system also employs HTTP 402 and the x402 protocol, facilitating stablecoin settlements via Coinbase.

The HTTP 402 status code, which had been largely dormant since its introduction in 1997, has been reactivated specifically to enable these new payment models for web content.

This development marks a transformation in the web’s foundational bargain, moving from an implicit agreement of free indexing in exchange for potential human traffic to a system where AI agents must pay for data access.

Other companies are also entering the AI bot monetization sector. TollBit and Akamai are offering usage-based pricing and network-layer enforcement for AI content access.

The move by Cloudflare and AWS is expected to redefine web content visibility and control, shifting more power to content creators and publishers in the AI economy.

Source: Search Engine Journal