Meta AI crawlers dominate web reading as Google faces publisher scrutiny

Image credit: Search Engine Journal

Global publishers are primarily focused on negotiating with Google regarding artificial intelligence access and compensation, even as Meta‘s AI crawlers have become the dominant, uncompensated readers of web content.

DataDome reported 17.7 billion AI agent requests in the second quarter of 2026, marking a 45 percent increase from the first quarter, with Meta’s agents accounting for the majority of this traffic.

Meta’s AI agents, specifically Meta-ExternalAgent and Meta-WebIndexer, showed significant quarter-over-quarter growth of 74 percent and 163 percent respectively, making them the heaviest readers of web content, DataDome said.

Despite Meta’s extensive content consumption, OpenAI‘s GPTBot remains the most frequently blocked AI crawler by publishers.

Publishers’ attention remains largely on Google, which historically drove significant referral traffic to their sites, leading to current negotiations over compensation for AI summaries that reduce this traffic, industry analysts said.

Meta, however, has historically prioritized keeping users within its own platforms and never promised referral traffic, leading to less perceived loss when its crawlers access content without providing compensation or traffic, according to industry observers.

While Meta has established licensing agreements with major publishers, including News Corp for $50 million annually, along with CNN, Fox News, and USA Today, most smaller publishers receive no compensation for their content.

The bargaining power of publishers is diminishing as large technology platforms increasingly prioritize user retention, making content compensation for AI training a difficult but primary negotiation point, according to analysts at No Hacks.

Source: Search Engine Journal