Analysis finds most AI assistant, Googlebot traffic is fake

Image credit: Search Engine Journal

More than 80 percent of website traffic requests claiming to be from AI assistants and Googlebot were fraudulent, according to an analysis of website traffic logs released Thursday.

The findings indicated a significant prevalence of spoofed bot traffic, with potentially malicious actors attempting to mimic legitimate web crawlers and AI tools to access sensitive information.

According to the analysis, 81.8 percent of traffic identifying as an AI assistant to CitationIQ.com was fake, with 27 of 33 such requests being spoofed.

The report also detailed that out of 799 requests claiming to originate from Googlebot, only 107 were legitimate, resulting in an 87 percent spoof rate.

Fake AI assistant requests were specifically observed attempting to access sensitive files on the server, including those named .env.production, secrets.yaml, and config.json.

To combat this issue, the author developed a Python script designed to verify the authenticity of bot requests by cross-referencing their reported names with published IP address ranges from major operators.

Major bot operators, including OpenAI, Google, Claude, Perplexity and Common Crawl, publicly disclose their bot’s IP ranges, enabling website owners to verify incoming traffic.

The analysis suggested that website administrators should implement IP verification methods to distinguish between genuine and malicious bot activity.

Such verification is important for maintaining website security and ensuring that only authorized crawlers access site resources, according to the findings.

Source: Search Engine Journal