Microsoft Clarity now flags bots ignoring robots.txt directives

Image credit: Search Engine Journal

REDMOND, Wash. — Microsoft Clarity introduced a new feature enabling website owners to identify and monitor bots that disregard robots.txt directives, providing insights into non-compliant crawler activity.

The update, integrated into Clarity’s Bot Analytics dashboard, aims to help site administrators track AI crawlers consuming server resources and potentially skewing analytics data by bypassing established rules.

The tool now displays requests from bots that violate a website’s robots.txt rules, calculating these disallowed requests as a percentage of total bot activity over a specified period, according to Microsoft.

This new functionality complements Clarity’s existing AI Visibility tools, which previously began showing grounding queries for AI citations.

Website owners can filter bot requests by operator, name, activity type, and specific URLs, allowing for direct comparison between compliant and non-compliant crawlers.

To utilize the feature, a project administrator must manually enable it within the AI Visibility section of Project Settings.

The update supports sites using major Content Delivery Networks (CDNs) including Fastly, Amazon CloudFront, Cloudflare, Azure Front Door, and Akamai, as well as those employing the latest WordPress plugin.

Microsoft stated that while robots.txt files are advisory and do not block access, Clarity’s new feature records requests that bypass these rules, offering a free method for tracking adherence.

The company noted that this enhancement addresses growing concerns among website owners regarding the impact of AI crawlers on server load and data accuracy.

Source: Search Engine Journal